Connecting Erlang nodes when an internal and external IP address are at play
What you are trying to achieve is definitely doable.
Erlang's distribution addresses are in two parts: the node name and the host name. They are separated by the
Host names can be numeric IPv4 addresses. They can also be domain names. There are two distinct modes, where host names are short (single word, e.g.
vm1) and where they are long (several words, e.g.
vm1.domain.com). IP addresses are long names. Nodes started in one mode (short or long) can only communicate with nodes started in the same mode. Nodes are also protected by a cookie: a node will only accept incoming connection with a matching cookie. The easiest is to start all nodes of a given cluster with the same cookie.
When an Erlang node tries to connect to another Erlang node, it needs to find the IP address of the distant node. If it is the same as itself, it will simply try to connect on the local host. If it is different, it will try to resolve this host name to an IP address.
Then it will connect to the
epmd daemon on this host to be told which port Erlang is running.
epmd as well as Erlang nodes listen on all interfaces (by default).
Solution and example
Based on this mechanism, you could use either short or long names, but exploit the resolution mechanism. The easiest on Unix would be to configure different IPs on each
/etc/hosts of your machines (especially on the two virtual machines) so they will connect to each other through their private addresses, while being accessed through their public addresses.
Let's say that Virtual machine A (VM A) has private IP address 10.0.0.2 and public IP address 126.96.36.199 and VM B has private IP address 10.0.0.3 and public IP address 188.8.131.52. Let's also say that you decided to go for short names.
You could put on VM A this entry in
You could put the matching entry on VM B's
And on all the external clients, you could put:
184.108.40.206 vma 220.127.116.11 vmb
You would start your nodes as follows:
# Node foo on VM A: erl -sname [email protected] -cookie RANDOMCOOKIE # Node foo on VM B: erl -sname [email protected] -cookie RANDOMCOOKIE # Client nodes: erl -sname client -cookie RANDOMCOOKIE
You can avoid the
/etc/hosts edits on client nodes if you have a domain name (e.g.
yourdomain.com) and you can get
vma.yourdomain.com to resolve to 18.104.22.168. You can also use a specific Erlang Inet configuration file.
Erlang distribution mechanism is not meant to be public facing. Besides, all communications will be unencrypted. I strongly suggest to configure firewalls on each host to only let connections from other cluster servers and use SSL distribution.
For the firewall: Erlang distribution uses port 4369 for
epmd as well as random ports for each node. You can limit the range of these random ports by using Erlang kernel application environment settings
inet_dist_listen_max. You will need to allow incoming TCP connections on these ports, but only from other hosts of the cluster.
SSL distribution is quite complex to setup but well documented. The main drawback in your case is that all connections should be over SSL, including those between the two virtual machines on their private network, and local connections to open remote shells.
Before answering the question, I want to point out, that Distributed Erlang is not secure and you should not use it outside of trusted local area network. (I had to leave that comment after I saw "external ip address", I assume it doesn't mean public). Below is a list of 4 important things, you should be aware of:
- Ip address is part of the node name.
When you start a node, it is good to give it a name like this:
erl -name [email protected]
When you try to connect to that node from other machine, you can use something like this in erlang shell:
The important part is that node name:
'[email protected]' is an atom. It is not "name and ip" - it is one thing. Even, if your second node is on the same machine, you can't use:
because it is different node name.
- Connections are on top of TCP
This means, that to connect two nodes, only one has to see the other.
Example: you have:
- one node on a machine with external ip address (the client) and
- one node on a machine with internal ip address (the caching node)
- you can't connect from client to caching node (because it can't see that ip address)
- you can connect from caching node to client (because caching node sees the external ip of client) and the connection is bidirectional!
In the second case, the connections is just as if they were on the same network. You only have to think about who should initialise the connection.
- When you connect new node to existing cluster, it tries to connect to all other nodes.
If you don't want to that - use hidden nodes.
- Make sure, that you use the same Erlang cookie everywhere.
But I think, you have that covered, if you were able connect other nodes.
The easiest solution is to use external ip addresses everywhere, because Erlang distribution was designed to run on local network. Harder solution involves making sure, that you connect from nodes in local network to nodes with external ip.