What is a distributed cache?
I have been using Distributed caching solutions for quite some time now (NCache , AppFabric etc) and I am going to answer all three questions based on my experience with Distributed caching.
1: Distributed caching solution allows you to keep data on all the servers by creating a cache cluster. Lets say you have 2 cache servers(server nodes) and you have added 10 items in your cache. Ideally 5 items should be present in both of the server nodes since the data load gets distributed between the number of servers in your cache cluster. This is usually achieved with help of hashing and intelligent data distribution algorithms. As a result, your data request load also gets divided between all cache servers and your achieve linear growth in transnational capacity as you as more servers in the cache cluster.
2: A cache cluster can contain many server machines which are also called server nodes. Yes, Server A is a server node or server machine in your example.
3: Typically Distributed caching system are very reliable using replication support. If one or more servers go down and you had the replication turned on then there will be no data lose or downtime. NCache has different typologies to tackle this such as replicated topology and Partition of replica topology where data of each server is replicated over to the other server as well. In case one server goes down, the replicated data of that server is automatically made available from the surviving server node.
In your example, the data of server A(1, 3, 5, 7, 9) is replicated to server B(2, 4, 6, 8, 10) and vice versa. If server A goes down, the data of server A that is present on Server B will be made available and used from there so that no data lose occurs. So if server A goes down and the application requests for data (1), this data will get retrieved from Server B as Server B contains backup of all the data of Server A. This is seamless to your applications and is managed automatically by the caching system.
Yes, half the data on server a, and half on server b would be a distributed cache. There are many methods of distributing the data, though some sort of hashing of the keys seems to be most popular.
The terms server and node are generally interchangeable. A node is generally a single unit of some collection, often called a cluster. A server is generally a single piece of hardware. In erlang, you can run multiple instances of the erlang runtime on a single server, and thus you'd have multiple erlang nodes... but generally you'd want to have one node per server for more optimum scheduling. (For non-distributed languages and platforms you have to manage your processes based on your needs.)
If a server goes down, and it is a cache server, then the data would have to come from its original source. EG: A cache is usually a memory based database designed for quick retrieval. The data in the cache sticks around only so long as its being used regularly, and eventually will be purged. But for distributed systems where you need persistence, a common technique is to have multiple copies. EG: you have servers A, B, C, D, E, and F. For data 1, you would put it on A, and then a copy on B and C. Couchbase and Riak do this. For data 2, it could be on B, and then copies on C and D. This way if any one server goes down you still have two copies.