How to send huge files from one server to another server

BitTorrent might be a good solution for you.

Create the torrent on the origin server, transfer the torrent file to the destination, and then use one of the clients as the bittorrent tracker. Bittorrent breaks up the file into manageable pieces and makes sure they're all transferred without error.

There's obviously some overhead in creating the torrentfile and there's no swarm to enhance your download speed, but if you're dealing with a crappy link, it may work very well for you.


FTP is awful -- it was born in the mid-1980s and, frankly, it should've died there.

I'd probably start with scp (Secure Copy), which should be part of the openssh or openssh-client package on your favorite Linux distro (including Cygwin), or available as part of the PuTTY package if you're running Windows without Cygwin. You'll need to configure an ssh server on the destination host, but that's pretty straightforward assuming you've got root/Administrator access (if you don't, things get harder); once you've got the ssh server running, and can access it from the source host, it's just a matter of

 user@source $ scp /path/to/file user@destination:/path/to/receiving/directory

This should satisfy your point 1 pretty well, since scp has reasonably low overhead; it should certainly satisfy point 2, as it will definitely detect a failing connection and can be (probably) configured or (certainly) scripted to retry as many times as necessary; it covers point 3 easily, as no intermediate host or service is required; and it covers point 4 nicely as well, since you can install an ssh server on both hosts and then transfer the file in whichever direction you prefer. You also get encryption for free, which may or may not be of use to you.

The OpenSSH manual is probably a good place to start, and I'll be glad to offer further assistance if you do end up going this route -- I have some experience in using scp/ssh for these sorts of transfers (although not from Canada to Africa or vice versa, and not for a single file topping 70GB in size, I admit!)

Hope this helps!


I think it would be a good idea to split the file into several small ones, transmit them, and then put them together again on the remote server.

An example (on Linux) of how to split and concatenate is here: http://www.techiecorner.com/107/how-to-split-large-file-into-several-smaller-files-linux/

You should also have something like an ssh connection to the remote server so that you can concatenate them there.

Helpful would also be a tool like "md5sum" to check if the transmitted files are unchanged by comparing the hashes.

You could either write a small shell script to automate some stuff, so that you can painlessly transmit a lot of small files, or you make it manually and try it with few but bigger chunks...