Large file transfer

Question

[philgev2]

Hi

I need to transfer large files between Linux servers in two locations. Is there a better approach than scp and rsync.

Thanks Philge

[Mensur Dlakic]

Better in what sense? Faster? Safer?

You may want to compress the files before the transfer, or use the -C switch with scp or -z with rsync, both of which will compress the stream.

[philgev2]

Both faster and safer

[ATpoint]

rsync is probably as efficient as it gets when it comes to freeware, so if you experience slow download rates then you probably have a bandwidth problem. Commercial software would be Aspera but depending on the size of the files you indeed could simply pack it onto a harddrive and ship it. That is fairly common, Novogene for example does that all the time.

Answer 1

[Jean-Karim Heriche]

rsync with compression is the standard way to go. I've also used lftp to parallelize transfer but be aware that this puts some pressure on the receiving end and some may have limits. You don't say what large is for you and where the locations are but for long distance transfer of several tens of TB, it may still be faster and safer to send copies on disks.