Using compression with rsync or scp when transferring *.fastq.gz files?

Question

[ropolocan]

Hello Biostars community,

I need to transfer *.fastq.gz files to a server, but unfortunately my bandwidth is an important bottleneck in this process. I have heard that using rsync with the -z flag or scp with the -C flag compresses the data that are being transferred, thus reducing the transfer time. However, how effective is this when the files are already compressed? I also had heard that using rsync over ssh is much faster than scp. In that case, is compression still necessary? Is there are another approach you would recommend to speed up the transfer of *.fastq.gz files?

Thank you in advance for your time.

Answer 1

[Alex Reynolds]

You can't get much more compression out of compressed data. Otherwise you could keep running compression algorithms on different data until there's nothing left, and you can't extract back to the original data that way.

[ropolocan]

Thank you very much for your answer, @Alex Reynolds. It makes sense to not compressing data that are already compressed. Do you have advice on what other parameters of rsync or scp could be used to speed up the transfer of the compressed FASTQ files? Thank you.

[Alex Reynolds]

Are you running rsync without encryption, i.e. not over SSH? Encrypting data (as scp will do) will slow down your data exchange. Compressing already compressed data will further waste time. You might try parallelizing your data accesses, trying two or more simultaneous rsync calls from one or more computers to try to saturate your network connection. Once you have all the files locally, you can collect them together.

[ropolocan]

Thank you very much for your answer. Yes, I do run rsync without encryption. I will look into trying the parallelization that you suggest.