Send BAM files from server to cluster

Question

[Manuel Sokolov Ravasqueira]

Hi,

I have 82 BAM files + BAI in a remote directory which I have acess via smb I want to send these files to server which I have acess via ssh from this remote directory. So from my computer (SMB) ------> lab files (SSH) ------> Cluster.

I have tried using SCP with and without the option -C and it the transfer ends before 100% The File is sent yet only 50/60% of the file.

I know this is not bioinformatics question directly yet having files in order for variant calling and other pipelines is crucial. Is there any better way to transfer over BAM files? What is the best practice in this situation?

Thanks.

Answer 1

[ATpoint]

Seconding both GenoMax and Zhenyu Zhang. rsync has the advantage that it has many checks in place to guarantee file integrity. It will only create the final visible file on the destination once upload finished properly. So you do not run into the relatively common scp issue that you have a file sitting at destination that you do not know whether it is complete.

Speaking of integrity: Run md5sum on the files at origin location, and then again at destination and compare them to ensure everything went properly.

Answer 2

[GenoMax]

Sounds like your connections are not very reliable. I would recommend trying rsync .

[Zhenyu Zhang]

second GenoMax. scp only for single small file, rsync for large ones.