Download multiple fastq files

Question

[lilyolmstead]

I want to download the following fastq files in Salmon all at once:

ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR106/014/SRR10611214/SRR10611214_1.fastq.gz ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR106/014/SRR10611214/SRR10611214_2.fastq.gz ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR106/015/SRR10611215/SRR10611215_1.fastq.gz ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR106/015/SRR10611215/SRR10611215_2.fastq.gz ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR106/016/SRR10611216/SRR10611216_1.fastq.gz ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR106/016/SRR10611216/SRR10611216_2.fastq.gz ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR106/017/SRR10611217/SRR10611217_1.fastq.gz ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR106/017/SRR10611217/SRR10611217_2.fastq.gz

Is there a way to do this using a bash for loop or fasterq-dump?

[GenoMax]

Put the links in a file (one per line). Then use wget -i url.txt (if your file is called url.txt. If you want to use curl instead then use xargs –n 1 curl –O < url.txt.

[Bastien Hervé]

Is wget -i your_list what you are looking for ?

[evelyn]

After making a input_files.txt file with only SRR names and put this in a input_file_folder. You can use this script:

cd input_file_folder
NUMB_LINES=$(wc -l input_files.txt|cut -f 1 -d ' ')
for i in $(eval echo {1..$NUMB_LINES}); do
FILE_NAME=$(head -n $i input_files|tail -n 1 |cut -f 1)
VAR1=$(head -n $i input_files|tail -n 1 |cut -f 2)
VAR2=$(head -n $i input_files|tail -n 1 |cut -f 3)
INPUT=$(echo ftp:link)
wget -i $INPUT -O $FILE_NAME
done

Answer 1

[Arup Ghosh]

Save the links in a text file and use GNU parallel with wget to download the files simultaneously.

parallel --verbose "wegt {}" ::: $(cat download.txt)

If you want to use multiple connections for download use aria2.

parallel --verbose "aria2c -x <number of conections> {}" ::: $(cat download.txt)

[GenoMax]

Assuming OP likely has a single ethernet port on their machine this sounds like an invitation for trouble. Multiple threads are just going to compete with each other and eventually may take more time to complete the download.