How to download all searched SRA result data at once?

Question

[sratestimon]

Dear Biostars,

There has been a lot of question and tutorial here teaching people how to download SRA data using SRA toolkits, R Bioconductor program, NCBI ENTREZ and so on, and I have learnt a lot from them.

It is easy to search on the SRA website to find specific data using keywords like (METAGENOMIC[Source]) AND (RNA-Seq[Strategy]).

I usually download SRA data using the SRA tookilts with fastq_dump and can easily get one specific data given the accession number, i.e. SRRXXXXXX.

However, when it comes to the situation that I would like to download all the searched SRA data resulted from the keywords, fastq_dump does not seem to be a good choice...

Is there any other method in downloading all the searched data from SRA? i.e. Is there any way to download ALL the data from the following link: https://www.ncbi.nlm.nih.gov/sra/?term=(RNA-Seq%5BStrategy%5D)

Thank you very much!

Answer 1

[timstuart]

Hi, I had a similar problem so wrote this script to do it, which I find is better than fastq-dump: https://github.com/timoast/ngs-tools/blob/master/download_from_sra.sh

You just need to get the list of accession names from the SRA by going to that page you linked, select send to > file > format = accession list

[sratestimon]

Thanks Tim! So now I just need to make a list of accession number from the searched result.

Is there any restriction on the list? Say does it have to be in .txt file? Is there any restriction on the delimiter too?

I am sorry for asking these, may seem silly, question as I am actually still a bit new to programming ...

Thank you very much!

[timstuart]

You just need one line per SRA accession, exactly as the file list is downloaded from the SRA. Then run the program like this:

sh download_from_sra.sh -f SraAccList.txt

[sratestimon]

Thanks Tim! This helped a lot!

[sratestimon]

Hi Tim, I am sorry that this might seem to be a silly question but I am still pretty new to linux.

So, I downloaded your script and had made the accession list and tried to run with the following command:

$ sh download_from_sra.sh -f ./test/test2.txt

Where the are some error popped out as followings:

download_from_sra.sh: line 2: $'\r': command not found download_from_sra.sh: line 4: $'\r': command not found download_from_sra.sh: line 9: $'\r': command not found download_from_sra.sh: line 11: $'\r': command not found download_from_sra.sh: line 17: $'\r': command not found download_from_sra.sh: line 24: syntax error near unexpected token $'in\r'' 'ownload_from_sra.sh: line 24: case $opt in

Did I missed anything or did I did something wrong?

[WouterDeCoster]

Did you by chance open this using Windows and saved it again? Looks like Windows line endings. Try dos2unix download_from_sra.sh to fix the line endings.

[sratestimon]

Oh yes... I did... Thanks Wouter!

Would you mind if I ask one more thing.

As I am using cygwin64, it is somehow strange that I cannot run SRAtool on bash... but I can do so when using window command terminal.

It seems that SRAtoolkit for Linux is not applicable on cygwin64 ... I know that virtual machine is one of the solution. However, is there other way to execute Tim's script on Window beside using virtual machine?

Thanks a lot!

[WouterDeCoster]

I don't know - I'm not a Windows guy :-)

[sratestimon]

I see ... Thanks Wouter anyway! Maybe I should start using virtual machine or even simply purchase a new desktop using linux ...

[WouterDeCoster]

Admittedly it sounds scary at first to change your operating system you got so used to, but I have no regrets of switching to Linux (Ubuntu).

[sratestimon]

Hi Tim, do you think there is any way to run your script on a window based terminal?

[timstuart]

I don't use windows myself so I'm not 100% sure, but I think you can now run bash on windows 10...or maybe try starting docker container with ubuntu bash: docker run -t -i ubuntu bash https://www.docker.com/docker-windows

You'll also need pigz so run apt-get update && apt-get install pigz

[sratestimon]

Thanks Tim, I will try it out !