please help :( ftp download from NCBI db is keep on being aborted during download

Question

[krmoon518]

hi, I've been trying to download refseq bacteria protein sequences from the NCBI database using ftp. I simply signed into the ftp.ncbi.nlm.nih.gov on my terminal and attempted the following:

mget bacteria.nonredundant_protein.*.protein.faa.gz

At first, the download seems to be working without any trouble, but after downloading about 40 files, the whole download process gets aborted with the following messages:

No control connection for command: No such file or directory local: bacteria.nonredundant_protein.1279.protein.faa.gz remote: bacteria.nonredundant_protein.1279.protein.faa.gz

I've been trying several different methods to download the data but it has been unsuccessful..

Could you please help..?

Thank you in advance!!

Answer 1

[massa.kassa.sc3na]

I recommend a workaround:

use the https interface of the ftp service. How to: use the https://ftp.ncbi.nlm.nih.gov/refseq/release/bacteria/ (instead of the ftp://) and the file you want with your favorite command-line downloader (e.g. wget, curl). (the https is in my experience significantly faster then the ftp interface for the ncbi).

example command would then be:

wget https://ftp.ncbi.nlm.nih.gov/refseq/release/bacteria/bacteria.nonredundant_protein.12.protein.faa.gz

The only downside is, that you need to make a list of the urls (files) you want to download yourself (no wildcards), but this should be easy enough (e.g. download the ftp listing or copy-paste the file names and batch paste the https... part of the address in front of the file names). Then e.g. wget -i FILE_WITH_URLS, and you are done.

If you are feeling fancy, you can use the aspera client (a bit tricky to setup and some institutions firewalls don't like it), info at the ncbi's website.

[krmoon518]

it was simple but brilliant way to solve the situation! i just downloaded all the data that i needed! thank you!!