download BLAST NCBI nt database

Question

[Sony]

Hi everyone,

I am trying to download BLAST NCBI nt database by using this

./update_blastdb.pl nt --decompress --num_threads 16

But it has only done for some of the files, and I also got this error:

The contents of https://ftp.ncbi.nlm.nih.gov/blast/db/nt.017.tar.gz are up to date in your system.
The contents of https://ftp.ncbi.nlm.nih.gov/blast/db/nt.018.tar.gz are up to date in your system.
The contents of https://ftp.ncbi.nlm.nih.gov/blast/db/nt.019.tar.gz are up to date in your system.
The contents of https://ftp.ncbi.nlm.nih.gov/blast/db/nt.020.tar.gz are up to date in your system.
The contents of https://ftp.ncbi.nlm.nih.gov/blast/db/nt.021.tar.gz are up to date in your system.
The contents of https://ftp.ncbi.nlm.nih.gov/blast/db/nt.022.tar.gz are up to date in your system.
The contents of https://ftp.ncbi.nlm.nih.gov/blast/db/nt.023.tar.gz are up to date in your system.
The contents of https://ftp.ncbi.nlm.nih.gov/blast/db/nt.024.tar.gz are up to date in your system.
The contents of https://ftp.ncbi.nlm.nih.gov/blast/db/nt.025.tar.gz are up to date in your system.
The contents of https://ftp.ncbi.nlm.nih.gov/blast/db/nt.026.tar.gz are up to date in your system.
The contents of https://ftp.ncbi.nlm.nih.gov/blast/db/nt.027.tar.gz are up to date in your system.
The contents of https://ftp.ncbi.nlm.nih.gov/blast/db/nt.028.tar.gz are up to date in your system.
The contents of https://ftp.ncbi.nlm.nih.gov/blast/db/nt.029.tar.gz are up to date in your system.
The contents of https://ftp.ncbi.nlm.nih.gov/blast/db/nt.030.tar.gz are up to date in your system.
The contents of https://ftp.ncbi.nlm.nih.gov/blast/db/nt.031.tar.gz are up to date in your system.
The contents of https://ftp.ncbi.nlm.nih.gov/blast/db/nt.032.tar.gz are up to date in your system.
The contents of https://ftp.ncbi.nlm.nih.gov/blast/db/nt.033.tar.gz are up to date in your system.
The contents of https://ftp.ncbi.nlm.nih.gov/blast/db/nt.034.tar.gz are up to date in your system.
The contents of https://ftp.ncbi.nlm.nih.gov/blast/db/nt.035.tar.gz are up to date in your system.
The contents of https://ftp.ncbi.nlm.nih.gov/blast/db/nt.036.tar.gz are up to date in your system.
The contents of https://ftp.ncbi.nlm.nih.gov/blast/db/nt.037.tar.gz are up to date in your system.
The contents of https://ftp.ncbi.nlm.nih.gov/blast/db/nt.038.tar.gz are up to date in your system.
The contents of https://ftp.ncbi.nlm.nih.gov/blast/db/nt.039.tar.gz are up to date in your system.
The contents of https://ftp.ncbi.nlm.nih.gov/blast/db/nt.040.tar.gz are up to date in your system.
Downloading https://ftp.ncbi.nlm.nih.gov/blast/db/nt.041.tar.gz...curl: (56) OpenSSL SSL_read: error:0A000126:SSL routines::unexpected eof while reading, errno 0
corrupt download, trying again.
Downloading https://ftp.ncbi.nlm.nih.gov/blast/db/nt.041.tar.gz...curl: (23) Failed writing body
curl: (23) Failed writing body
Use of uninitialized value $_ in split at ./update_blastdb.pl line 718.
Use of uninitialized value $rmt_digest in string ne at ./update_blastdb.pl line 619.
corrupt download, trying again.
Downloading https://ftp.ncbi.nlm.nih.gov/blast/db/nt.041.tar.gz...curl: (23) Failed writing body
curl: (23) Failed writing body
Use of uninitialized value $_ in split at ./update_blastdb.pl line 718.
Use of uninitialized value $rmt_digest in string ne at ./update_blastdb.pl line 619.
too many failures, aborting download!

I tried again many times to download this nt database, but it also got the same error. Are there any solutions to solve this problem ? Or any ways to download ncbi blast nt database instead of download one by one nt* file by wget command ?

Thank you.

[GenoMax]

While not the exact errors you have noted here we have also encountered errors downloading pre-formatted databases from NCBI for last 2-3 months. The process seems to fail on random files and may take several iterations before it completes.

I am going to tag PeterC_NCBI who works at NCBI to see if he can shed some light.

[PeterC_NCBI]

Thanks! Looking at this now.

[GenoMax]

Most of the times failures are because of missing .md5 files for a specific piece (random ones) if that helps.

[PeterC_NCBI]

Good to know. Thanks.

[PeterC_NCBI]

I had the BLAST development team look at this. They say the errors here from curl reporting network errors and errors writing to local disk: CURLE_RECV_ERROR (56): Failure with receiving network data. CURLE_WRITE_ERROR (23): An error occurred when writing received data to a local file, or an error was returned to libcurl from a write callback.

As far as the missing ,md5 files. They think this is possible if the download was taking place at the same time the databases were being updated.

They have created tickets to improve error handling on failure to download files and are investigating whether whether the updates to the FTP site impact downloading BLASTDBs from NCBI.

[Sony]

I tried again many times to download this nt database, but it also got the same error. Are there any solutions to solve this problem ? Or any ways to download ncbi blast nt database instead of download one by one nt* file by wget command ? Thank you.

[GenoMax]

You have several hundred GB's of space available correct? Last I had looked nt db was over 350GB.

[Sony]

yes, I have 4.1TB space for download

[GenoMax]

You could try setting up your own download solution directly wget/curling nt files from: https://ftp.ncbi.nih.gov/blast/db/

Otherwise keep trying the update script until you succeed.

[Sony]

Hi all, I am constructing the pangenome on rice. And after I assembled my novo contigs, I want to detect the contamination (non-green plant species and fungi). On the article reference, they used Blast to search the assembled contigs with the Blast NCBI nt database to detect the contamination from non-green plant species and fungi. Here is the list of nt files of blast nt database. I know that (nt_euk) is associated with eukaryotic sequences ; prokaryotic (nt_prok); viral (nt_viruses); and other sequences (nt_others). But on the list file of blast nt database, I saw files: nt.000.tar.gz to nt.000.tar.gz .... nt.124.tar.gz , I just want to ask what is it database ?

in case I focus on non-green plant species and fungi contamination, which nt files should I download (nt_euk and ??? ) ? Thank you for all.

[Istvan Albert]

this is a different question altogether and should be posted as such.

Your new post should have the title

How to construct a blast database for non-green plant species

Then describe the above what you have tried.

Answer 1

[Istvan Albert]

Try downloading the file manually and see what happens. See the files here:

• https://ftp.ncbi.nlm.nih.gov/blast/db/

It is as simple as

curl -O https://ftp.ncbi.nlm.nih.gov/blast/db/nr.41.tar.gz

then unzip it.

You can easily automate the above command for all other links.

I had some problems with the perl wrapper in the past, really all it does is that it avoids download files where the hash has not changed.