How Long Should It Take To Make A Blast Database?

Question

[Andrea_Bio]

Hi

I'm trying to install a local copy of polyphen and one of the stages is to make a blastdb from uniref100.

I ran this command last night and its still going nearly 12 hours later. I don't know if this is normal:

==============================================

makeblastdb -in uniref100-formatted.fasta -dbtype prot -out uniref -title 'UniRef100 2010_07, 15-Jun-2010' -parse_seqids

Building a new DB, current time: 11/03/2010 23:51:19 New DB name: uniref New DB title: UniRef100 2010_07, 15-Jun-2010 Sequence type: Protein Keep Linkouts: T Keep MBits: T Maximum file size: 1073741824B

============================================

The program just seems to be sitting there. The uniref file is 233GB so I don't expect it to be quick but I wasn't expecting 12 hours. I am running linux on a virtual machine on an external hard drive with only 1 gig of RAM so its not on the fastest machine in the world either.

Thanks

[Khader Shameer]

Are you running a local installation of PolyPhen-2 ? I had similar problems while running the makeblastdb during PolyPhen-2 installation. Usually it wont take that much time for NCBI-nr/Uniprot.

[Khader Shameer]

I tried to install PolyPhen 2, but had the same issue with the Uniref steps, after 2 days or so, the makeblastdb exited on its own with a segmentation fault error message. You may change the BLAST+ version as suggested by Daniel. Please let us know if you were able to install it completely. From my experience, other steps won't take much time.

[Andrea_Bio]

I'm at 3 days on the uniref database stage!

[Andrea_Bio]

I'm in the process of making a polyphen2 local installation. I've been told one of the steps takes 4 days??? Is this preparing the pdb database?

[Andrea_Bio]

What should the program produce and how big should it be? I can see in my directory i have a file called uniref100.00.pin. I've never done this before. I might try and get the output from someone else who might have already done it.

[Sam]

Does anyone have the output of this task downloadable? I've reverted to 2.23 and I still find that it's running for a long time. I'm trying this on my local macbook pro

Answer 1

[Daniel Standage]

This might be a version issue as well. I've found that several of the BLAST utilities (makeblastdb, blastx in particular) run slower in the latest release (2.2.24+). I haven't taken the time to try and track down what exactly in the data causes the slowdown, but if one of the earlier versions (2.2.23+ or 2.2.22) goes quicker, that's something we need to report to the developers.

[Daniel Standage]

Confirmed. I just tried indexing a 41MB data set using makeblastdb 2.2.24+ and it wasn't a quarter done by 5 minutes. I killed the job, ran it with makeblastdb 2.2.23+ and it finished in about 30 seconds.

[Andrea_Bio]

Are any of these blast programs optimised to make use of the graphics card cpu (using cuda libraries i think?). That would speed things up a bit

[Daniel Standage]

I'll have to admit that my knowledge of hardware and hardware acceleration is quite limited. My guess is no, but I can't be sure.

[Andrea_Bio]

I might downgrade to 2.2.23+. Do you know what files the program produces?i only have a file called uniref100.00.pin.

[Daniel Standage]

For a recent task, I indexed a file called AdhA.A2.ffn and makeblastdb 2.2.23+ created the following files: AdhA.A2.ffn.nhr, AdhA.A2.ffn.nin, AdhA.A2.ffn.nog, AdhA.A2.ffn.nsd, AdhA.A2.ffn.nsi, AdhA.A2.ffn.nsq. When you're working with protein data sets, the extensions begin with "p" instead of "n". To be honest, I've never seen documentation on what each of the index files contains, and my curiosity has never been enough to research the issue. I spent a bit of time looking this morning to no avail. I typically rely on the output from makeblastdb to know whether the file was indexed correctly.

Answer 2

[Paulo Nuin]

Depending on your processor speed, 12 hours or more doesn't sound abnormal, specially with you 1Gb memory cap. Using a machine with 64Gb of memory, but a slow clock, it took me some hours to create a local copy of nr and nt.

[Neilfws]

Agreed; at first I thought that this did sound slow, but it's the 1GB RAM that's killing you. Write to an external HDD could be slow too.

Answer 3

[Bamyasi]

Do not use version 2.2.24+ use 2.2.23+ instead! Version 24+ is buggy and in some cases may run 1,000 slower than previous releases. Had this reported to NCBI BLAST developers 3 months ago, got a confirmation but still fixed binaries nowhere to be found on NCBI FTP.

Answer 4

[Marc Jan Bonder]

Here i have the same problem, I'm also trying to create the uniref100 BLAST database. But it already has a runtime of more than 2 weeks. Using blast 2.2.24+, running on a intel core 2 duo 2.8 ghz with 8gb of ram, ram used 16% and cpu used 100%. I contacted uniprot for a downloadable version of the formated uniref100 database but they don't provide this.

[Andrea_Bio]

2 weeks??? Mine has been running for 3 days now on the slowest machine in the world. It sounds like it might be best to go back to blast 2.2.23???

Answer 5

[Yannick Wurm]

You're using the -parse_seqids option. That slows things by several orders of magnitude.

Answer 6

[Magnus B]

I am trying to make the same database for polyphen, and makeblastdb has run for over 1 week. The files are still growing, but I do thing there is something wrong with the latest version of the blast program.

Answer 7

[Martijn]

Same here. Running for a month and not even half of it processed. Killing it now and will move back to 2.2.23. Thx bamyasi.

Answer 8

[Julio514]

Had the same problem. Swtiched back to 2.23 and worked fine!