Getting BLAST to give output whilst running

Question

[CC]

I run large BLAST jobs that can sometimes take days or weeks to complete. With other jobs I find I can see the output of the job as it is running, which lets me know it is working and how its progress is doing, however with BLAST it doesn't give any output until it is finished, which means I never really know it's working until it's done. I was wondering if there is any way to get BLAST to flush output as it is running, or some other way for it to show it is working? I am using Version 2.3.0.

[GenoMax]

Are you using a job scheduler? Different schedulers handle intermediate output differently. e.g. LSF may buffer the output in a temp location while the job is running. You should be able to look at that output while the job is in progress (e.g. in LSF you would bpeek {job_id}).
If you are running blast standalone it should be writing output as it goes.

[CC]

Yes, I am using a job scheduler as I am using a cluster at work. I've been told it is managed by a combination of Torque and Moab... is there a way to look at intermediate output with those schedulers?

[GenoMax]

I am not familiar with torque but this may help.

[CC]

Thanks, this makes it clear.

Answer 1

[lakhujanivijay]

I don't know if this helps but what I usually do is that I enable desktop notification on my PC; something like this:

This notification pops up; say every hour and tells me how many of the query sequences have been blasted so far (I already know how many were there in total). So, that provides me an idea of the progress without disturbing me from my current work. Here is a short bash script to achieve the same:

while true; do 
count=`ls -1 *.xml | wc -l`
   if [ $count != 0 ]
   then 
     total_blastx_done=`grep "Iteration_query-def" *.xml -cH`
     notify-send "Blastx compltd: $total_blastx_done"
     sleep 900
   fi 
done

Notice the following:

1. my blastx output format is xml format i.e. -outfmt 5
2. I grep the tag "Iteration_query-def" from the xml file. This could be anything, but should be unique for each query sequence.
3. Script looks for all xml files in the current folder where blast was fired and notify status of all of them.
4. sleep command takes no. of seconds as input i.e. how frequently you want yourself to be notified. In my case (script above, it is set to 900i.e every 15 minutes). You can set it accordingly.
5. You can run the script in the background like mentioned below:

$ nohup sh count_blastx.sh &

[Ken]

If I understand correctly, Blast can tell you how many results it has gotten but it can't tell you how many it is going to get, therefore it is impossible to build a progress bar for Blast. Am I right? Do we think Blast even knows, internally, how far along it is?

Answer 2

[Manu Prestat]

Some answers here: Show Progress (Bar?) For Blast+ Query