How To Use Perl To Submit Mpiblast Jobs Via Qsub

Question

[Mchimich]

Hello, i'm writing a perl script that use qsub (for mpiblast). My problem is that the perl code don't wait to qsub finish. Did someone have a idea about that problem ?

$command = ( "qsub mpiblast_$VAR.sh" ); system($command);# I would like to wait for qsub process and then use the output to parse the result

Thans for your help !

[Vivek]

Did you check the options to add dependency to the job with qsub?

[SES]

system() just returns the status of the command (1 for success). You may want to capture the output using backticks, or using somthing like Capture::Tiny, and then do something with the output. By the way, you might get more responses on Stack Overflow since this is not really a bioinformatics question. This answer seems relevant if you have multiple jobs to run.

Answer 1

[Stefano Berri]

I don't think this is the right approach.

The strength of a queing system is that you dispatch the jobs and let them run when the queing system allocates you resources. you don't want your perl script to wait for the job to finish. If you want to analyse the results of that job, you should prepare and run another job that DEPENDS on the previous one. Let's assume you have two jobs: blast and parse. You run the job blast with a particular name (on my system option -N) and then you run the parse job with option that waits for completion of the blast job (on my system option -hold_jid)

something like

qsub -N blastJob blast.q.sh
qsub -hold_jid blastJob parse.q.sh

you can, of course, put those options in the file itself

[Giovanni M Dall'Olio]

just a short note: the system you are using is Sun Grid Engine. For other queue management software (e.g. Torque) the option may be different.

[Mchimich]

Thanks for yoyr reply ! I undertrand you Steano. But the parsing job is included in perl script (Bio perl) and it's just a small part of the script.

Answer 2

[Fwip]

Qsub doesn't block, does it? From my understanding, it submits the job and immediately returns, without waiting for the job to complete.

This link: http://stackoverflow.com/questions/5982857/making-qsub-block-until-job-is-done mentions the following workaround:

qsub -I -x myscript.sh

If the script writes to standard out, you can capture the output by using backticks ( ` ` ) instead of the system() call.

my $output = `qsub -I -x myscript.sh`

Edit: There is also the -sync y option, which from the qsub man page,

-sync y causes qsub to wait for the job to complete before exiting. If the job completes successfully, qsub's exit code will be that of the completed job. If the job fails to complete successfully, qsub will print out a error message indicating why the job failed and will have an exit code of 1.

Which does not seem to provide output capturing, but may be more suitable (especially due to the issue mentioned below by Jeremy).

[Jeremy Leipzig]

beware your sysadmin might be unhappy with qsub -I since it launches an interactive session and basically locks out that node

[Fwip]

Thanks, I didn't know this. It's been quite a while since I've worked with qsub.

Answer 3

[Sirus]

One we have implemented this problem but we did it in a different way. You can lunch your job as you did in you code and then create a function that roughly do as follow (I didn't test it , if it doesn't work just try to play arround with the code)

sub CheckQsub{

         my $job_name = shift;
         my $stop = 1;

         while($stop){
                $command =  "qstat";
                my $status=qx[$command];
                my @jobsStatus=split(/\n/,$status);
                my @targetJobStatus  = grep(/$job_name/,@jobsStatus);  #Here we get only the status of the of interest

               if(scalar(@targetJobStatus )== 0) #if no job is running
               {
                     $stop=0;
               }
           sleep(10);
         }
        }

Roughly this is the code as you can notice we just check if the action finished we don't know if it there was an error or a normal finish you can addition check points after calling this function. hope that helps.

[Stefano Berri]

Rather than checking qsub, I would produce and check a log or result file.

[Sirus]

Yeah; it is a good idea also. But some times some programs don't generate log results and it is hard to tell if they are finished or not from the log file. I think one can combine the two strategies. First check "qsub" when no job is there check the log file to see if every thing worked well. Checking the result file is ok, but if it has a huge size maybe it will be that fast.

[Stefano Berri]

But your queue job could, as last command do something like: echo "Done" > myJob.log