Calling Qiime scripts sequencially in Python 2.7 program

Question

[uxue33]

Hello all,

I'm beginning to learn Python 2.7 and need to develop some applications for a lab project.

I want to run an external program and then use its output as input for another external call in the same script. The issue is that the script tries to call the second program even before the first one has finished working, so the output is not yet available to be used as its input. How can I solve this problem?

Here is my code:

picker = Popen ([pick_open_reference_otus.py, -i, allseq_file, -r, refseq_file, -o, noName, -s, 0.1, -m, usearch61, -p, param_file ], shell = True)
picker_out = os.path.join(path, "otu_output/otu_table_mc2_w_tax.biom")

Thank you all!

[skbrimer]

I'm not 100% sure because I have not done any work with the subprocesses module, however I think you can just use the wait() module. something like picker.wait(picker_out) or something to that effect.

[skbrimer]

or picker_out = picker.wait(os.path.join(path, "otu_output/otu_table_mc2_w_tax.biom") sorry. I'm not sure about the syntax

[skbrimer]

last one :)

picker = Popen ([pick_open_reference_otus.py, -i, allseq_file, -r, refseq_file, -o, noName, -s, 0.1, -m, usearch61, -p, param_file ], shell = True)
picker.wait()
picker_out = os.path.join(path, "otu_output/otu_table_mc2_w_tax.biom")

[uxue33]

Thank you a lot for your help!

[pld]

https://docs.python.org/2/library/subprocess.html

Subprocess has what you want.

Python can be rather clunky for this kind of thing. If you need data from some of these programs you're calling to end up being processed in python, it is better to pull the data into python from the "outside". You may want to explore other options for managing pipelines, there are plenty of posts and tutorials on here about it.

[uxue33]

Thank you for the link!

Answer 1

[WouterDeCoster]

I used the following code for similar jobs:

retmode = subprocess.call(shlex.split('your shell command with arguments'))
if retmode > 0:
    sys.exit("Prinseq returned error code {} for sample {}".format(retmode, target))

I find the shlex module useful to structure command prior to using subprocess.call, it allows you to just write the string of the command and it will take care of the formatting in a list.

[uxue33]

Than you a lot for your code!

[JL]

Dear all,

I found this post very useful and tried to use the solution myself. The thing is that I presume I get an error in the variable interpolation process that I can't figure out how to fix (I haven't used Python very often for this kind of task, because I usually use Perl for external program related pipelines). Here's my code and the error it produces.

import sys, os
import subprocess
import shlex

allseq_file = "/home/jllavin/Metagenomics/Test.fasta"
refseq_file = "/home/jllavin/Metagenomics/Dbs/99_otus.fasta"

picker = subprocess.call(shlex.split('pick_open_reference_otus.py -i allseq_file -r refseq_file -o output2 -s "0.1" -m "usearch61" -p parameters.txt -f'))
    if picker > 0:
    sys.exit[(picker)]

The error:

Traceback (most recent call last):
  File "/usr/lib64/python2.7/lib-tk/Tkinter.py", line 1470, in __call__
    return self.func(*args)
  File "QIIME_shlex.py", line 45, in button_ex_callback
    sys.exit[(picker)]
TypeError: 'builtin_function_or_method' object has no attribute '__getitem__'

[WouterDeCoster]

hi joluito,

for sure sys.exit[(picker)] is wrong. The square brackets don't serve a purpose here. Try sys.exit(picker). (This is just the return code of the script, in case something goes wrong)

for the rest, I don't think your -i allseq_file is going to work. Or is allseq_file an actual file in the current directory?

If it's a variable in your python code (what I suspect) you need to construct the string differently:

picker = subprocess.call(shlex.split('pick_open_reference_otus.py -i ' + allseq_file + ' -r ' + refseq_file + etc etc'))

As such the value of the variable is inserted in the string, fed to shlex.split() and used for the process.

Note the spaces!

It can be helpful to first construct your command:

cmd = 'pick_open_reference_otus.py -i ' + allseq_file + ' -r ' + refseq_file + etc etc'
print(cmd)
picker = subprocess.call(shlex.split(cmd))

and print that to see if everything is okay

[JL]

Sorry, I was adding the rest of the code while you answered

;)

[WouterDeCoster]

So as I suspected, you are passing the variable's name to pick_open_reference_otus.py, rather than the variable's value. Is that clear?

[JL]

Interesting point you raised here.

So, I'm not interpolating the variable's "string value" by passing its "name" to the command...

Therefore the way to proceed is to generate the command in a "print style" and then using it via subprocess.call(shlex.split)

Thank you very much for your patience, I will try it and come back with the outcome.

Best

[WouterDeCoster]

Yes, just concat the strings together with the variables and don't forget spaces.

[JL]

It worked smoothly.

Thank you very much for your kind support.

Best

JL