cwltool SGE support for CWL parallel tasks

Question

[Stefano Berri]

I am experimenting with CWL and cwltool. I have successfully written a workflow that can scatter jobs. However, cwltool runs them sequencially, whereas I would like to to use the Sun Grid Engine (SGE) to run jobs in parallel. I looked into Toil to run it but I have found it not capable of running the workflow so far. Here my questions:

• Does cwltool support integration with SGE?
• If not, is it planned, and if not, why? Would patches be accepted?
• Beside Toil, are there working implementation that do interact with SGE?
• What tools do you use to run cwl workflows with parrallel tasks?

Any other suggestion is welcome.

Thank you

Answer 1

[sayerskt]

CWL tool I believe is always planned to just be local runner, and Toil is the only implementation currently with cluster support I believe. Can you post the workflow?

[Stefano Berri]

I am getting this message: "CWL Directory inputs not yet supported in Toil"

[Michael R. Crusoe]

That is correct, Toil's CWL support for v1.0 is ongoing: https://toil.readthedocs.io/en/releases-3.6.x/running/running.html#running-cwl-workflows