CWL: Loops and if conditionals

Question

[kevin.o.oluoch]

Does the Common Workflow Language have if statements and for or while loops? If it does, what is the syntax?

Answer 1

[Michael R. Crusoe]

It does not. This feature is under active discussion at https://groups.google.com/d/topic/common-workflow-language/JU7PSEKr97M/discussion

However, if you have several files that need to be processed (or an array that you want to iterate over for one or more other parameters) then you can use the scatter/gather feature http://www.commonwl.org/v1.0/Workflow.html#WorkflowStep

[unique_g]

So just to confirm: There is currently no way of implementing a single workflow that, say, dynamically skips a step such as in the following example?

[Michael R. Crusoe]

Good news: this is easily done if you know that there will always be some data that needs preprocessing and some that doesn't!

The "Pre-processing required?" step would need output the files in two segments: one to be pre-processed and one without. Javascript via and ExpressionTool can be used to avoid copying files around.

If sometime all data will either need or not-need pre-processing then it gets a little more complicated. In theory it might work if the various inputs and outputs were marked optional (a.k.a accepting null as a valid type) -- but I haven't personally tested that.

The situation that CWL v1.0 doesn't at all support would be two or more divergent paths where the overall outputs would be radically different.

[unique_g]

I'm sorry, but it seems that I didn't explain my question well. In fact, the whole data will indeed either need pre-processing or not need pre-processing (in which case the whole pre-processing CommandLineTool should ideally be entirely skipped).

Moreover, "Pre-processing required?" wasn't actually supposed to be a CommandLineTool, but the logic that decides whether pre-processing should occur or not, based on the output of the "Inspect data" component (a string or file containing a string as suggested by Stefano). I am now realizing that the flowchart is unclear on this because I have used the same arrows both for decisions and data flow. What I meant is that the data should actually flow from either "Data" or "Inspect data" (which one is irrelevant in this example) to either "Pre-process data" or "Process data" depending on the decision, but not to both.

Hope it is clearer now.

[Stefano Berri]

I am a beginner with cwl, but here my approach to conditional events. Make the workflow linear, like this:

Inspect data | Preprocess-Required? | Pre-process data | Process Data

L'et's say that Preprocess-Required spit out a file with "YES" or "NO" Then preprocess data read that file and spit out either the same input files (if previous step said "NO") or the preprocessed files. That, of course, requires that you have some control over the Pre-process data step and can add the ability to take this decision. Basically, your "if logic" need to get into the step.

[unique_g]

Thanks a lot for the suggestion.

However, I would like to avoid wrapping scripts/tools or even refactoring them (assuming I have control over the code) just to meet such a specific purpose. It would be kinda weird to end up having a flag --execute or --pass-through in all future scripts, no? :-) Also, it would pretty much defeat the purpose of having a modular workflow framework in my opinion. By the way, I would feel similarly uneasy about tampering with CommandLineTools in a way suggested by Michael, i.e. setting all input/output ports optional, if they're actually not meant to be optional. A global, CWL-supported --pass-through option on the level of Workflows would be something different of course.

Anyway, my current workaround is to split up the workflow and outsource the logic to a global wrapper script (or web service or whatever) that takes care of coordinating sub-workflows. At least for me that seems to be the least invasive option.

[kevin.o.oluoch]

Thanks, I hope the feature will be up soon. I'll be following the discussion

[steve]

I think this has been implemented ( https://www.commonwl.org/v1.2/Workflow.html#Conditional_execution_(Optional) ) but I am having trouble finding solid official docs on examples of implementing all the features of it.