Add an item from input to an output array in CWL

Question

[drkennetz]

Hi all, I have a cwl workflow where step1 outputs an array of items file. In step 2 I would like to add a file or an array of files to the array that is output from step1. Here is my example workflow:

cwlVersion: v1.0    
class: Workflow    
requirements:
 - class: ScatterFeatureRequirement
 - class: StepInputExpressionRequirement
 - class: InlineJavascriptRequirement

inputs:
  file1: File[]
  file2: File[]
  input_added_in_step2: File
  string_input: string_input
steps:
  step1:
    run: step1.cwl
    scatter: [file1, file2]
    scatterMethod: dotproduct
    in:
      file1: file1
      file2: file2
    out:
      [output_array]
  step2:
    run: step2.cwl
    in:
      input_array:
        source: step1/output_array
        valueFrom: $(self.concat(inputs.input_added_in_step2))
      string_input: string_input
    out: [output_array_and_file_added_and_rename]
outputs:
  output_array_plus_file:
    type: File[]
    outputSource: step2/output_array_and_file_added_and_rename

The biggest issue here is that the workflow actually runs to completion, but it doesn't ever add input_added_in_step2 to the array created by step1. It just returns the array created by step1 with the string addition.

Thanks

Answer 1

[Tom]

In the valueFrom: $(self.concat(inputs.input_added_in_step2)) expression, inputs will reference the inputs of the workflow step, not the input section at the beginning of the workflow document. Looking at the valueFrom section in this segment of the specifications leads me to believe there is no way to reference input_added_in_step2 in the StepInputExpression. You will need to define the file as input for step2 or create a separate step (presumably an expressionTool) with the necessary inputs to prepare the array for step2.

A similar problem was discussed here, maybe the examples can help: Combine subject with sample type in a scatter?

Answer 2

[Michael R. Crusoe]

You can combine multiple inbound inputs using https://www.commonwl.org/v1.0/Workflow.html#MultipleInputFeatureRequirement

Try this:

cwlVersion: v1.0    
class: Workflow    
requirements:
 - class: ScatterFeatureRequirement
 - class: StepInputExpressionRequirement
 - class: MultipleInputFeatureRequirement

inputs:
  file1: File[]
  file2: File[]
  input_added_in_step2: File
  string_input: string_input
steps:
  step1:
    run: step1.cwl
    scatter: [file1, file2]
    scatterMethod: dotproduct
    in:
      file1: file1
      file2: file2
    out:
      [output_array]
  step2:
    run: step2.cwl
    in:
      input_array:
        source: [ step1/output_array,  input_added_in_step2 ]
        linkMerge: merge_flattened
      string_input: string_input
    out: [output_array_and_file_added_and_rename]
outputs:
  output_array_plus_file:
    type: File[]
    outputSource: step2/output_array_and_file_added_and_rename

[Tom]

This is great! I didn't know about linkMerge, wish i would have learned this sooner.

[Michael R. Crusoe]

Thanks Tom, maybe you could help us write a section in the CWL user guide about this?

https://github.com/common-workflow-language/user_guide/issues/103

You don't have to make a perfect lesson, even just writing what you can is helpful; we'll clean up and format as necessary :-)

[Tom]

I would be very happy to do so. Since i have to submit my master's thesis this month i'm a little stressed out and won't be able to take care of it right now. I have made a calendar entry to make sure i won't forget.

I spent a lot of time reading and writing CWL documents in the past months. A more in-depth user guide would have saved me some some long hours of being confused. It's great that you are expanding it!

*edit: Learning about linkMerge has just allowed me to purge an abhorrent ExpressionTool from one of my workflows, which makes me very happy :)

[Michael R. Crusoe]

Congratulations on your pending thesis! Yes, no rush. Any contribution is appreciated!

[drkennetz]

The merge_flattened is actually something I found on another one of your posts that ended up solving my problems:

CWL: combine two arrays as workflow step input

Super helpful for me!