Upgrade to Toil 3.18 no longer allows optional parameters to be omitted when passed through subworkflow

Question

[ionox0]

I've upgraded from 3.15 to 3.18 and found that previously it was possible to have an optional argument, that when left blank by the parent workflow, would run the CommandLineTool with the argument supplied as null:

inputs:
  fci__basq_fix: string?
...
steps:
  run: tool.cwl
  step_one:
    input_one: fci__basq_fix             # <-- An optional parameter for tool.cwl

but in 3.18 it appears that the workflow will error when fci__basq_fix is not supplied by the parent workflow:

WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CWLWorkflow' Y/m/jobvXFcXW
WARNING:toil.leader:Y/m/jobvXFcXW    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
WARNING:toil.leader:Y/m/jobvXFcXW    INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be-dirty.
WARNING:toil.leader:Y/m/jobvXFcXW    WARNING:toil.resource:Can't globalize module ModuleDescriptor(dirPath='/home/johnsoni/virtualenvs/pipeline_0.1.0-rc1/lib/python2.7/site-packages/toil_ionox0-0.0.4-py2.7.egg', name='toil.cwl.cwltoil', fromVirtualEnv=True).
WARNING:toil.leader:Y/m/jobvXFcXW    Traceback (most recent call last):
WARNING:toil.leader:Y/m/jobvXFcXW      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-rc1/lib/python2.7/site-packages/toil_ionox0-0.0.4-py2.7.egg/toil/worker.py", line 314, in workerScript
WARNING:toil.leader:Y/m/jobvXFcXW        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
WARNING:toil.leader:Y/m/jobvXFcXW      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-rc1/lib/python2.7/site-packages/toil_ionox0-0.0.4-py2.7.egg/toil/job.py", line 1351, in _runner
WARNING:toil.leader:Y/m/jobvXFcXW        returnValues = self._run(jobGraph, fileStore)
WARNING:toil.leader:Y/m/jobvXFcXW      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-rc1/lib/python2.7/site-packages/toil_ionox0-0.0.4-py2.7.egg/toil/job.py", line 1296, in _run
WARNING:toil.leader:Y/m/jobvXFcXW        return self.run(fileStore)
WARNING:toil.leader:Y/m/jobvXFcXW      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-rc1/lib/python2.7/site-packages/toil_ionox0-0.0.4-py2.7.egg/toil/cwl/cwltoil.py", line 878, in run
WARNING:toil.leader:Y/m/jobvXFcXW        jobobj[key][0]] is None:
WARNING:toil.leader:Y/m/jobvXFcXW    KeyError: u'fci__basq_fix'

Just wanted to confirm that this is indeed the case. Is it no longer possible to omit an input parameter using the ? syntax?

-

In other words, perhaps there is a difference between omitting an optional input parameter ?, and specifying it as null. For example, for the following tool, which of the following should be considered valid:

The tool:

inputs:
  optional_input_param: string?
  another_param: string

Specifying input_param as null

run: tool.cwl
  in: 
    input_param:
      valueFrom: $(null)
    another_param: 'dummy'

or, omitting the input_param entirely

run: tool.cwl
  in: 
    another_param: 'dummy'

Thank you!

[i.sudbery]

Can you please clarify which tool it is you have upgraded from 3.15 to 3.18?

[ionox0]

Sorry, to be clear that tool is Toil

Answer 1

[ionox0]

Update:

I have found that this error is dependent on the tool.cwl having a default for the optional parameter. I've included a minimal reproducible example here:

https://github.com/DataBiosphere/toil/issues/2512

I believe it is related to the combination of a subworkflow with an optional tool paramter that is not supplied.