CWL: Using the "EnvVarRequirement" to pass an environment variable to a script

Question

[skanwal]

Hello,

I am running a bash script inside a docker container that expects DEPLOY_ENV variable to be set to an appropriate value. However, when running cwltool ./runFolderCheck.cwl ./runFolderCheck-job.yml, I get:

INFO /Users/kanwals/miniconda3/envs/cwl/bin/cwltool 1.0.20190915164430
INFO Resolved './runFolderCheck.cwl' to 'file:///Users/kanwals/Documents/UMCCR/git/bcl2fastq/tools/runFolderCheck.cwl'
ERROR Got workflow error Traceback (most recent call last):
   File "/Users/kanwals/miniconda3/envs/cwl/lib/python3.6/site-packages/cwltool/executors.py", line 169, in run_jobs
    for job in jobiter:   File "/Users/kanwals/miniconda3/envs/cwl/lib/python3.6/site-packages/cwltool/command_line_tool.py", line 506, in job
    self.updatePathmap(builder.outdir, builder.pathmapper, l)
   File "/Users/kanwals/miniconda3/envs/cwl/lib/python3.6/site-packages/cwltool/command_line_tool.py", line 295, in updatePathmap
    for sf in fn.get("secondaryFiles", []): AttributeError: 'str' object has no attribute 'get'
ERROR Workflow error, try again with --debug for more information: 'str' object has no attribute 'get'

And with the --debug flag:

INFO /Users/kanwals/miniconda3/envs/cwl/bin/cwltool 1.0.20190915164430
INFO Resolved './runFolderCheck.cwl' to 'file:///Users/kanwals/Documents/UMCCR/git/bcl2fastq/tools/runFolderCheck.cwl'
usage: ./runFolderCheck.cwl [-h] --denv DENV [--input_folder INPUT_FOLDER]
                            [job_order] ./runFolderCheck.cwl: error: the following arguments are required: --denv

The cwl-tool definition runFolderCheck.cwl, I am using is:

#!/usr/bin/env cwl-runner

cwlVersion: v1.0
class: CommandLineTool

hints:
 DockerRequirement:
   dockerImageId: umccr/pipeline

requirements:
  EnvVarRequirement:
    envDef:
      DEPLOY_ENV: $(inputs.denv)
  InitialWorkDirRequirement:
    listing: $(inputs.input_folder)

inputs:    
  denv: string

  input_folder:
    type: string?
    inputBinding:
      position: 1

outputs:
  log_out:
    type: stdout

stdout: output/output_runFOlderCheck.txt

baseCommand: [bash, '-c', /scripts/runfolder-check.sh]

And the input yaml runFolderCheck-job.yml is:

denv: dev

input_folder: /Users/kanwals/Documents/UMCCR/data/bcl2fastq

Any help to solve the issue will be appreciated.

Sehrish K

Answer 1

[Tom]

Hi Sehrish!

There seems to be an issue with the InitialWorkDirRequirement. You cannot pass a string to the listings field. It has to be either files, directories or dirents. We can, however, build a dirent for an empty directory named after the input_folder string:

requirements:
  EnvVarRequirement:
    envDef:
      DEPLOY_ENV: $(inputs.denv)
  InlineJavascriptRequirement: {}
  InitialWorkDirRequirement:
    listing:
      - entry: "$({class: 'Directory', listing: []})"
        entryname: $(inputs.input_folder)
        writable: true

I can't do any testing because i do not have the docker container you are using, but this should fix the error you are describing.

Best regards,

Tom

Answer 2

[skanwal]

Hi Tom,

Thanks for the response. Changing the tool definition according to your suggestion is raising the following issue:

INFO /Users/kanwals/miniconda3/envs/cwl/bin/cwltool 1.0.20190915164430 INFO Resolved './runFolderCheck.cwl' to 'file:///Users/kanwals/Documents/UMCCR/git/bcl2fastq/tools/runFolderCheck.cwl' ERROR Workflow error, try again with --debug for more information: No mandatory DockerRequirement, yet path is outside the designated output directory, also know as $(runtime.outdir): MapperEnt(resolved='_:23a2471b-ce8d-4d95-84c6-27b195f675fc', target='/Users/kanwals/Documents/UMCCR/data/bcl2fastq', type='WritableDirectory', staged=True)

However, trying:

#!/usr/bin/env cwl-runner

cwlVersion: v1.0
class: CommandLineTool

hints:
 DockerRequirement:
   dockerImageId: umccr/pipeline

requirements:
  EnvVarRequirement:
    envDef:
      DEPLOY_ENV: $(inputs.denv)
  InitialWorkDirRequirement:
    listing: $(inputs.input_folder.listing)

inputs:    
  denv: string

  input_folder:
    type: Directory
    inputBinding:
      position: 1

outputs:
  log_out:
    type: stdout

stdout: scripts/runfolder-check.sh.dev.log

baseCommand: [bash, '-c', /scripts/runfolder-check.sh]

Seems to be progressing a little more and I get:

$ cwltool ./runFolderCheck.cwl ./runFolderCheck-job.yml
INFO /Users/kanwals/miniconda3/envs/cwl/bin/cwltool 1.0.20190915164430
INFO Resolved './runFolderCheck.cwl' to 'file:///Users/kanwals/Documents/UMCCR/git/bcl2fastq/tools/runFolderCheck.cwl'
INFO [job runFolderCheck.cwl] /private/tmp/docker_tmp3ifase1d$ docker \
    run \
    -i \
    --volume=/private/tmp/docker_tmp3ifase1d:/XCMkDM:rw \
    --volume=/private/tmp/docker_tmpo87g08oy:/tmp:rw \
    --volume=/Users/kanwals/Documents/UMCCR/data/bcl2fastq:/var/lib/cwl/stgf816efbf-5870-4786-bc21-856ea92e2ddd/bcl2fastq:ro \
    --volume=/Users/kanwals/Documents/UMCCR/data/bcl2fastq/RTAComplete.txt:/XCMkDM/RTAComplete.txt:ro \
    --volume=/Users/kanwals/Documents/UMCCR/data/bcl2fastq/190704_A00130_0109_BHLHLWDSXX:/XCMkDM/190704_A00130_0109_BHLHLWDSXX:ro \
    --volume=/Users/kanwals/Documents/UMCCR/data/bcl2fastq/SequenceComplete.txt:/XCMkDM/SequenceComplete.txt:ro \
    --volume=/Users/kanwals/Documents/UMCCR/data/bcl2fastq/SampleSheet.csv.custom.1.truseq:/XCMkDM/SampleSheet.csv.custom.1.truseq:ro \
    --volume=/Users/kanwals/Documents/UMCCR/data/bcl2fastq/output:/XCMkDM/output:ro \
    --volume=/Users/kanwals/Documents/UMCCR/data/bcl2fastq/CopyComplete.txt:/XCMkDM/CopyComplete.txt:ro \
    --volume=/Users/kanwals/Documents/UMCCR/data/bcl2fastq/SampleSheet.csv.custom.2.10X:/XCMkDM/SampleSheet.csv.custom.2.10X:ro \
    --volume=/Users/kanwals/Documents/UMCCR/data/bcl2fastq/SampleSheet.csv:/XCMkDM/SampleSheet.csv:ro \
    --workdir=/XCMkDM \
    --read-only=true \
    --log-driver=none \
    --user=1457398319:2094513965 \
    --rm \
    --env=TMPDIR=/tmp \
    --env=HOME=/XCMkDM \
    --cidfile=/private/tmp/docker_tmp5erahzgb/20191007223553-774195.cid \
    --env=DEPLOY_ENV=dev \
    umccr/pipeline \
    bash \
    -c \
    /scripts/runfolder-check.sh \
    /var/lib/cwl/stgf816efbf-5870-4786-bc21-856ea92e2ddd/bcl2fastq > /private/tmp/docker_tmp3ifase1d/scripts/runfolder-check.sh.dev.log
/scripts/runfolder-check.sh: line 31: /scripts/runfolder-check.sh.dev.log: Read-only file system
INFO [job runFolderCheck.cwl] Max memory used: 0MiB
WARNING [job runFolderCheck.cwl] completed permanentFail
{
    "log_out": {
        "location": "file:///Users/kanwals/Documents/UMCCR/git/bcl2fastq/tools/runfolder-check.sh.dev.log",
        "basename": "runfolder-check.sh.dev.log",
        "class": "File",
        "checksum": "sha1$dce702ab890b6b1974157b1437d75482a2fea8cd",
        "size": 85,
        "path": "/Users/kanwals/Documents/UMCCR/git/bcl2fastq/tools/runfolder-check.sh.dev.log"
    }
}
WARNING Final process status is permanentFail

Any idea why it could be failing now?

[Michael R. Crusoe]

/scripts/runfolder-check.sh: line 31: /scripts/runfolder-check.sh.dev.log: Read-only file system

The script is trying to write a log file in its directory. It should either use $TMPDIR, or write the log to a user specified path.

[skanwal]

Hi Michael,

Thanks a lot for the response. I have edited the bash script to write to the standard output and using stdout in the cwl definition to write it to a file. This works as expected.

---

Sehrish K