CWL: Getting the outputs(File) of a command line tool

Question

[kevin.o.oluoch]

I've executed the following Command Line Tool.

#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: CommandLineTool

baseCommand: spades.py
requirements:
  - class: ShellCommandRequirement

inputs:

  SAMPLENAME: string
  OUTPUT_DIR: string

  FORWARD: 
    type: string
    inputBinding:
      position: 1
      prefix: '-1'

  REVERSE: 
    type: string
    inputBinding:
      position: 2
      prefix: '-2'

  THREADS: 
    type: int
    inputBinding:
      position: 4
      prefix: "--threads"
      shellQuote: false

outputs:
  OUTPUT_FILE:
    type: File
    outputBinding:
      glob: "contigs.fasta"

arguments:
  - valueFrom: $(inputs.OUTPUT_DIR)/$(inputs.SAMPLENAME) 
    position: 3
    prefix: "-o "
    shellQuote: false

The command on the terminal is:

./spadesTool.cwl  --SAMPLENAME="11866_1#70" --FORWARD="/media/kevin/2251479E904444F7/data/11866_1#70_1.fastq.gz" --REVERSE="/media/kevin/2251479E904444F7/data/11866_1#70_2.fastq.gz" --OUTPUT_DIR="/media/kevin/2251479E904444F7/Output_Directory" --THREADS=4

The baseCommand (spades.py) executes successfully and produces results. However CWL produces the following error

[ job spadesTool.cwl ] Job error :
spadesTool.cwl:38:3: Error collecting output for parameter "OUTPUT_FILE" :
spadesTool.cwl:41:7: Did not find output file with glob pattern: '[' contigs.fasta ']' 
[ job spadesTool.cwl ] completed permanentFail
{}
Final process status is permanentFail

What could be the problem?

[BioinfGuru]

Why not just copy it to another directory?

$cd Desktop

$cp OUTPUT_DIR/contigs.fasta .

[kevin.o.oluoch]

The file and its parent directory will be created by the tool. I need the file to be capture as output of the Command Line TOOL, using the variable OUTPUT_FILE

[BioinfGuru]

Post the command you are using please

It may be as simple as using a pipe.

[kevin.o.oluoch]

./spadesTool.cwl  --SAMPLENAME="11866_1#70" --FORWARD="/media/kevin/2251479E904444F7/data/11866_1#70_1.fastq.gz" --REVERSE="/media/kevin/2251479E904444F7/data/11866_1#70_2.fastq.gz" --OUTPUT_DIR="/media/kevin/2251479E904444F7/Output_Directory" --THREADS=4

I'll do this for several samples, and I want the results as sub-directories, in the output directory.

[BioinfGuru]

Well now you mention sub-directories...If you are just running this in the command line then I suggest a for loop. Either in the command line or in a script. You would need to store a list of sample names for iterating through. then on each iteration create a new folder (named same as sample name)... use the folder as the output folder

something like this in command line should work (excluding minor synthax errors):

for i in SAMPLE_LIST; do mkdir $i; ./spadesTool.cwl  --SAMPLENAME=$i --FORWARD="/media/kevin/2251479E904444F7/data/$i_1.fastq.gz" --REVERSE="/media/kevin/2251479E904444F7/data/$i_2.fastq.gz" --OUTPUT_DIR=">/media/kevin/2251479E904444F7/$i" --THREADS=4; done

If you follow the idea then a folder will be created for each sample containing only outfiles for that sample.

[kevin.o.oluoch]

This does not solve my original issue. Kindly see the change I've made to the question

[BioinfGuru]

Ok this is starting to sound like a job for TEE

sorry I can't be of more help

Yes Ram thank you :)

[Ram]

makedir? Do you mean mkdir?

Answer 1

[Michael R. Crusoe]

Hello kevin.o.oluoch,

Thank you for your question.

In CWL it is incorrect to pass a path to a local directory as a string for a variety of reasons. In this instance you are trying to communicate to spades where to write its output -- since CWL tools could be executed remotely it would be simpler to use the empty working directory one is guaranteed to have available by the CWL spec:

arguments:
  - valueFrom: $(runtime.outdir)
    prefix: -o

FYI: You may be interested in the CWL description I made for SPAdes operating in metaSPAdes mode:

https://github.com/ProteinsWebTeam/ebi-metagenomics-cwl/blob/master/tools/metaspades.cwl

Feel free to copy and modify it under the terms of the Apache 2 license.

[kevin.o.oluoch]

Thanks, This is very Helpful