CWL to write into for writing lines one from one file to new file

Question

[a.james]

Dear All,

I have small Shell script to write the lines in a file to another file

writer.sh
#!/usr/bin/env bash
infile=$1
outfile=$2
while read line
do
        echo $line >> "$outfile"
done < "$infile"

I converted it for CWL as following, writer.cwl

cwlVersion: v1.0
class: CommandLineTool
baseCommand: [sh,writer.sh]


inputs:

 infile:
  type: File
  inputBinding:
   position: 1

 outfile:
  type: string
  inputBinding:
   position: 2

outputs:
 out_message:
  type: File
  outputBinding:
   glob: $(inputs.outfile)

And the YML writer.yml file looks as following,

infile:
 class: File
 path:  /user/home/hello.txt
outfile: monday.txt
   class: File
   path:  /user/home/tuesday.txt

When I run as following, cwltool writer.cwl writer.yml. it is throwing permanent fail error as,

[job writer.cwl] /tmp/tmpmht4svls$ sh \
    search.sh \
    /tmp/tmpv3evb_ic/stg1a74b4ca-15ba-4d28-9598-62b7604e8185/hello.txt \
    monday.txt
sh: writer.sh: No such file or directory
[job writer.cwl] Job error:
Error collecting output for parameter 'out_message':
writer.cw.cwl:18:2: Traceback (most recent call last):
writer.cw.cwl:18:2: 
writer.cw:18:2:   File "/cluster/home/aalva/.local/venvs/cwltool/lib/python3.6/site-packages/cwltool/command_line_tool.py", line 724, in collect_output
writer.cwl: 18:2:     raise WorkflowException("Did not find output file with glob pattern: '{}'".format(globpatterns))
writer.cwl:18:2: 
writer.cwl:18:2: cwltool.errors.WorkflowException: Did not find output file with glob pattern: '['monday.txt']'
writer.cwl:18:2: 



  writer.cwl:21:4: cwltool.errors.WorkflowException: Did not find output file with glob pattern: '['monday.txt']'
    [job writer.cwl] completed permanentFail
    [job writer.cwll] {}
    [job writer.cwll] Removing input staging directory /tmp/tmpv3evb_ic
    [job writer.cwl] Removing temporary directory /tmp/tmp8v9yxvvo
    {}
    Final process status is permanentFail

I am not understanding what is wrong here.Any suggestions would be appreciated

Answer 1

[Tom]

Hi fellow pipeline enthusiast,

Is the script named "writer.sh" or "search.sh"? In the error you posted, the shell seems to throw the error search.sh: No such file or directory. I guess it can't find the script, does nothing except throw the error message and CWL complains because no output file is being generated.

Is the script stored in a location that is referenced in your $PATH? A dirty but quick fix would be to just pass your script as an argument in inputs.

Regards, Tom

[a.james]

Hey, Sorry, it is an error , It should be writer.sh. Is the script stored in a location that is referenced in your $PATH? Yes, it is store the place where it is being invoked. A dirty but quick fix would be to just pass your script as an argument in inputs. >> I did not quite understand what you mean?

You mean without outputs ? or by passing something like

   arguments:
         valueFrom: $(runtime.outdir)/$(inputs.outfile) ?

[Tom]

The place were you invoke is most likely not referenced in the $PATH environment variable. This means if you call sh writer.sh then writer.sh has to be in the directory you are calling sh from. However, CWL creates a temporary directory (in your example it was /tmp/tmpmht4svls ) where it copies all your input data (or symlinks to the data) and then tries to run what you have specified in baseCommand. In this temporary directory, writer.sh is not present, so sh can't find it. If you pass the script as one of your input files, then CWL will know to copy it into the temporary directory during runtime and sh will find it.

.cwl-description:

cwlVersion: v1.0
class: CommandLineTool
baseCommand: sh

inputs:
  script:
    type: File
    inputBinding:
      position: 0    
  infile:
    type: File
    inputBinding:
     position: 1
  outfile:
    type: string
    inputBinding:
     position: 2
outputs:
  out_message:
    type: File
    outputBinding:
      glob: $(inputs.outfile)

.yml-file:

script:
  class: File
  path: /WHEREVER/THE/SCRIPT/LIVES/writer.sh
infile:
  class: File
  path:  /user/home/hello.txt
outfile: monday.txt
  class: File
  path:  /user/home/tuesday.txt

[a.james]

Thanks Tom, this is working. I tried it to on scatter workflow with array of file and string as input. It got finished successfully