Converting stdout (File) to string

Question

[eduardo]

I am constructing a workflow that contains two steps:

1. the output of the first step is stdout (compatible, I believe, with File type)
2. the input of the second is a string

I am not able to modify these steps. How can I connect these two steps? Do I have to make an adaptor step?

[cpad0112]

Please post your example code and expected output

[eduardo]

Output from first step:

outputs:
    accessions:
        type: stdout

Input from second step:

inputs:
    sequences:
        type: string

[WouterDeCoster]

Besides the tag it's unclear that this is about cwl - would be good to mention that. The question doesn't make sense if you don't read the tags.

Answer 1

[Michael R. Crusoe]

Here's the ExpressionTool version:

$ cat u.cwl 

#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: ExpressionTool

requirements: { InlineJavascriptRequirement: {} }

inputs:
  a_File:
    type: File
    inputBinding:
      loadContents: true

expression: |
  ${ return { "a_string": inputs.a_File.contents }; }

outputs:
  a_string: string

And it's output

$ echo "Hello, World!" > message
$ cwltool u.cwl --a_File message 
/home/michael/cwltool/env/bin/cwltool 1.0.20171107133715
Resolved 'u.cwl' to 'file:///home/michael/u.cwl'
{
    "a_string": "Hello, World!\n"
}
Final process status is success

[eduardo]

This solution also has the 64kb limitation, correct?

[Michael R. Crusoe]

Yes, thank you for noticing that. The only way to get around the 64kb limit is with a dedicated CommandLineTool.

[Michael R. Crusoe]

Which I've added an example at A: Converting stdout (File) to string

Answer 2

[Michael R. Crusoe]

And a third way, that converts any size of File that will fit into system memory into a CWL string. This is achieved using the JSON feature from http://www.commonwl.org/v1.0/CommandLineTool.html#Output_binding

$ echo 'Hello,
World!' > message

Our input file

$ cat v.cwl 
#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: CommandLineTool

inputs:
  a_File: File

baseCommand: python
arguments:
 - prefix: -c
   valueFrom: |
     import json
     a_string = ""
     with open("$(inputs.a_File.path)", "r") as a_File:
         a_string = a_File.read()
     with open("cwl.output.json", "w") as output:
         json.dump({"a_string": a_string}, output)

outputs:
  a_string: string

The CWL CommandLineTool itself

$ cwltool ../v.cwl --a_File message
/home/michael/common-workflow-language/env/bin/cwltool 1.0.20171107133715
Resolved 'v.cwl' to 'file:///home/michael/v.cwl'
[job v.cwl] /tmp/tmp4YLbgd$ python \
    -c \
    'import json
import os
a_string = ""
with open("/tmp/tmp3xHM90/stgdc883f3a-b4e7-4eab-aa42-3cddb08a822b/message", "r") as a_File:
    a_string = a_File.read()
with open("cwl.output.json", "w") as output:
    json.dump({"a_string": a_string}, output)'
[job v.cwl] completed success
{
    "a_string": "Hello,\nWorld!\n"
}
Final process status is success

And the CWL File to CWL string converter in action. As you can see, this reads the entire file into memory.

Answer 3

[Michael R. Crusoe]

Hello eduardo,

If the stdout is less than 64kb then you can express it as a string either during the gathering of the outputs.

#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: CommandLineTool

inputs:
  message: string

baseCommand: echo

arguments: [ $(inputs.message) ]

stdout: message

outputs:
  message_File:  # Just to illustrate that you can process a file multiple times, not necessary for message_string to work
    type: File
    outputBinding:
      glob: message

  message_string:
    type: string
    outputBinding:
      glob: message
      loadContents: true
      outputEval: $(self[0].contents)

And the output:

$ cwltool ../t.cwl --message "Hello, world!"
/home/michael/cwltool/env/bin/cwltool 1.0.20171107133715
Resolved '../t.cwl' to 'file:///home/michael/t.cwl'
[job t.cwl] /tmp/tmpAljfTQ$ echo \
    'Hello, world!' > /tmp/tmpAljfTQ/message
[job t.cwl] completed success
{
    "message_string": "Hello, world!\n", 
    "message_File": {
        "checksum": "sha1$09fac8dbfd27bd9b4d23a00eb648aa751789536d", 
        "basename": "message", 
        "location": "file:///home/michael/u/message", 
        "path": "/home/michael/u/message", 
        "class": "File", 
        "size": 14
    }
}
Final process status is success

If you don't want to change your tool definition, then you can use an ExpressionTool to convert the File to a string.

[eduardo]

Thanks! That's what I was looking for.