Why is ShellCommandRequirement necessary for Pipes

Question

[ionox0]

This is a general question regarding the ShellCommandRequirement feature.

Would anyone happen to be able to explain why ShellCommandRequirement is required to use Pipes in the CWL arguments section?

The pipes will work if I use ShellCommandRequirement and set ShellQuote to false for the pipes.

However, the following CWL syntax looks like it should run fine based on the command line that is generated in the cwltool logging, but for some reason it seems the pipe is not working.

Arguments section:

arguments:
- /opt/common/CentOS_6-dev/vardict/v1.5.1/bin/VarDict
- -E
- $(inputs.column_for_region_end)
- -G
- $(inputs.reference_fasta)
- -N
- $(inputs.tumor_sample_name)
- -c
- $(inputs.column_for_chromosome)
- -b
- $(inputs.tumor_bam.path + '|' + inputs.normal_bam.path)
- -g
- $(inputs.column_for_gene_name)
- -f
- $(inputs.allele_freq_thres)
- -S
- $(inputs.column_for_region_start)
- -r
- $(inputs.min_num_variant_reads)
- $(inputs.bed_file)
- '|'
- /opt/common/CentOS_6-dev/vardict/v1.5.1/vardict_328e00a/testsomatic.R
- '|'
- /opt/common/CentOS_6-dev/vardict/v1.5.1/vardict_328e00a/var2vcf_paired.pl
- -N
- $(inputs.tumor_sample_name + '|' + inputs.normal_sample_name)
- -f
- $(inputs.allele_freq_thres)

Resulting command from cwltool logging:

[job vardict] /scratch/tmpbKz3yH$ /opt/common/CentOS_6-dev/vardict/v1.5.1/bin/VarDict \
    -E \
    3 \
    -G \
    /scratch/tmpcR4isu/stg9c4f932e-84fb-4108-99a2-627fd7aceec5/Homo_sapiens_assembly19.fasta \
    -N \
    C-4PU567-L002-d \
    -c \
    1 \
    -b \
    '/scratch/tmpcR4isu/stgeb5ea926-d425-468c-871a-1d593ae22d49/C-4PU567-L002-d_cl_aln_srt_MD_IR_FX_BR__aln_srt_IR_FX-duplex.bam|/scratch/tmpcR4isu/stgb8386cf2-387e-41b4-b999-3d9c9d60c9c1/C-4PU567-N002-d_cl_aln_srt_MD_IR_FX_BR__aln_srt_IR_FX-duplex.bam' \
    -g \
    4 \
    -f \
    0.0002 \
    -S \
    2 \
    -r \
    1 \
    /scratch/tmpcR4isu/stg1e7697e8-7bd3-4e8e-a5fc-d5dea2b240f7/MSK-ACCESS-v1_0-probe-A.sorted.no_fp_snp.bed \
    | \
    /opt/common/CentOS_6-dev/vardict/v1.5.1/vardict_328e00a/testsomatic.R \
    | \
    /opt/common/CentOS_6-dev/vardict/v1.5.1/vardict_328e00a/var2vcf_paired.pl \
    -N \
    'C-4PU567-L002-d|C-4PU567-N002-d' \
    -f \
    0.0002 > /scratch/tmpbKz3yH/C-4PU567-L002-d.C-4PU567-N002-d.vardict.vcf

Answer 1

[ionox0]

Answered my own question after a bit of research:

ShellCommandRequirement in CWL allows us to use /bin/sh -c in order to pass our command as a string to a shell intrepreter. ShellQuote:false leaves shell metacharacters unquoted, and thus they can still be interpreted and used by a shell.

From CommandLineTool.job():

    shellcmd, _ = self.get_requirement("ShellCommandRequirement")
    if shellcmd is not None:
        cmd = []  # type: List[Text]
        for b in builder.bindings:
            arg = builder.generate_arg(b)
            if b.get("shellQuote", True):
                arg = [shellescape.quote(a) for a in aslist(arg)]
            cmd.extend(aslist(arg))
        j.command_line = ["/bin/sh", "-c", " ".join(cmd)]

Without the ShellCommandRequirement the job will run with subprocess.Popen(shell=False) and thus no shell metacharacters can be used.

My misunderstanding was I thought that all CommandLineTools were run using a shell and could use metacharacters.

[Tom]

I just wanted to drop by to say i love that you always take the time to come back and explain, even after solving the issue for yourself.

[ionox0]

Thanks Tom! I still don't understand whether additiona arguments need to be quoted after the pipe when using /bin/sh -c, or whether that doesn't matter.

[Tom]

I have no idea either! I'm not great at cwl yet. Just facing my own cwl-reated issues almost daily atm, and your threads often come up when i'm looking for solutions. Thank you for providing so many answers!

[ionox0]

Glad it's helpful! It is also helpful for myself to be able to explain it. Another thing I realized was that even though the command is printed like that by cwltool when it runs the command, cwltool is a python process, and it's not the same as running the command from a shell / terminal outside CWL