Minimal CWL (Common Workflow Language) workflow from Makefile

Question

[Pierre Lindenbaum]

Common Workflow Language (CWL) https://github.com/common-workflow-language/common-workflow-language / http://common-workflow-language.github.io/draft-3/ has been trending on my twitter timeline during the last weeks.

However the spec is quite large and I find it hard to get some simple examples.

Furthermore, I have the feeling that all engines require a lot of dependencies or docker. I'd like to test my makefile-based workflows using CWL, how should I write and test the following simple Makefile using CWL:

SHELL=/bin/bash
.PHONY:   all
all :  database.dna

database.dna :  seq1.dna seq2.dna seq3.dna
    cat seq1.dna seq2.dna seq3.dna > database.dna

seq3.dna :  seq3.rna
    tr "U" "T" < seq3.rna > seq3.dna

seq3.rna :
    echo "AUGCGAUCGAUCG" > seq3.rna

seq2.dna :  seq2.rna
    tr "U" "T" < seq2.rna > seq2.dna

seq2.rna :
    echo "AUGAAGACUGCGAUCGAUCG" > seq2.rna

seq1.dna :  seq1.rna
    tr "U" "T" < seq1.rna > seq1.dna

seq1.rna :
    echo "AUGAAGACUGACUCGUCG" > seq1.rna

EDIT: feel free to add the file for your favorite workflow-engine as an answer.

@yokofakun oh, still no reply.. I can provide it for @nextflowio DSL, but I guess you are not interested to that ;) @smllmp

— paoloditommaso (@PaoloDiTommaso) July 28, 2015

@yokofakun Could have 2-3 levels of "advanceness" though :) Have used this from a tutorial before: https://t.co/GivahZipKW @PaoloDiTommaso

— Samuel Lampa (@smllmp) July 28, 2015

[peter.amstutz]

Responded with an example on this github issue: https://github.com/common-workflow-language/workflows/issues/1

Answer 1

[herve.menager]

To describe this workflow using CWL you need to:
- describe the 3 different tools (tr, echo, cat)

- describe the workflow structure

- describe the input data for the workflow job, i.e. the 3 input strings (AUGAAGACUGACUCGAUCGAUCG. etc.)

As an example to see the implementation of "tr", you can see this gist: . The github repository for common-workflow-language also has a number of tool and workflow descriptions that might help (https://github.com/common-workflow-language/common-workflow-language/tree/master/conformance/draft-2).

The reference implementation should be fairly easy to set up (pip install cwl-runner) but it is minimal (no cluster integration, etc.). Docker is not and should never become a requirement of CWL, some tools have been described with docker "requirements" but this is absolutely not mandatory. You can also ask your questions on all the channels mentionned here: https://github.com/common-workflow-language/common-workflow-language#community-and-contributing

Hope this helps,

Hervé

[Pierre Lindenbaum]

many thanks Hervé !

Answer 2

[peter.amstutz]

Responded with an example on this github issue: https://github.com/common-workflow-language/workflows/issues/1

Answer 3

[sahiilseth]

I do plan to create a CWL format importer for flowr in the future. Though, here is a example, using two simple tab-delim files. Feedback, really welcome!

Install flowr following these steps.

You may edit this file, to run the same flow in various clusters, or local envir.

vi simple_make.def

Additionally, for easy visualization, you may use this command:

flowr plot_flow x=simple_make.def pdffile=simple_make.pdf

When done, submit:

flowr to_flow x=simple_make.tsv def=simple_make.def execute=TRUE

Output: https://github.com/sahilseth/flowr/files/4729/simple_make.pdf

Answer 4

[Pierre Lindenbaum]

I'm copying Paolo Di Tommaso's gist for http://www.nextflow.io/ below

@yokofakun OK, I've uploaded it to the gist below. Feel free to comment if you have any doubt. https://t.co/n9rk8asJhI

— paoloditommaso (@PaoloDiTommaso) July 28, 2015

[Jeremy Leipzig]

correct me if I'm wrong but this isn't CWL-compliant

[peter.amstutz]

"EDIT: feel free to add the file for your favorite workflow-engine as an answer."

That's right, that's nextflow.io.

[Jeremy Leipzig]

is CWL a workflow engine itself or a standard for describing the interface between components?

[peter.amstutz]

The second one. It's a specification for a declarative document format for describing portable workflows. The goal is to have lots of interoperable implementations.

Answer 5

[PoGibas]

Will try to contribute. This is not an actual answer, but how I try to keep my projects tidy using Makefile (aka "reproducible research").

Dependency for every script is a dummy file (shed for sh or Red for R)

analysis=\
    download.shed \
    clean.shed \
    statistics.Red \
    figures.Red

###########################################
# Analysis Workflow
###########################################

all: $(analysis)

# Download data
download.shed: download.sh

# Clean data
clean.shed: clean.sh download.shed

# Summary statistcs
statistics.Red: statistics.R clean.shed

# Plot data
figures.Red: figures.R statistics.Red

###########################################
# Execute
###########################################

%.shed: %.sh
    ./$< 2> $(basename $<).out && touch $@

%.Red: %.R
    R CMD BATCH --no-save --no-restore $< && touch $@

clean:
    rm -f *out *.shed *.Red

[Pierre Lindenbaum]

thanks, I know how to write a makefile ( e.g. https://github.com/lindenb/ngsxml ). However, what I need is a solution for the alternative solutions, especially CWL.