Tool:Compi: Application Framework for Portable Computational Pipelines

Question

[Hugo]

Dear community members,

We present Compi, an open-source application development framework for portable computational pipelines. The source code is available at GitHub and the full documentation is available here. In addition, Compi Hub allows users to discover, browse and reuse Compi-based applications easily.

The main features of Compi are:

• Language agnostic: Compi pipelines are defined in XML, where each task is run in an external program written in any programming language. If your program is a mere combination of existing tools, it is not necessary to code at all, only to define the steps of the pipeline and their parameters.
• Portable: Thanks to Docker, pipelines can be packaged in a Docker image along with their dependencies, making them really portable and reproducible. Using compi-dk, developers only need to complete the Dockerfile template provided with the dependencies that the pipeline needs. Notwithstanding, pipelines can be also run locally without Docker.
• User interface generation: the Compi workflow engine automatically generates a Command-Line user interface to facilitate the users the usage of the pipeline. Thus, Compi is in fact an application framework in charge of dealing with user interaction, multi-threaded pipeline execution and logging, saving developers’ time with these aspects. Developers can focus in things that are really specific to their pipeline-based applications.
• Parallel execution: Compi pipelines run independent tasks in parallel and pipeline users ande developers do not have to worry about parallel execution management. Pipeline execution can be resumed from any step, without repeating previous steps that may have completed in previous runs.

Compi is an ecosystem that comprises:

• compi: the workflow engine with a command-line user interface to control the pipeline execution.
• compi-dk: a command-line tool to help in the development and packaging of Compi-based applications.
• Compi Hub: a public repository of Compi pipelines that allows other users to discover, browse and reuse them easily.

Answer 1

[Hugo]

Dear community, our paper about Compi at PeerJ Computer Science has just been published, check it out at https://peerj.com/articles/cs-593/

Also, since the first post, we have also released several versions with important updates in Compi and improved the online manual (https://www.sing-group.org/compi/docs/index.html).