Tool:BioFlows - Container-enabled Bioinformatics pipeline engine

Question

[drsami]

I am thrilled to announce that the alpha release for #BioFlows (Container-enabled bioinformatics pipeline engine) is out. This engine was built entirely in #Golang. I encourage Bioinformaticians, Computational Biologists and Computational Chemists to give it a try, it is the best of all worlds combined..... if you are a biologist, a chemist and don't know any computer programming, you are still able to run your sophisticated bioinformatics pipelines peacefully. Please visit the documentation to learn more about #Bioflows You are more than welcome if you want to test, contribute or send any comments via the main github project page.

Click here to go to Bioflows Official website

I very welcome anyone who wants to test or contribute to the project. Please drop me a message or open an issue or a discussion on the project's main GitHub repository. I want more people for writing documentation, test and give back comments or new features to the project.

[steve]

congrats on the release. Since there are many workflow frameworks out there, it might be helpful to include a comparison of Bioflows to other systems, especially CWL, Nextflow, and Snakemake, since those seem to be some of the most popular in bioinformatics.

[csharpizer]

Bioflows will supersede these engines soon. CWL is a standard that will be supported by Bioflows engine soon. NextFlow and SnakeMake were there and people used to use them. People always resist new things. But the flexibility and power of Bioflows will make these tools obsolete in few years. I already made a comparison in a power point, but I have hard time to keep up writing documentation , adding new features, testing current release and doing other things. But I will publish these comparisons when i publish the second release of BioFlows with distributed support and HPC. Currently, Bioflows 0.0.3a supports DRMAA specification, supports PBS torque, Slurm and Sun Grid Engine but I am testing that now and will soon merge that into master branch soon.

[WouterDeCoster]

That's a bold and ambitious statement! Looking forward to new features, and I hope to have time to play with this at some point.

[WouterDeCoster]

Hi, this looks great. Quite impressive documentation too. I got hooked on using snakemake for my pipelines, so it's always a challenge to convince people that they have to invest time to learn something new (like bioflows). Are there specific features that are different between snakemake and bioflows?

Answer 1

[Istvan Albert]

One of the fundamental problems with tools like yours above is that they do not help with the idiosyncrasies of bioinformatics.

One still needs to get well trained and understand bioinformatics, what the parameters are called etc.

A better approach would be to seamlessly integrate especially the inputs and outputs. A user should not need to know that a file is gzipped or not, it should be trivial to detect that and unpack it, unzipping files should not be something we need to remember. Gzipping is not a pipeline element, it is not science.

Same with inputs, say I want to pass a reference to bwa or blast or minimap or bowtie, I don't want to have to remember that it should be -in for makeblastdb, but it should be and indexed -x for bowtie2 etc. The pipeline's smarts should take care of that.

Right now your tool is just a variant of a makefile, or a snakemakefile or nextflow. Each of these approaches suffers from the same problem. You already have to know and be an expert command line user, only then can you meaningfully use the pipeline. But basically what it means it requires another layer of expertise.

Make it so you can just combine a genome + query and pass it to a tool, then the framework figures out what it needs to do to make the files suited for processing.

[steve]

not sure I would fault a program for something it wasnt designed to do

[Istvan Albert]

I am not faulting the tool, I am pointing out flaws in the design of the tool, the intent of creating another tool just like the one before.

Say I have a reference genome and I have a paired fastq file. I would like to run five aligners and get an overview of how well each works without having to figure (or remember yet again) that one takes the reference with a little -x, and the index builder is called blah-builder, then the other takes files with -1 and -2, the third has to have the little -S otherwise all hell breaks loose and so on

make people more productive, help them solve their chores, and that would do a lot more for making the tool popular than anything else

[Ram]

As long as the tools themselves don't change, someone along the way has to write out these parameters. I don't see the downside to giving pipeline designers the control (and responsibility) of getting the exact commands right - much like snakemake does. I don't see a way where no one involved in developing/using a pipeline has to know those pain-in-the-behind details.

My simple RNAseq pipeline, for example, detects if the FASTQ files are gzipped before using rsem-calculate-expression (which for some reason, much like STAR, needs a separate option to be added in if reads are gzipped). Unless RSEM/STAR start detecting input formats on their own, I'm going to have to program it in somewhere.

[drsami]

I really lost ground of what are you fighting for. If a tool uses specific switches, users of this tool must provide these switches to make it work properly, and that what ALL pipeline engines do.

I am not faulting the tool, I am faulting the design of the tool, the intent of creating another tool just like the one before.

BioFlows and NextFlow and SnakeMake are workflow managers, they share some common features, because we are on planet earth abide by common physical laws. but each one has its own flavor of doing things. Just wait till BioFlows complete and then complain if It is exactly the same or better, Ok.

Say I have a reference genome and I have a paired fastq file. I would like to run five aligners and get an overview of how well each works without having to figure....

You can do just in BioFlows, who said you can't do that. You can create A SINGLE tool that controls which command to run based on user inputs OR you can wire up five tools each for a different aligner in your DAG graph.

[Istvan Albert]

you are misreading my comment, if I thought the tool had little utility I would not even bother to say anything at all

For every workflow engine, the real challenge is providing the shims that connect the tools. That is the bottleneck in bringing bioinformatics to biologists. When you drag one file into another, like a reference into a blast, your tool should provide the means and as much as possible figure out if that data is ok as is or needs some transformation (unzipping, indexing etc). But not by the creator having to painstakingly write out all paths with if-else, bugs, and errors abound that way.

I have been teaching bioinformatics to students and life scientists for over a decade. I understand coders, I understand hackers and I understand biologists. The design and setup of your tool put it in a bit of "no-mans-land". A hacker already uses snakemake, nextflow, or bds, it is not appetizing to learn another framework with its own quirks. For biologists, what you have there is too complicated. YAML is too tricky to write by hand since indentation matters a lot. Having your tool solve some of the pain points and annoyances, not just add even more constraints, will go a long way in getting people on board.

You are welcome to disagree, but don't construct my criticism as negativity towards the idea of what you are trying to do. I may be terse, but I actually trying to help you. Someone will get it right eventually, if not this project then someone else. The withering criticism of any project is when people don't even bother to comment.

[drsami]

On Contrary, You are very welcome to comment and give feedback, I really appreciate it, without this, the project will not go any further. I am building this tool for everyone and for myself. I want to concentrate on the science behind the project not a groovy code or scala.

BioFlows should not care about if the fastq is raw or Gzipped, this is a matter of the bioinformatics tool that will take this file as input, BioFlows is subject-agnostic. I will not couple BioFlows with the intricacies of Bioinformatics. Anyone else could use it to build the next generation pipeline for computational astrophysics to detect our next home. BioFlows should not care what are the data formats used in this discipline. All I want is a tool writer, write his tool once, another user understands the parameters, read the description, execute the thing, get the results. I don't need him to care about prerequisites, docker or not, HPC or not. I need him to concentrate on what he is searching for, the science behind his project.

YAML is tricky because of indentation, but you can write it not using notepad, but advanced text-editor which understands YAML, like Atom.io,TextMate, Pycharm , IntelliJ, Eclipse and thousands of others.

Moreover, BioFlows, already gives you a command called "validate" which basically lints the file and check if it correctly formatted or not before running the pipeline and if there is any self-references or problems in the DAG structure.

I will tell you one feature which I am currently working on but I will test the audiences view about it, I didn't want to announce it now, but I will.

What if you want to commercialise your skills of bioinformatics by selling specialized pipelines or pipelines containing your special custom scripts or analysis. In Bioflows, you can send a single ".bf" file which is generally a gzipped file, containing inside the pipeline and any additional custom scripts, which is encrypted by your encryption key and no one else could run this pipeline except when you give him another key or a certificate to run this pipeline with a time frame which expires after some time. No one on earth can know that script because BioFlows extracts that file and put it on memory decrypted use it and destroys it.....

That is the Power of having your own Tool. Sky is the limit.

At the end, BioFlows will be there, if you don't want to use it, It is fine. but apex predator will ROAR :)

[Ram]

I applaud your ambition, but advertising your product with phrases like "apex predator will ROAR" serves less as hype and borders on over-compensation. Please let the product speak for itself. Trust me when I tell you that "colorful" statements like the one above and other similar phrases you've used in other comments work against your credibility.

[drsami]

Let the product speaks for itself. very nice sentence, I agree. So Let the rolling begins.

[WouterDeCoster]

Well, as soon as someone writes the workflow rules or wrappers (at least that what those things are called in snakemake) then it becomes as simple as plugging in {input} and {output}. Of course, the one who wrote the rule has to know the parameters of the tool. Are you suggesting we should all stop reading the documentation? Surely most of the users are capable of reading? I'm not sure which solution you have in mind that is going to prevent different tools having different input and output parameters.

Even more, what you are describing is roughly solved by workflow managers: as soon as you create the 'building blocks' (rules, etc) you can plug and play and combine e.g. different aligners and different read trimmers. You just need to link {input} to {output}. You can forget about -S, -1, and -2.

[jared.andrews07]

Yeah, I am not sure I see what differentiates this from other options yet. As to Istvan's point, there are some efforts to handle the nitty gritty and build solid pipelines that require relatively little expertise to run (the nf-core pipelines come to mind immediately, implemented in nextflow).

Tools like this are fine for experienced bioinformaticians who are aiming to put together their own pipelines, but you don't do a great job differentiating it from other options (CWL, WDL, snakemake, nextflow, etc) that may already have highly vetted pipelines available to use/build upon.

[drsami]

Hello Istvan, Thanks for your reply. I think your comment is based on the sample workflow I implemented in the documentation website. but you should not conclude on a simple workflow. When I write the documentation, I should point every single piece of information and what are the possibilities. If you read my documentation, you will immediately read that I said, This example could be done without even downloading anything. As you knew, there are multiple solutions to a given problem. I tried to make Gzipping as a step just to show newcomers how to do things, but you are definitely fine if you want to embed it in another bigger step or have your files ready in a data directory, the example i copied had the data on the cloud, so instead of downloading it, i wanted to do the download and gzipping as separate steps but this is merely an example. The pipeline designer has the full options to do whatever he wants, to make Gzipping explicit or not, it is all a design decision taken by the designer him/her self. This is not a problem with the engine, It may be one way the designer has written his pipeline that you don't find it quite appealing. but definitely you are completely free to do whatever you want.

You said "Same with inputs,.......The pipeline's smarts should take care of that.". Of course, and I totally agree with you and you can do just that with bioflows. The tool designer can make his tool react to parameters given by the user. Like I can make a single tool to fire bwa, blast or minimap or even hisat2 with all its special command switches based on an external parameter, I can utilize both embedded bash or javascript scripting to do that as well. but any new user can utilize this tool immediately and run it by just giving the data, This new user (Y) needs not to remember anything, he just need to read online about what this tool does.

You said "Right now your tool is just a variant of a makefile, or a snakemakefile or nextflow", Yes, I agree to some extent. with the exception, that my tool is very easy for newcomers, they need not learn groovy or scala or a specific DSL to write tools or customize them, YAML is easy for anyone if he knows the tools.

Moreover, this is only Phase1 release. When I add distributed processing (Bioflows clusters) and Bioflows Hub (Like docker hub for tools,pipelines and containers with tools versioning), running bioinformatics pipeline will be as easy as pulling a pipeline from the hub and run the pipeline.

You said "But basically what it means it requires another layer of expertise.". I need you to differentiate between a pipeline writer and a pipeline user. Bioflows was meant to make a pipeline user's life easy just pull the pipeline give it the data and run it. but pipeline writers are already experienced users, I don't think writing YAML or bash or javascript is a problem for them.

Moreover, Bioflows allow tools/pipelines inheritance, you can inherit from a tool from (URL or through BioflowsId) and further customize its directives to suit the new project, by just writing YAML, no need to write groovy or scala to accomplish this.

nf-core.re for nextflow is "BioFlows Hub" for Bioflows. Bioflows hub will be more advanced with categories for doing a lot of things and with pipeline versioning (Which nf-core now suffers from, if you submited a RNA-seq pipeline that performs traditional Differential gene expression, no other users can upload his own variant of that pipeline). This is ridiculous since, new packages are invented all of the times, and tools/steps in RNA-seq might differ.

I will support CWL and WDL later, so basically, bioflows will recognize pipelines written in these standards and will be able to execute them the same way as YAML definition files. CWL is not a pipeline, it is a standard data format for expressing the logic of a pipeline in a common standard.

[jared.andrews07]

nf-core.re for nextflow is "BioFlows Hub" for Bioflows. Bioflows hub will be more advanced with categories for doing a lot of things and with pipeline versioning (Which nf-core now suffers from, if you submited a RNA-seq pipeline that performs traditional Differential gene expression, no other users can upload his own variant of that pipeline). This is ridiculous since, new packages are invented all of the times, and tools/steps in RNA-seq might differ.

I don't want 20 different RNA-seq or variant calling pipelines to choose from, I want 1 that can do more or less whatever I want built on reliable, well-regarded software. nf-core does a pretty good job at maintaining flexibility and offering several options for most steps of each pipeline to make them appealing to a broad audience. If I wanted, I could take their pipeline and alter it however I see fit and throw it in my own github. Why would I learn yet another new workflow language to craft my own pipeline over adding functionality to an already existing pipeline (whether it be nf-core or any other popular collection)?

My point is that it's really not clear what sets this apart from any other workflow language or the motivation behind it.

[drsami]

Of course, having one pipeline in nf-core.re for specific goal is a limitation, because new better tools are invented continuously why should I use RNA-seq pipeline that still uses Tophat while I have Hisat2? What if I wanted to use Callisto instead or salmon or..... If you customized it by hacking the code, how many bioinformaticians are capable of hacking a groovy code. nf-core pipeline for RNA-seq is approximately 1181 Lines of groovy code. how many bioinformaticians are capable of customizing it to their needs. Moreover, what about the newly customized pipeline, why don't you share it with the world, what If someone needs to build upon it, what if he doesn't know groovy or DSL, Do we leave him stranded ???

There is no problem in having 20 pipelines for doing the same thing as long as you know what you are doing and is capable of choosing properly, people will give upvotes for better pipelines that you can choose the best. Moreover, a single maintainer in Bioflows is capable of controlling tools within a single pipeline using versioning. Simply, you have one pipeline from a single maintainer but with different versions. first version uses this and second version uses that. and it is all about you to decide which one to use. There is no Limitations for human innovations in BioFlows Hub. but in NextFlow if you customized that pipeline or updated one of its tools, you are DOOMED, you can not submit it back to nf-core. you need to upload it somewhere and let other people find you by recruiting DOGS to smell for it.

The motivation behind this engine is giving an engine that just works, no steep learning curve is required upfront to customize a pipeline or write your own. I am an experienced developer with +12 years experience doing software development in nearly all languages, I even wrote groovy and groovy on grails systems before. but I was faced with one problem, to know their DSL. I didn't have time to leave the project and concentrate on understanding their DSL. So I invented a miniflow python library, it is on my github account, but It was suffering from many issues, so I wanted to create something that just works, that allow a researcher focuses on his research only. second goal was that, I wanted to build a computational empire which I would like to see. I can now design visual canvas like "Galaxy" for people to compose new pipelines from smaller tools, similar to galaxy but on better advanced distributed container-enabled engine. You can't have this feature in nextflow, Nextflow is just another SCRIPT. Third Goal, If i want to design a new precision medicine system, Bioflows will be the main cornerstone behind such system, just think of something like seven-bridges or Arvados but for specialised purposes.

Furthermore, Having experienced bioinformaticians like you compare Bioflows with nextflow and snakemake is a success for me on its own. Since, these tools are backed up by spanish institute with many developers and years of development. but Bioflows is just below 3 Months old. In the future, Bioflows will be the leader in the open source scientific community. The purpose of posting this in here is to announce for it and perhaps have people interested in joining the efforts to develop something better and easy for pure biologists, chemists and everyone to use.

[jared.andrews07]

Of course, having one pipeline in nf-core.re for specific goal is a limitation, because new better tools are invented continuously why should I use RNA-seq pipeline that still uses Tophat while I have Hisat2? What if I wanted to use Callisto instead or salmon or...

Pipelines can and are updated to add new/better tools all the time. Many use several of them for the same step (say, transcript quantification or alignment) if you want. And if I customized a pipeline on my own and thought it would be useful, people would be free to use it. Or I could make a PR to the originating pipeline to get the functionality added.

Third Goal, If i want to design a new precision medicine system, Bioflows will be the main cornerstone behind such system, just think of something like seven-bridges or Arvados but for specialised purposes.

I don't at all see how this is related, but I wish you the best with it and with this project as a whole.

[drsami]

I will tell you how this is related. Just think of bioinformatics tools as recipes for data analysis, not every published pipeline will fit your specific needs, so from time to time, you would need to write your own pipeline from scratch by composing tools/steps to build a pipeline. Assume, you need to design a pipeline that detect MiRNAs or lncRNAs as biomarkers for a specific disease, you searched nf-core and there is no pipeline for that, you will build yours, right, others will be stranded. using Nextflow, snakemake will only allow you to execute that from command line only. Ok, now you need to have a system which understands Web technologies to submit, monitor, retrieve statuses, get results and communicate with your backend. Basically, you need an engine that speaks Restful APIs or RPCs and at the same time, you need to give the system users' the ability to modify pipelines, compose new pipelines and submit and monitor pipeline executions as a single node or in a cluster of machines. How will you do that by using NextFlow ? Moreover, asking for a pull request, basically deletes the older version, and the older version will be superseded by your recent version, which breaks the backward compatibility for others used to the previous version and removes the older version altogether which might be still required for specific projects by others. which in its own right breaks the concept of scientific reproducibility, because assume you already published a paper which used that older version :)

[Ram]

I know basics of snakemake and no nextflow. I use shell/python scripts to integrate my tools into a pipeline, as well as craft options for each tool based on the inputs (such as gzipped options etc)

I have a question - by the time I've learned the command line options for a tool and figured out how to put a pipeline together, I'm already good at the command line and prefer it over a graphical interface. At this point, I am just looking for something that will save me from editing a command to fit a bunch of input files - think of it as programmatically generating commands in the pipeline. Why would I now want to go to a graphical interface? I need more control and customization, not more abstraction. This is where snakemake is closer to what I need than any graphical version of snakemake would be.

Let's say I don't want to deal with the command line at all. I'd go for Seven Bridges, given how easy it is to run any tool there. I have zero need to learn the command line options for SB pipelines as long as I know what to change (which I can get from reading documentation). Is your tool designed for people who, say, develop SB pipelines? Because as a lone bioinformatician, it's asking too much to learn tool usage, programming languages AND workflow languages just to get a task done.

[csharpizer]

Yes, Bioflows will support visual canvas for users who used to drag and drop. this canvas will be integrated with the hub. so basically, you will see a palette with tools and pipelines as they are in the bioflows hub. At the same time, these recipes will also available for command-line users who prefer to fire their pipelines from command line. no need for you to use galaxy or Seven-Bridges for visual design and another tool for command-line. I think SB supports CWL. If this was the case, next versions of bioflows will support running these pipelines as well and at the same time, supports running bioflows pipelines in Seven-Bridges (Bidirectionally).

[jared.andrews07]

I still do not see how any of that relates to precision medicine. I worry you're shooting for the moon here by trying to do everything. I don't care about a GUI interface.

Basically, you need an engine that speaks Restful APIs or RPCs and at the same time, you need to give the system users' the ability to modify pipelines, compose new pipelines and submit and monitor pipeline executions as a single node or in a cluster of machines. How will you do that by using NextFlow ?

After reading your docs, I am not sure how this would be done in Bioflows. Nextflow pipelines can be made extremely flexible via config files and active monitoring of running pipelines is already a feature.

Moreover, asking for a pull request, basically deletes the older version, and the older version will be superseded by your recent version, which breaks the backward compatibility for others used to the previous version and removes the older version altogether which might be still required for specific projects by others. which in its own right breaks the concept of scientific reproducibility, because assume you already published a paper which used that older version :)

Sorry, but this is just wrong. You can pull versioned pipelines just fine, they don't "delete" the older version - you are free to use whatever version you'd like. That is rather the point of generating pipeline releases via github. You can read more about this for Nextflow here or snakemake here.

Again, I'm not trying to rough up what has clearly been a serious and ambitious project - I would just recommend you make more clear what the exact benefits of it are over competitors in as clear and succinct a manner as possible.

[csharpizer]

If you downloaded the bioflows executable, you will see a command called "bf Node" which will start a server running on the local system and will seek to join the cluster, but this is just a scaffold for the current development phase which is not released yet, but I am expecting to release it soon. because I am currently implementing the distributed part of the engine

Sorry, I had to use a different account to reply to you.

Of course, I know how github works. but for normal users trying to clone your repository and run the pipeline, they will have tough time till they understand how to clone by a specific commit ID or a tag Id to reach your pipeline version, they will always clone the head. This will be made explicit in BioFlows hub by having one command such that "bf Workflow run --bioflowId YXZXYZ" which will pull that pipeline by this ID from a centralised location which is the hub which is basically the same command written in your manuscript for the paper.

[jared.andrews07]

Of course, I know how github works. but for normal users trying to clone your repository and run the pipeline, they will have tough time till they understand how to clone by a specific commit ID or a tag Id to reach your pipeline version, they will always clone the head. This will be made explicit in BioFlows hub by having one command such that "bf Workflow run --bioflowId YXZXYZ" which will pull that pipeline by this ID from a centralised location which is the hub which is basically the same command written in your manuscript for the paper.

For most well-established/maintained pipelines, they are usually pulled from github via the pipeline manager rather than by users themselves.

For bioflows, how would specifying different versions work? Does the YXZXYZ bit change? I'd argue that if I'm publishing something and say I ran my nextflow pipeline via nextflow run nextflow-io/hello -r v1.1, that's pretty dang explicit and hard to screw up.

[csharpizer]

Yes, In Bioflows will be similar to that, You will have two options to accomplish the same thing. The first strategy would be to mention the version explicitly in the command line, I am thinking of something like this one "bf workflow run --bioflowId XYZXYZ:1.1" or by having an explicit switch like this "bf workflow run --bioflowId XYZXYZ --version 1.1". If you didn't specify a version, the latest will be grabbed. The other strategy would be if you want to grab the pipeline for a specific version but want to either override and/or extend the same pipeline. you will do something like this

id: "Something"
name: "Something"
bioflowId: XYZXYZ
version: 1.1
steps:
..
..

the second strategy allow you to customize the same pipeline with the same version by explicitly mentioning the bioflowId and version together. if the steps directive mentions the same step ID in the original pipeline, it will be overriden by what you will write, you can even impede a step or rewire it to some other step, otherwise, this step will be appended to the original pipeline.

This inheritance paradigm works currently, when you mention a pipeline or a tool remotely through "url" directive.

But because, I didn't yet start the bioflows hub, these will work in the upcoming releases when bioflows hub is developed and become online. Now, you can mention a pipeline or a tool by "url" directive with a full github or any other URL.

[csharpizer]

Precision medicine here is just an example. What I wanted to convey is that having to develop a system that is flexible, extensible and automated at the same time that could be deployed with GUI interfaces not for you as a bioinformatician only but also for clinicians will require integration with any bioinformatics pipeline engine. having nextflow will not help in here or snakemake or even manual written scripts. because these kind of systems are beasts, you would need flexibility for bioinformatics users to modify pipelines, monitor them and clinicians to receive reports all from GUI interfaces. like if you want to build a personal genotyping startup. you need to develop web based automated system, right. you need to develop a pipeline, fire it when patients genotyping data is available, monitor and receive results all from within that web system which will have login for bioinformaticians different from clinicians different from patients. you would need somehow an integration between your backend, frontend and bioinformatics pipeline engine. this is where bioflows Node solves the problem. It will give you a restful endpoints to control and manage the cluster, monitor the health of nodes, the status of pipelines, and web hooks when pipelines finish....etc.

[drsami]

Please check my comments.

[Ram]

Please stop editing comments to add more content. Remember - nobody likes reading walls of text.