In a recent Nature column, one researcher states, "precision medicine cannot advance without full disclosure of how commercial genome sequencing and interpretation software works."
Link to the article: http://www.nature.com/news/no-more-hidden-solutions-in-bioinformatics-1.17587
I don't really follow the logic there - evaluating the performance of a tool should be independent of understanding how the tool works or why it works. This is like saying that we cannot evaluate an car unless the manufacturer discloses all their trade secrets and shows us the blueprints.
(Moreover the overwhelming majority of people that use an open source tool do not understand what it does internally - yet they should still be able to evaluate it)
The author was not asking for full details. He wanted to know how a commercial pipeline was trained such that he could avoid evaluating the tool on the training data. This request was refused. Evaluation on training data is a real concern. It is also my experience that callers usually work best on well known datasets but are not as good on others.
Well there may be some specific issues and situations that triggered writing the piece but I still think the author ends up with what I would call misleading and false conclusion - that of needing to know absolutely all internal details to evaluate a tool.
One can always generate synthetic datasets, use recently published data etc .. there are many options.
Training on known datasets is common - but then even graphics card companies are known to tune their hardware to optimize the results of the graphics bench-marking programs ... so it is not something new - but again that does not mean that the chips do not improve.
The original post should focus not on mandating rules but on educating the decision makers - remember companies do what it takes to survive - have well trained people make financial decisions and all of sudden only those companies remain whose tools are actually useful.
I agree. It is too bad that the author does not make a clearer distinction between comercial tools which can be benchmarked (a commercial short read aligner) and black box pipelines. Some companies do perform clinical sequencing/analysis/interpretation with little to no details of their methods or (public available) benchmarks and that is a problem.
For me the veiled message is that we need more independent datasets and more benchmark studies, so that the highly overfitted black-box software tools can be identified. I think it is a common practice to compare results from open-source software once one gets a dataset processed on the side of a company offering sequencing services using their proprietary protocol. However, the results of those benchmarks are mostly left behind once a paper is published.
PS In my primary field of research the situation is almost identical to the one described in the paper, with almost no way to test commercial software using synthetic data or raw sequencing data except for splitting a sample into two parts and doing a parallel library prep & analysis.
Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
version 2.3.6
I added a little context (usually a lone link is a bit suspicious). Feel free to change/update the post further.