Do I need to test the repeatability and reproducibility of a fixed bioinformatics pipeline for SNV calling

Question

[lzy]

Next-generation sequencing based genetic testing requires proof-of-concept validation with established performance metrics before introducing into the clinical laboratory. The performance metrics evaluated to establish the analytical validity of test results include accuracy, precision, analytical sensitivity, analytical specificity. Precision is typically determined by assessing repeatability and reproducibility. Repeatability (within-run precision) means testing the same sample repeatedly under the same operating conditions (the same people, same time, same place etc.) to evaluate the closeness of agreement between repeated tests. Reproducibility (between-run precision) means the closeness of agreement between the results of measurements when operating conditions are varied(different people, different time etc.). As bioinformatics analysis pipeline is a key component of next-generation sequencing, several guidelines recommended that the bioinformatics analysis pipeline should also be validated. My question is how to evaluate the precision (repeatability and reproducibility) of a bioinformatics analysis pipeline? Or, do I need to evaluate the precision (repeatability and reproducibility) of a bioinformatics analysis pipeline?

[jnf3769]

Izy | You might want to make your question clear. As phrased, your post does not contain a question. I think you are wondering what, after clinically important SNVs are discovered/approved, needs to be done to be sure of an individual's status for that SNV. But I am unsure.

[lzy]

Thanks, jnf3769. I have revised my question. I just want to know "How to evaluate the precision (repeatability and reproducibility) of a bioinformatics analysis pipeline? Or, do I need to evaluate the precision (repeatability and reproducibility) of a bioinformatics analysis pipeline?"

Answer 1

[NB]

So, in a clinical setting, we do test the specificity and sensitivity of our bioinformatics pipeline. This can be done in many ways, most common ones are-

1] run the pipeline on 2-3 previously analysed samples and compare results to see if the expected variants are called

2] run the pipeline using, for example, GIAB samples to compare, again, to see if expected variants are called.

More details are here : Guidelines for Validating Next-Generation Sequencing Bioinformatics Pipelines

[cpad0112]

In addition, use a framework that stores all the logs and versions of the tools, databases and references (provenance) you have used. If you have used custom scripts, try to version them

[lzy]

Hi, Nandini. Thanks for your answer. The specificity and sensitivity can be evaluated by running the pipeline on the benchmark data. I want to know do I need to run the pipeline on the same benchmark data 2-3 times to evaluate the repeatability and reproducibility of the pipeline. In my opinion, run the pipeline on the same data will get exactly the same result.

[NB]

Hi Izy, you need to run the pipeline on different sets of previously analysed data from an older or verified version of the pipeline and see if the variants are called as expected- that is repeatability . As you said, if you run the pipeline 2-3 times on the same data, you will get the same results. Hope this helps.