How to compare the alignment quality of two or more mapping software

Question

[ravi.uhdnis]

Hi,

I need to compare the performance of 2 or more alignment software (e.g. BWA, a new software X) while aligning a particular gene (e.g. HLA-A) against hg19 as human reference genome. Basically, I need to know sensitivity, Specificity, Accuracy, False Positive, False Negative of these software taking the same set of gene as input and against same reference human genome. Although BWA gives basic alignment summary and we can have a overall look via evaluating mapping file (.bam) using IGV but is there any other better way to look the overall alignment summary by different software of same genomic region of human genome after mapping by different software for their performance comparison ?. Looking for suggestions, Many thanks.

Answer 1

[Devon Ryan]

The only way to do this is to generate synthetic (i.e., fake) data with known results, align it with all of the tools, and compare the results. You can do this with tools like wgsim, art, etc. Here's an example from Heng Li using wgsim to compare a number of common aligners (N.B., the comparison is very very outdated).

Answer 2

[harold.smith.tarheel]

BBMap 'randomreads' command lets you generate synthetic data of specified read lengths, error types and rates, etc for benchmarking. And you can use 'samtoroc' from the same package to generate ROCs for comparing aligner performance.