Plotting ROC curve for categorical variables

Question

[vigprasud]

I have results from a manual exercise and an automated exercise to predict pathogenicity of variants. How do I show an ROC that shows FP rate and the performance of the tool ?

Variant Predicted Reported

2:17959282:C:T LikelyPathogenic Likely Pathogenic

2:17961268:C:T LikelyPathogenic Likely Pathogenic

2:48025764:C:G Pathogenic Pathogenic

3:142188289:G:A VUS VUS

4:84383824:AT:A Pathogenic VUS

7:6043640:ATTGTCTGAAAC:A LikelyPathogenic Pathogenic

8:90983441:ATTTGT:A VUS Pathogenic

[Jean-Karim Heriche]

ROC curves are obtained by plotting the true positive rate against the false positive rate. Performance is usually measured by the area under the ROC curve.

[Ahill]

In the results you pasted, it looks like you have a single prediction of 7 cases, with overall accuracy = 4/7, if 'Reported' column is the ground truth. This observation would define a single point on a ROC curve. Plotting a ROC curve requires a classifier/predictor that can be parameterized to adjust sensitivity/specificity over a range, for example one that provides a continuous score that is cut to make a class prediction. If your automated classifier provides such a score, you would run your classifier at a range of cutoff levels, and capture the TPR and FPR at each level to form the ROC curve. Several R packages can do that for you, e.g. ROCR. Since your classifier is multi-class (you show 3 possible prediction levels above) you would need to define what a positive and a negative are for your case. For example, is a "positive" call one where predicted == "pathogenic", and all others "negative"? If you define positive="pathogenic", then in your prediction above, you have TPR=1/3 and FPR=1/4 for this classification of 7 cases.