Calculating The Performance Of A Tool

Question

[Naga]

Usually bioinformatics papers uses Precision, Recall, Accuracy, and MCC for demonstrating the performance of a tool.

What are the other alternative statistical measures, one can use to calculate the performance of a tool (using the same TP, TN, FP and FN) ?

Answer 1

[Michael Kuhn]

The ROCR package for R can calculate lots of performance measures:

Accuracy, error rate, true positive rate, false positive rate, true negative rate, false negative rate, sensitivity, specificity, recall, positive predictive value, negative predictive value, precision, fallout, miss, phi correlation coefficient, Matthews correlation coefficient, mutual information, chi square statistic, odds ratio, lift value, precision/recall F measure, ROC convex hull, area under the ROC curve, precision/recall break-even point, calibration error, mean cross-entropy, root mean squared error, SAR measure, expected cost, explicit cost

[Naga]

@michaeal: thanks michaeal.. thats a long list. :) just installed the ROCR, will play with it.

Answer 2

[Andra Waagmeester]

I would propose uptime in the last x days/weeks/months. If your tool has the ideal scores but has really bad accessibility rates, it still is a bad tool.

It also would be nice to quantify the numbers of issues in a bugtracker and how fast things get fixed. Although it might be difficult to interpret this last metric, since an empty bugtracker could also mean that people are just not interested.

[Michael Kuhn]

Did you see the "using the same TP, TN, FP and FN" part of the question? Your answer is more geared towards already-published web tools.

[Neilfws]

Yes, the question is about getting predictions correct, not other definitions of performance.

[Naga]

Yes, I wanted to know what are all the different performance measure one can use to measure the predictions for publishing.