Constructing a mean ROC curve based on 5 iterations

Question

[Swimming bird]

I did a 5-fold cross-validation for testing a binary classifier. Using the results, I constructed a ROC curve for each iteration with ROCR package:

Is there any way to construct a mean ROC curve based on these 5 curves?

[Len Trigg]

This seems off-topic for biostars? Perhaps more appropriate to ask on stats.stackexchange.com

Answer 1

[Collin]

What happens in 5-fold cross-validation is that you train on 4 of the folds and predict on the hold-out fold. This procedure is repeated until all folds have hold-out predictions on them. Given this preamble, you could just concatenate the scores and labels from the 5 hold-out predictions (which should comprise all of your data points), and then construct a single ROC curve based on that.

Ideally with enough data and randomization of which samples fall into each fold, there should not be large discrepancies between the ROC curve of each fold individually.

[Swimming bird]

What you mean is to join the results of each interation in a single vector, right? The problem is that I am using a program that doesn't give a score of each datapoint, it only sorts the data points. Therefore, I can't join the results maintaining the rank...

[Collin]

Yes, I meant to concatenate them into a single vector. Nearly all methods provide a score which is used for ranking. It's odd that it's not provided as output. They may have an internal score that is just not getting returned to the user. You might need to contact the authors or see if you can modify their code to return the score as an output.

[Swimming bird]

Yes, it's internally calculated but I can't change the code. I'll try to talk with the authors.