Weka: ROC Area Value and PRC Area Value

Question

[atif.yilmaz7]

I m using Weka 3.7 to get some ROC Area value and PRC Area value by using HMM Classifier of Naive Bayes.

For multiple csv files, I always get 0.500 for ROC area values and multiple different PRC area values. Which value (ROC or PRC) is more important for HMM ?

Although I change the parameters of HMM classifier, I always get 0.500 for different csv files. Why Do I always get 0.500 for ROC Area value?

[karl.stamm]

a ROC AUC of 50% means the classifier is as accurate as a coin-toss. Something's not working or broken.

Answer 1

[Jean-Karim Heriche]

In my experience, precision-recall curves are more useful in practice. This is because the ROC curves only give you an idea of how the classifiers are performing in general. They give you the same result regardless of what the class probabilities are, i.e they consider equally the positive and negative classes. In contrast, PRC is more useful if you're only interested in how the classifier is behaving on one class. Imagine trying to classify patients as diseased or healthy. In this case, you are not interested in how many healthy predictions are correct, you want to predict correctly all diseased cases and not miss any.
You may want to have a look at this paper and this paper.
The behaviours of ROC and PRC in your case suggest a strong class imbalance.
Note also that a ROC AUC of 0.5 means your classifier is random.