Cross Validation Software Or R Packages..

Question

[Ron]

If i have a microarray gene expression dataset e.g eisen or gasch what software or R package can I use to do a cross validation to find the optimum value based on the AUC for certain features such as minimum number of genes to be used in developing gene network or pathway.. can anyone help me...

[Michael]

Don't know if many people know about Akaike's Informatino Criterion: http://en.wikipedia.org/wiki/Akaike_information_criterion

[Michael]

... or Area Under the Curve http://en.wikipedia.org/wiki/Area_under_the_curve ?

[Ron]

i mean area under curve (AUC)

Answer 1

[Mikael Huss]

ROCR is a good R package for computing ROC-AUC values and plotting many types of curves such as ROC, precision-recall, lift charts and so on. It supports cross-validation in the sense that it can plot average ROC (and other) curves for multiple runs with error bars in a way that you don't need to calculate averages and standard errors yourself. The cross-validation itself is just a matter of writing a loop where you select a different subset of the data at each iteration.

[Ron]

R packages are very interesting..unfortunately i'm still a noob in R.. do u have any idea on how to complete the task ask i mention in the question using this package..or do u have any reference that i can refer which r doing the similar task? anyway..thx

[Mikael Huss]

It's hard to give an all-purpose answer to a relatively vaguely specified task - for one thing, you haven't specified what kind of classifier you will be using.

Answer 2

[fran supek]

You could use Weka, a popular general-purpose data mining software. It supports a variety of attribute selection schemes that you could use out-of-the-box, and some can be configured to use crossvalidation AUC as a guiding criterion. For instance, you could use the "WrapperSubsetEval" which can use any of the supported classifiers in Weka (and there are many!) to find a subset of features that maximizes a classifier's AUC.

Or, you may want to check out Weka's SVMAttributeEval which iteratively removes attributes by using attribute weights derived from Support Vector Machine models (not AUC!) and has been shown to work well on microarray data (see Guyon I et al, Machine Learning 2002, 46:389-422.)

[Ron]

its quite a complex software for me as a newbie..will explore it sometimes..anyway thx..

Answer 3

[aquaq]

I use caret R package, it's flexible and quite easy. It has built-in cross-validation and other tools. You can also train for optimizing AUC.