Gage Vs Romer Vs Camera Vs Roast Vs Spia Vs ...

Question

[January]

I am working on a relatively complex experimental setup and trying to get meaningful results for a pathway-centered analysis. So far I have tried gage , camera, roast/mroast from limma and SPIA.

Of course, the results are not identical, but worse than that, some of the results are completely different. For example, camera reports no significant pathways whatsoever, while at the same time mroast reports about 50 (and 188 out of 202 tested have the "FDR.Mixed" value smaller than 0.05). gage shows a number somewhat in between, however I cannot easily compare the results; the limma functions camera and mroast allow the use of complex contrasts, while gage apparently can only do group vs group (paired or unpaired) comparisons. SPIA uses a network analysis combined with a simple enrichment p-value and seems to give results that stand out.

On the figure below you have a quick-and-dirty comparison of the results. Each dot is one pathway, and its position corresponds to -log10( pvalue ), where pvalue is the FDR value from the given method (so the larger the value on the plot, the smaller the p-value). The Spearman correlations between the different methods were calculated based on the logarithmized p-values and are shown on the lower panels.

My question is arguably not very specific: what would be your advice? Maybe there is some other tool I should test?

Answer 1

[bigmawen]

This discrepancy is more or less expected because these methods have very different focuses and assumptions. GAGE is consistent applicable to different sample size and experimental design. Camera tries to account for inter-gene correlation. SPIA considers pathway topology.

It is hard to compare by just looking at p-values or number of significant calls, and not fair to say one is better than others because it has special considerations or assumption. Even simulation study wouldn’t work well as its assumption may be biased. A fair comparison would use gold standard testing data or experimental validation. Another way, less rigorous but intuitive, is to visualize the results and see whether it make sense.

Answer 2

[pkpekka]

I would say that using limma-based gene set test is always preferable. They have lots of smart features and have been constructed by very top-level statisticians. Of those test I would prefer romer or camera. Maybe results that are significant in both (they should give somewhat similar results).