Should I do P-value adjustment in this case?

Question

[deep771992chanda]

Hi there!!! I have done functional analysis of WGS metagenome data HUMAnN. I will be testing differentially abundant functional pathways among the control and test groups. I have around 400 features (pathways) to test. Considering, this, should I find out differentially abundant pathways depending upon adjusted P-value (q-value)? Same question I also have for MetaPhlAn taxonomical analysis, where I have to check for around 300 taxa.

Thanks

[Mehmet]

You can report your results as:

1. Raw p-values
2. FDR correction at 0.05
3. FDR correction at 0.10

Answer 1

[Carlo Yague]

Short answer is yes, it is always better to use pvalue correction when you test for multiple things. I don't know where is the limit exactly, but IMHO, 300 or 400 tests fully justify the use of pvalue correction.

[deep771992chanda]

Thanks, CarloYague. But, when I did P-value adjustment I got only 2-3 significant results, which is around 30-35 when not adjusted. Can you please guess what may be the reason this happens?

[rpolicastro]

When you perform multiple comparisons, your type one error rate (false positives) increases by 1-(1-alpha)^comparisons. For 300 comparisons, your false positive rate is 99.99%, so you expect a fairly large number of significant values even if the null hypothesis is true. As for why you have a low number of values after correction, there could be a few explanations.

If you performed Bonferroni correction, the correction is considered too conservative and you may be increasing your type II error rate (false negative). A less conservative method is often used, such as FDR.

If you got these results with a less conservative p-value correction, such as FDR, then that could signify a few things. Either the null hypothesis you are testing against is indeed true for most samples, or the effect size is too small to see for most samples given your sample size.

[deep771992chanda]

Thanks, Rpolicastro- I am working on already submitted datasets and I am expecting a lot of significant features as a similar type of test has been done in other populations. Definitely, there should be enough significant features. I have done a LefSe analysis with my dataset. LefSe first does the Kruskal-Wallis test followed by effect size estimation (LDA score > 2.0). I get 30-35 significant features (P <0.05) after LefSe whereas the number reduced drastically while P-value adjusted. At this level, it is confusing whether it is necessary to adjust P-value as LefSe is doing two-step filtering (Kruskal-Wallis, Effect Size), and also I am getting almost no significant result after adjustment. What is your take call in this condition?

Thanks

[rpolicastro]

Is it only outputting the significant features? You need the p-value for all comparisons to compute FDR.

[deep771992chanda]

No, the output contains all the p-values. I have taken the table and calculated the adjusted p-value with p.adjust() in R.

[rpolicastro]

If you are doing 400 multiple comparisons with a p-value threshold of 0.05, you would except roughly 20 significant tests even if the null hypothesis was true for all tests. Since you are only getting 30-35, It's likely that most of the significant p-values are just consequences of multiple comparisons and not a significant deviation from the null hypothesis.

[andres.firrincieli]

What kind of adjustment method did you used?

[deep771992chanda]

Hi andres, I have done FDR correction (Benjamini-Hochberg). Thanks

[Carlo Yague]

If your pvalue threshold is 0.05, then you should expect about 20 false positives from 400 test. As stated by rpolicastro, Bonferroni correction is a bit over-conservative, so it makes sense that only a few of your 35 significant hit are still significant after correction.