Correcting for multiple testing: 12 multiple regressions, with several covariates.

Question

[Elizabeth]

Hi, I am running 12 multiple regressions. Each has a different outcome variable, but the independent variable and covariates (age, sex, education) remain the same. The outcome measures are correlated (they are all measures of cognitive performance on various tasks). The 12 regressions are completed in the same sample.

How should I correct a family-wise error rate of 0.05 for the above analyses?

Thanks!

p.s. I have seen Bonferroni correction is often used, but I don't think I can use Bonferroni correction as this assumes that all of the hypothesis tests are statistically independent...

Answer 1

[Steven Lakin]

It's hard to know without seeing your exact problem, but I would recommend looking into multivariate multiple regression and not worry about multiple test correction in this case. Because your dependent variables are correlated, you should model that correlation, which requires a multivariate distribution with a variance-covariance matrix. This will allow you to assess your independent covariates without the need for test correction, since the entire experimental design is taken into account in one procedure. There are several R packages that can walk you through this. If your outcome variables are non-normal, you may need more advanced methods, and stats stackexchange might be a more helpful place to ask about that.

Answer 2

[Lemire]

The simplest way would be to perform permutations: start with one replicate where you randomly permute the values of the independent variable then do your 12 regressions. Keep track of the smallest p-value you got over the 12 regressions. Do this for many replicates. At then end, compute the proportion of replicates for which the smallest p-values were actually smaller than the smallest one from your real analysis. That's your FWER.

Alternatively, you could compute the "effective" number of (uncorrelated) tests, then perform Bonferroni with that number. This has been done in the context of correlated SNPs (e.g. https://www.nature.com/articles/6800717) but these methods can be applied to correlated phenotypes as well.