Best practices for FDR correction during multiple conditions analyses

, ,
1

Hi,

My question is regarding multiple comparison, especially the FDR method. It is said that we should apply the correction whenever we perform non-independant tests. However, this criteria seems vague to me in some cases.

Consider this:

I use FDR correction on a set of p-values obtained from a parcel-wise analysis, where I perform a statistical test independently for 200 parcels of the brain (regression, ISC, etc.). I then get 200 p-values that I correct with the FDR algorithm (implementation in nlTools.stats.fdr, Tal Yarkoni: API Reference — nltools 0.5.0 documentation) to know the p threshold that yields q < 0.05.

However, if I have 4 conditions and perform 4 x one-sample t-test across the 200 parcels, then the number of independent tests is 4 x 200. Similarly, if I perform 4 contrasts between these conditions, should I apply the FDR on the whole set of p values (4 x 200 parcels), or independently on the p values of each contrast?

Depending on what we consider to be independent, the results will change, since the corrected threshold could be lower for a condition that has a lot of small p-values (e.g., p <0.003), while it would be more stringent for a condition that has very high p-values (e.g., p < 0.000001).

In nilearn, for example, the correction is applied to one statistical image at a time, passing a Z_map into the multiple comparison function. My intuition would be that if correcting on a set of contrasts at the same time, such function would be built to take a list of images from the contrasts performed during GLM.

Curious to know the best practices regarding this,
Bests,
Dylan S.

2

Hi @dylanSutterlin,

It really depends on what your hypotheses are. If you will only ever interpret results within a condition, then correcting within-condition is defensible. If you will be making comparisons across conditions (e.g., “parcel X is significant in condition A but not B”), then the whole set should be treated as one “family”. You may also consider techniques like FDR within-condition and Bonferroni across-condition. There’s also an argument to be made that even different contrasts within a person are not independent because they arise from the same brain, in which case you might correct across the entire 800 values.

These are not always cut-and-dry questions, so the most important thing is to be transparent in your reporting. You could even report two q-values alongside the uncorrected p, to account for lenient and conservative estimates.

Best,
Steven

3

Thank you greatly for this nuanced answer!

I will give it some thought, and report transparently anyway.

Best,
Dylan

1 Like