I am hoping to get a better understanding of the multiple comparisons correction options available for ALE CBMA subtraction results in NiMARE 0.4.1. Could someone kindly help me answer the following questions?
Is there a multiple comparisons correction for ALE subtraction results in NiMARE 0.4.1?
My assumption is, yes, because I can run FWECorrector.transform(subtraction_result) without errors and with plausible results. What makes me doubt is that Dr. Salo has explained in previous posts that no multiple comparisons correction is currently implemented for ALE subtraction results (Sanity check NiMARE analysis ALE - #16 by tsalo; NiMARE ALE are results two-tailed? - #4 by tsalo). This may certainly have changed in the meantime, but I have not been able to find a corresponding entry in the NiMARE change log. I am therefore wondering whether I am maybe misinterpreting what FWECorrector.transform(subtraction_result) does.
Which multiple comparisons correction options are available for ALE subtraction results?
According to FWECorrector.inspect(subtraction_result), both Montecarlo and Bonferroni FWE corrections are available. Choosing method = āmontecarloā will, however, leave me with an error: ALESubtraction.correct_fwe_montecarlo() got an unexpected keyword argument ān_itersā. Maybe, despite inspect(), Montecarlo FWE correction is just not available for ALE subtraction results? Which seems to be what the documentation suggests (NiMARE: Neuroimaging Meta-Analysis Research Environment ā NiMARE 0.4.1+2.g3e5becf documentation). Am I then correct to assume that Bonferroni FWE and FDR correction are the two only options available?
Which multiple comparisons correction option is recommended for ALE subtraction results?
I understand that you generally recommend Montecarlo FWE correction for CBMA algorithms including ALE (NiMARE: Neuroimaging Meta-Analysis Research Environment ā NiMARE 0.4.1 documentation). Assuming that this #1 choice might not be available for subtraction results, would you prefer any one of the remaining options over the other(s)? I am leaning towards FDR given how conservative Bonferroni FWE correction is, but you may have experience/evidence/literature that suggests taking a different pathā¦
I am happy to provide any more information that may be needed. Thank you very much for your time and your support!
ALESubtraction doesnāt support Monte Carlo FWE correction, which is the one we recommend for CBMA methods, when available. That said, you can use other MCC methods that arenāt Estimator-specific.
You can use the basic FWE and FDR MCC methods. For FWE, thatās just ābonferroniā. For FDR, thatās āindepā or ānegcorrā.
I think thatās a bit of a bug. Optimally, FWECorrector.inspect(subtraction_result) would only return Bonferroni. If you did call ALESubtraction.correct_fwe_montecarlo() directly, youād get a NotImplementedError.
Iād go with the FDR methods for the same reason you brought up.
Thank you very much for your reply, that helps a lot! We will probably FDR-correct our subtraction results then.
If I may ask one follow-up question regarding MCC for within-group ALE analyses:
I have simulated data to configure my analyses before running them on the actual data. My simulated data contains some (convergence) effects that I would expect to show up in analyses. Indeed, the results maps show strong spatial convergence descriptively. However, this convergence does not usually survive Montecarlo FWE MCC. In comparison, it does survive FDR (āindepā method) correction.
My question is which MCC procedure (FDR vs. Montecarlo FWE) to choose for our within-group analyses: On the one hand, FDR appears more liberal and would also be nicely consistent with the between-group / subtraction analyses. On the other hand, you will have good reasons for recommending Montecarlo FWE here (NiMARE: Neuroimaging Meta-Analysis Research Environment ā NiMARE 0.4.1 documentation). If you donāt mind, could you explain a little bit what the reason for recommending Montecarlo FWE is and whether/why it is a better choice than FDR MCC?
Thank you very much, I really appreciate your support!
There are a number of papers on multiple comparisons correction with coordinate-based meta-analysis methods, but two that come to mind are Eickhoff et al. (2016), which used simulations to show that the cluster-level Monte Carlo approach worked best for ALE meta-analyses below a certain number of studies (I think in the hundreds), and after that voxel-level Monte Carlo correction worked best; and Frahm et al. (2022), which looked at threshold-free cluster enhancement with ALE and concluded that it isnāt worth the computational cost. Both of those papers also have robust introductions that will point toward a number of relevant papers.
The basic gist is that methods that arenāt designed for neuroimaging data (like the FDR methods and the Bonferroni FWE method in NiMARE) arenāt as good as the Monte Carlo approach- the FDR methods are more likely to have false positives in particular.
Thank you, Dr. Salo! Eickhoff et al. (2016) helped a lot. I understand now that FDR correction is really not an appropriate choice for ALE CBMA. We might even consider using Bonferroni FWE rather than FDR correction for our subtraction / between-group analyses. We will certainly use cluster-level Montecarlo FWE correction for any within-group analyses.
We will probably FDR-correct our subtraction results then.