Group level inference with limited number of first level permutation

1

Hi Martin and all,

I have a first level cross-validation decoding desing including 4 fMRI runs/chunks (one beta per run).
Like this the maximal number of first level permutation designs is restricted to 8 per participant.

How would you proceed for the 2nd level statistics? Would you consider a bootstrapping procedure to create enough data for a population null distribution for the second level analysis?

Many thanks for your advice,

Best regards,

Basil

2

Hi Basil,

For second-level statistics, you can normally assume that your decoding-level accuracies are not positively biased (i.e. that chance level is really chance and that effects don’t seem larger than they really are). That means you can run a simple t-test. When people tell you that you must use a permutation test, that only applies to the decoding-level results, because cross-validated results are not normally distributed.

The caveat is that the hypothesis you are testing at the second-level is not a random effects test, but turns out to be a fixed effects test. The reason is that the null distribution cannot go below chance (true below chance does not exist). And since chance is 50%, this leads to a distribution with zero variance. For that reason, all remaining variance has to come from the within-subject variability. So, in essence your test assumptions that you can ignore the subject-level variability by running a random effects test are violated. This is all explained in detail here. You can also have a look at my video lecture on MVPA statistics here.

The solution is to run permutations at the decoding-level and test if a notable proportion of participants is carrying an effect. This does not work for the type of analysis you ran since there are too few permutations. I know that Carsten is currently working on a statistical testing procedure that should allow you to run valid permutation testing at the level of the model estimation. Until then I would suggest using a t-test and making sure you mention in your publication that this is in effect a fixed-effects test, citing the above reference.

Best,
Martin

1 Like