TDT permutation test

Dear Martin and all,

I am trying first to do permutation test and then do the prevalence inference.

when I do permutation test using demo9_permutation_analysis, the TDT toolbox asked me to input the cfg.boot.n_boot = ? , I have two groups and every group have 40 participants.
do I need to input 1 or 80?
what does n_boot mean?
when I try to input cfg.boot.n_boot = 1 or cfg.boot.n_boot = 80, it goes all right.
it said:
creating permutation designs…
design for CV decoding for 80 files X 1(80) steps created

when input 1, it finished several hours.
when input 80, it may need several days.

which one is right?


Yongming Wang

Hi Yongming,

This seems to be unrelated to permutation testing but more to the decoding design being created in the first step when you do your original analysis before permutation testing. I would look into the example in make_design_permutation, that should provide all detail. What TDT does is figure out what design you initially created (e.g. leave-one-run-out using make_design_cv) and will then implement the same analysis but permuting labels within run (or if you assume full exchangeability setting cfg.permute.exchangeable = 1; and then running the permutation test that way. But in any case, it will run the same analysis you did originally. Since you probably created you design using make_design_boot or make_design_boot_cv, TDT cannot know how many steps you originally put in (since this is called separately) so you need to provide it with that number again.

For permutation testing, usually 1,000 permutations is a good rule of thumb (depending on the size of the effect less or a lot more). And yes, this takes forever to run if you run a searchlight analysis (about 50-100 hours per subject if you use SVM and depending on the size of the data).