Dear TDT community,
I have some questions regarding significance tests for different within-subject classification analyses within ROIs. Our task design is as follows: We acquired two task conditions presented blockwise for 5 times with the same two stimulus conditions in both tasks (with 2 stimuli per stimulus condition and task condition) in one run. Next, we set up GLMs with a separate regressor for each stimulus, resulting in one beta image per stimulus.
Because we don’t have different runs to use for cross-validation, we set up the analyses with the task blocks as chunks (“leave-one-block-out CV”) as they are apart in time. One block also contained further task conditions, which are not of interest here. My first question is whether this approach is valid or whether the blocks are not independent enough to treat them as different chunks and use for CV?
As a sort of sanity check we first classified the different task conditions. For each block, we have 4 stimuli per task condition in a 5-fold CV design (here, the stimuli were labeled according to the task they belonged to, not the stimulus condition). In each fold, the training set thus consisted of 32 stimuli and the test set of 8.
Next, we classified the stimulus conditions separately for both tasks (“within-modality”). For these analyses, we have 2 stimuli per stimulus condition also in a 5-fold CV design. Thus, we have 16 stimuli as a training set and 4 stimuli as a test set per fold.
For these three analyses, we were planning to use the prevalence inference approach incorporated in the TDT toolbox. The first question here is, whether we have to permute the labels within chunks (meaning blocks here) for the permutation on the subject-level, as has been suggested for leave-one-run-out CV? Is such a block permutation implemented in the toolbox? And would this still provide us with enough permutations to test for significance?
And lastly we classified the stimulus conditions across task conditions (“cross-modal”). For this analysis, we trained on all stimuli of one task condition and tested on all stimuli on the other task condition (i.e., no CV). Is this approach valid regarding the dependencies of stimuli within block? How would you recommend to test for significance in this analysis? On a subject-level we could probably use a binomial test but how about the group-level that we are interested in?
Thanks and best,
Lara