Summary of what happened:
Dear @Martin,
I am using the decoding_between_group.m template for a two-class between group searchlight SVM classification. I have a sample of 60 participants who performed a threat escape task. 30 participants belong to class 1 (parents) and the other 30 to class 2 (non-parents). I am not chunking the data, as we do not have specific pairs that belong together. Instead, I use the make_design_boot function to randomly make pairs from our sample. I calculated in MATLAB with the combinations() function that all possible unique pairs are 900. With that in mind, I made a design with 900 iterations. However, the resulting design shows that some pairs occur multiple times, while others do not. This also means that some participants occur more often in the test set than others (lowest = 18, highest occurrence = 42). On another try, the differences in occurrence per participant were smaller but still there.
I got the advice that such a design is not preferred in machine learning, as a potential ‘bad pair’ may contribute more to the model cross-validation than another. Or the other way around with a ‘better paiur’. Also, it does generally not make sense to cross-validate on the same training and test set I guess.
I was given the option to do a between groups design with leave-one-subject out. To my understanding, TDT between_group script does not allow this as it logically expects a pair to be left out. However, with leaving one subject out, the pair occurrence does not matter anymore and we can test whether the model is able to go against its majority group bias (eg. 30 parent vs. 29 non-parent participants in training set). Q: Is this an option with the TDT?
From a machine learning perspective, we could also randomly select a pair for 30 iterations so that every participant has been in the test set once. However, with the variability and complexity in human fMRI data, we think that exhausting all possible pairs is best. Another option would therefore be to manually create a design with 900 iterations where we put in the restriction that each unique pair can only occur once. Q: What cross-validation design do you think is best to use here?
If useful, see a snippet of the code below.
Thanks in advance for your reply!
Best regards,
Florien
Command used (and if a helper script was used, a link to the helper script or the command generated):
...
% You can type in help decoding_defaults and it shows the default setting +
% additional settings that can be added to this script.
%
% Set defaults
cfg = decoding_defaults;
% Set the analysis that should be performed (default is 'searchlight')
cfg.analysis = 'searchlight';
%Make a cell array with all contrast maps of the 60 participants. To check
%whether the right contrast maps are selected, the maps are stored in an
%extra variable with the subject number (and later on also whether they are
%a parent or non-parent.
cMaps= cell(60,1);
checkCmaps = cell(60,2);
for i = 1:length(sub)
cMaps{i} = [data_dir, sub{i}, '\', 'con_0009.nii'];
checkCmaps{i,1} = sub{i};
checkCmaps{i,2} = cMaps{i};
disp(['contrast map of participant ', sub{i}, ' selected'])
end
parClass = [ones(3,1);-ones(2,1);ones(4,1);-1;ones(2,1);-ones(3,1);1;-1;1;1;-1;ones(3,1);-1;ones(6,1);-1;ones(7,1);-ones(6,1);1;-ones(13,1);1;-1];
checkCmaps = [checkCmaps, num2cell(parClass)];
%% Adding contrast maps, labels, and chunks
cfg.files.name = cMaps;
% and the other two fields if you use a make_design function (e.g. make_design_cv)
% (1) a nx1 vector to indicate what data you want to keep together for
% cross-validation (typically only matched controls in between-group,
% because subjects are else independent). If you don't have an obvious way
% to create chunks, set all participants to 1 (e.g. cfg.files.chunk = ones(n,1) )
%F: Parent and non-parent labels assigned to parClass. Will automatically follow the order of the files that were put in to the decoding setup.
cfg.files.chunk = ones(60,1);
%F: There are no specific pairs of parents and non-parents that need to stay
%together in either training or test set.
% (2) any numbers as class labels, normally we use 1 and -1. Each file gets a
% label number (i.e. a nx1 vector)
% F: parents and non-parents separated in parClass, automatically corresponding to the order of the files that were put in.
% parent == 1, non-parent == -1.
cfg.files.label = parClass;
% Set additional parameters manually if you want (see decoding.m or
% decoding_defaults.m). Below some example parameters that you might want
% to use:
cfg.searchlight.unit = 'mm';
cfg.searchlight.radius = 12;
% this will yield a searchlight radius of 12mm. Our smoothing kernel is 4 mm
% so that includes 3 voxels in each sphere. For subcortical we don't wat it
% to be too big, for amygdala, PAG. But not too small either for the larger structures
% --> leads too many clusters in one structure.
cfg.searchlight.spherical = 1;
cfg.verbose = 2; % you want all information to be printed on screen
% cfg.decoding.train.classification.model_parameters = '-s 0 -t 0 -c 1 -b 0 -q';
% Enable scaling min0max1 (otherwise libsvm can get VERY slow)
% if you dont need model parameters, and if you use libsvm, use:
cfg.scale.method = 'min0max1';
cfg.scale.estimation = 'all'; % scaling across all data is equivalent to no scaling (i.e. will yield the same results), it only changes the data range which allows libsvm to compute faster
% Decide whether you want to see the searchlight/ROI/... during decoding
cfg.plot_selected_voxels = 25; % 0: no plotting, 1: every step, 2: every second step, 100: every hundredth step...
% Add additional output measures if you like
cfg.results.output = {'accuracy_minus_chance', 'AUC_minus_chance'};
% Assuming there are no matched controls between groups, the way in which
% data are split up is arbitrary. For that reason, you can repeatedly
% subsample from both groups, in this case 100 times. This creates the
% leave-one-pair-out cross validation design with 100 decoding steps:
cfg.design = make_design_boot(cfg,900,1); % the 1 keeps test data balanced, too
% If you used a bootstrap design, then you might speed up processing using
% this function:
cfg.design = sort_design(cfg.design);
Version:
Environment (Docker, Singularity / Apptainer, custom installation):
Matlab 2024a, spm12, tdt_3.999I
Data formatted according to a validatable standard? Please provide the output of the validator:
PASTE VALIDATOR OUTPUT HERE
Relevant log outputs (up to 20 lines):
PASTE LOG OUTPUT HERE