I would like to use recursive feature elimination (RFE) to reduce/equalize the number of voxels in different pre-defined anatomical ROIs. I have 4 sets of runpairs with 25 trials for each condition, so a total number of 100 trials per condition. I understand that if I use RFE, I need to be careful about which sets to use for RFE, training and testing to avoid double dipping/circularity. Following the paper by De Martino et al. (Combining multivariate voxel selection and support vector machines for mapping and classification of fMRI spatial patterns, 2008), I would use the following scheme:
- Select one set as the test set out of the 4
- Out of the remaining 3 sets, select one as the training set for the RFE.
- Do the RFE using the selected training set and testing it on the test set from step 1
- With the selected features from step 3, now do the classification with the remaining 2 training sets, testing on the test set chosen in step 1
- Repeat step 2, choosing a different set for the RFE until all combinations are exhausted
- Repeat step 1, choosing a different set as the test set until all combinations are exhausted
Did I understand this correctly or did I get something wrong?
Thanks for the help!