Question about recursive feature elimination and cross-validation

decoding
scikit-learn
mvpa
roi-based-decoding
#1

Hello,

I would like to use recursive feature elimination (RFE) to reduce/equalize the number of voxels in different pre-defined anatomical ROIs. I have 4 sets of runpairs with 25 trials for each condition, so a total number of 100 trials per condition. I understand that if I use RFE, I need to be careful about which sets to use for RFE, training and testing to avoid double dipping/circularity. Following the paper by De Martino et al. (Combining multivariate voxel selection and support vector machines for mapping and classification of fMRI spatial patterns, 2008), I would use the following scheme:

  1. Select one set as the test set out of the 4
  2. Out of the remaining 3 sets, select one as the training set for the RFE.
  3. Do the RFE using the selected training set and testing it on the test set from step 1
  4. With the selected features from step 3, now do the classification with the remaining 2 training sets, testing on the test set chosen in step 1
  5. Repeat step 2, choosing a different set for the RFE until all combinations are exhausted
  6. Repeat step 1, choosing a different set as the test set until all combinations are exhausted

Did I understand this correctly or did I get something wrong?

Thanks for the help!

-Johannes