Nilearn searchlight variance cross folds

Hi experts,

I’m trying to run a nilearn.decoding.Searchlight cross subject level but found that the function does not give individual score on each fold or some kind of a variance output. It only outputs an averaged accuracy score as searchlight.scores_. Is that right?

I am asking because I want to perform a simple t-test against chance level (I understand eventually a permutation test is what we want but just for now) and it’s not possible without some kind of a variance score. Specifically, I’m imagining something like when you run a classifier in nilearn.decoding.decoder, decoder.cv_scores_ gives you the individual accuracy score on each fold, so we can calculate the variance. Thanks for any input!

Hi, that is correct it only gives the mean score.
as cross-validation folds are not independent, you cannot do a t-test or estimate the variance of generalization score from cross-validation scores.
adding permutation tests is being discussed, see this issue