Nilearn searchlight variance cross folds

, ,
1

Hi experts,

I’m trying to run a nilearn.decoding.Searchlight cross subject level but found that the function does not give individual score on each fold or some kind of a variance output. It only outputs an averaged accuracy score as searchlight.scores_. Is that right?

I am asking because I want to perform a simple t-test against chance level (I understand eventually a permutation test is what we want but just for now) and it’s not possible without some kind of a variance score. Specifically, I’m imagining something like when you run a classifier in nilearn.decoding.decoder, decoder.cv_scores_ gives you the individual accuracy score on each fold, so we can calculate the variance. Thanks for any input!

2

Hi, that is correct it only gives the mean score.
as cross-validation folds are not independent, you cannot do a t-test or estimate the variance of generalization score from cross-validation scores.
adding permutation tests is being discussed, see this issue

1 Like