Get models from TDT

tdt
#1

Hi TDT experts,

I am a new user of TDT. After finished some practices following decoding tutorial I have a confusion. Can I get the proper model after cross-validation to test on another independent testing data? Is it reasonable for my thinking? If it is possible which function in TDT should be used?

Best,
Xinqi

0 Likes

#2

Dear Xinqi,

I understand your confusion. Usually, when running machine learning classification, you first run cross-validation to optimize hyperparameters and then apply them to independent test data. In decoding for studying brain function, the goal is usually not to build a classifier, but to know whether there is statistical dependence between the labels and the data. Usually, we don’t optimize the hyperparameters (e.g. c in the support vector classifiers), and if we do, then we use nested cross-validation. In contrast, we usually use cross-validation on all data, only to get estimate of the information content that is not positively biased, i.e. there is no separate test set. See my paper that explains this distinction in more detail (if you have no access, the preprint is here) .

Now, in TDT you will get a model for each different cross-validation iteration, so which one is the “correct” one to apply to separate data? Well, assuming that your estimate of the information content is fine (i.e. you are happy with the accuracy for a given set of hyperparameters), you can just run TDT on all your data as training data and set separate test data as test data. You just cannot use the automatic routines, because with separate data you need to determine the file names, labels, and chunks manually. But that should all be covered in the tutorial and the templates.

If you just want to inspect the model parameters, set
cfg.results.output = {'accuracy_minus_chance', 'model_parameters'};

Best,
Martin

0 Likes

#3

Dear Martin,

Thanks for your detailed explanation! It really solved my confusion.

Best,
Xinqi

0 Likes