TDT: How to realize c-optimization via grid search and nested cross-validation for cross-classification

Software Support
1

Summary of what happened:

Hi!

We want to optimize the cost parameter c for multi-class SVM via grid search and nested cross-validation. This worked fine for a “normal” classification design with cross-validation, when calling it like this:

cfg.parameter_selection.method = 'grid';
cfg.parameter_selection.parameters = {'-c'};
cfg.parameter_selection.parameter_range = {[0.0001 0.001 0.01 0.1 1 10 100 1000]};

Now we would like to do the same in a cross-classification. Here we receive the following error message:

Command used (and if a helper script was used, a link to the helper script or the command generated):

Version:

Relevant log outputs (up to 20 lines):

Error using decoding_parameter_selection>run_nest (line 207)
Could not create design for nested cross-validation. Need correct information in field 'cfg.parameter_selection.design.function!'

Error in decoding_parameter_selection (line 116)
selected_parameters = run_nest(cfg,data_train,i_train_external,all_combinations);

Error in decoding (line 471)
cfg = decoding_parameter_selection(cfg,data_train,i_train);

Error in D5_Decoding_batch_XClass (line 145)
results = decoding(cfg);

Screenshots / relevant information:

We’ve already tried assigning the field cfg.parameter_selection.design.function different outputs from the function make_design_xclass_cv(cfg), receiving the same or similar errors.

So the question is: Is this even possible? Do we use the function incorrectly or do we assign something the wrong way? To our understanding, the grid search for the optimal c-parameter should work the same for the cross-classification as it did for the normal design?

Any help would be appreciated!
Thanks in advance :slight_smile:

Best,
Sara

2

Hi Sara,

Nice work! I am not entirely certain since it has been quite some time since I wrote this code but I would try cfg.parameter_selection.design.function = 'make_design_cv';

Essentially, nested cross-validation needs to know how the internal cross-validation should be performed. I think it takes a guess by doing the same as the outer CV but this probably doesn’t work. Now, for make_design_cv, of course you would have to have separate chunks in your training data, else this cannot work.

See if this works and if not I’ll dive more deeply into this.

Best,
Martin

3

Thanks! That was quite helpful :slight_smile:
Although cfg.parameter_selection.design.function = ‘make_design_cv’; gave a similar error, cfg.parameter_selection.design.function.name = ‘make_design_cv’; worked!

Thank you very much!

Best,
Sara

1 Like