An error computing 'AUC_minus_chance' doing multiple class classification

1

Dear tdt experts:
I’m doing a 4 class classification using SVM with linear kernel. My fMRI beta map was extracted by AFNI 3dLSS.
When I set:

cfg.results.output{'accuracy_minus_chance','AUC_minus_chance','confusion_matrix'};

Here comes an error:

Assignment has more non-singleton rhs dimensions than non-singleton
subscripts

Error in AUCstats_matrix (line 24)
label_position(:,i_label) = true_labels == labels(i_label);

Error in decoding_transform_results (line 175)
output =
100*mean(AUCstats_matrix(decision_values,true_labels,labels));

Error in decoding_generate_output (line 50)
output =
decoding_transform_results(curr_output,decoding_out,chancelevel,cfg,data);

Error in decoding (line 558)
results =
decoding_generate_output(cfg,results,decoding_out,i_decoding,curr_decoding,current_data);

Error in main_mvpa_zl (line 69)
results = decoding(cfg);

If I exclude the ‘AUC_minus_chance’, it works perfect.
Thanks in advance!

Best,
Lei

2

Dear Lei,

This is a little weird, and I’m not sure what the issue could be. It looks as if a label doesn’t exist, but that cannot be the case based on how the code is structured. You didn’t use 0 as a label by any chance?

In any case, please send me an email with more details and I will try to figure out the issue.

In any case, in a multiclass case, running AUC in our code equals the average AUC of all pairwise comparisons. The reason is that the typical decision for a class in the multiclass case (e.g. a majority vote) doesn’t make much sense in a binary signal detection framework. The closest to our typical one-vs-one classification for multiclass (in contrast to one-vs-all) is averaging all binary AUCs (micro-averaging).

Martin

3

Dear Martin,
Thanks for reply! I thought I figure out this error by modifying line 24 of the subfunction AUCstats_matrix. However, there’s still something wrong.
I’ll send an email to you with more details today.

Best,
Lei

4

Dear LeiZhang and Martin,

I’m returning to this feed you created, as I just ran into a similar a problem - I’m curious how did you solve LeiZhang’s problem from before.

I’m trying to run a 3 class classification with SVM linear kernel, using t-maps (instead of beta maps). The t-maps were created using fMRIPrep.

My error code;

Warning: More than 2 labels for AUC. Running all pairwise comparisons and averaging (using
AUCstats_pairwise.m).

In warningv (line 72)
In decoding_transform_results (line 190)
In decoding_generate_output (line 35)
In decoding (line 568)
In SMELLS_MVPA_tdt_searchlight (line 96)

Unable to perform assignment because the size of the left side is 84-by-1 and the size of the right
side is 84-by-3.

Error in AUCstats_matrix (line 24)
label_position(:,i_label) = true_labels == labels(i_label);

Error in decoding_transform_results (line 192)
output = 100*mean(AUCstats_matrix(decision_values,true_labels,labels));

Error in decoding_generate_output (line 35)
output = decoding_transform_results(curr_output,decoding_out,chancelevel,cfg,data);

Error in decoding (line 568)
results =
decoding_generate_output(cfg,results,decoding_out,i_decoding,curr_decoding,current_data);

Error in SMELLS_MVPA_tdt_searchlight (line 96)
results = decoding(cfg);

As LeiZhang posted, excluding ‘AUC_minus_chance’ from the cfg.results.output seems to solve the issue also in my case.

Many thanks in advance for your help!

Best, Aino

5

Hi Aino,

Sorry, it was not possible to fix this with Lei since at that time I wasn’t able to reproduce the error but I checked again and it seems that there is a bug in the AUC function for more than two classes.

For now, I would recommend running 3 pairwise analyses and averaging them together, rather than running all in parallel (the result will be the same, with chance = 0.5). The only downside is that the analysis will take a little longer to run. Essentially, you set up you decoding design in a way that it is replicated 3x but always setting one of the three classes to 0. (I’m assuming three labels here, for more it is nchoosek(n,2) replications you need)

Say you have created you decoding design. Then this might work (it’s not very elegant and not sure there might still be a bug, never tested it - but I think it might do the job).
n_columns = size(cfg.design.train,2);
design_orig = cfg.design;
for i_comp = 1:3
design_tmp = design_orig;
design_tmp.train(design_orig.label==i_comp) = 0;
design_tmp.test(design_orig.label==i_comp) = 0;
cfg.design.train = [cfg.design.train design_tmp.train];
cfg.design.test = [cfg.design.test design_tmp.test];
cfg.design.label = [cfg.design.label design_tmp.label];
cfg.design.set = [cfg.design.set design_tmp.set];
end

Best,
Martin

6

Hi Martin,

Thank you for your reply!

I see - this seems like a good workaround for the bug in the AUC function. Thank you very much for your help and for the lines of code, I will be sure to try them!

Best wishes,
Aino