I have a general question regarding cross-classification designs using the TDT toolbox. Between using make_design_xclass and make_design_xclass_cv, is there a general rule on which type of designs would be preferable to use one versus the other? I’ve seen some posts here noting that xclass is preferable when there are an unbalanced number of labels/runs, but is that the only situation? Are there others were xclass is better? Does xclass_cv require more runs/trials, or is xclass more prone to errors? Any feedback would be great, thanks!
Great question! I personally like xclass if you just want to cross-classify: It takes all training data, runs a classifier, and applies it to all test data. That way, it only requires one iteration (well, actually two, because it reverses training and test data, if you don’t want that, you need to set oneway = 1).
I created xclass_cv for two cases: (1) you don’t trust that data associated with test labels in each run (in our lingo chunk) is independent from the data associated with the training labels in each run, i.e. there may be leakage, and (2) the case when you would like to compare results to classical cross-validation. In the latter case, we usually have a leave-one-run-out design. Since xclass may give you more training/testing samples, it might give you a more powerful estimate of whatever you are interested in. To get the same statistical power, you can make things more comparable by using xclass_cv. In practice, I haven’t seen much of a drop in performance, though.
I have played around with it a bit, and have one additional question. When setting twoway=1, I had expected that I would get two outputs, one for each swap of the training/testing classes, but I only get one output. Does that then mean that this output is from the combination of both? (so any significant clusters here would represent regions where decoding successfully worked both ways?). When the image designs pop up however, it only shows train->test in one direction, and so I still have to swap the labels in the cfg to have it go the other way. Thanks!
Just wanted to bump the above question. Could you explain a little bit what the results of twoway=1 are supposed to be? The results only output a single .nii file, which could be the combination of decoding in both directions (so an invariant representation in either direction), or a bug (because the image and text files output only ever show decoding in one direction). Thanks!
Sorry for the delay, things are quite busy at the moment. If you like to inspect both directions separately, you could either run two analyses setting twoway = 0, of alternative add the set variable to your design, setting it to [1 2]. That would lead to separate results for each of the two iterations.
The reason twoway exists is because most people are interested in whether one condition generalizes to the other, irrespective of direction. Averaging across both should give you higher statistical power.
I’m not sure why your analysis had only one column despite setting things to twoway, are you sure it was really set that way / really only one column? Could you check if the results are identical when setting twoway = 0?
No worries on the delay! I completely understand how crazy things are right now
Glad to hear that twoway does what I assumed it was doing.
Unfortunately, it doesn’t seem to be doing that for me. I tried setting twoway=0 and got identical results to when I had twoway=1. For some reference, I have two conditions, and within each condition there are 7 labels, with an equal number of trials/label. Here is how I defined everything:
This way will do training of set 1 and testing on set 2. If I swap the xclass labels (say, make it [2 2 2 2 2 2 2 1 1 1 1 1 1 1 ], then train->test goes in the opposite direction. But again, what I’d like to do is combine these together. Just curious - if I were to simply average the two unidirectional .nii files for each train->test direction, would this give me the same thing? Thanks for all your help!
