TDT with fast event-related design - incredibly long processing time

1

Hi,

I’m trying to use TDT to perform classification on a factorial fMRI study that used a fast event-related design. I am using AFNI to do preprocessing, and per Mumford et al. (2012), I am using the LSS method of estimating betas. This leaves me with one beta estimate per trial per condition. I have 8 overall stimulus types, and each was presented 15 times in each of 10 runs. Ultimately, I end up with 1200 beta values overall that I am submitting to TDT to decode (I have collapsed down to two conditions by using wildcards in the label names).

Nothing in the cfg file looks out of the ordinary, and the training/test matrix looks perfectly accurate. However, when I submit it to a searchlight decoding with a mask of ~175k voxels, it estimates it will take a full year per subject. At this point I can’t really tell if something has gone wrong, or if it’s simply that complex computationally. When I preprocess the data using LSA (the default method for 3dDeconvolve) and submit one beta per condition per run to TDT, it takes about 50 minutes to finish a single subject decoding with the same mask.

If anyone has any suggestions, they would be much appreciated.

Than you,
Dave

1 Like
2

Hi Dave,

1200 beta values is a lot and will take a while, but at the same time, one year is a lot, as well, so something seems to be off. Since the runwise LSA already takes so long (I would expect no more than 15 minutes in your case), this already narrows the problem down.

Since you seem quite experienced with Matlab, the first thing you can try is identify the bottleneck yourself, using the Matlab profiler. Type profile on into the command window. Then execute your code, wait for 5-15 minutes (depending on how long it takes to load your data this may otherwise obscure results). Then cancel execution with CTRL + c, and then execute profile off and profile viewer. You will get a nice html through which you can navigate the code tree structure, the number of times each line was executed, and the time each line took to run, to identify the culprit. My guess it is the classifier itself, rather than an integral part of TDT.

For simplicity, keep the problem focused on two classes and LSA at the moment, only then try it with LSS and finally all 8 classes. Also, turn off displaying the searchlight while it is running (should only really affect speed if you plot more than every 100th searchlight). Finally, run only 100 searchlights, which will give you a more precise estimate of the time it takes.

rng(3) % get reproducible subset
cfg.searchlight.subset = sort(randsample(1:170000, 100));  % get random sample

First, try changing the classifier from LIBSVM to the correlation classifier and see if it speeds things up (else this might be a more complicated problem).

cfg.decoding.software = 'correlation_classifier;
cfg.decoding.method = 'classification';

If this speeds things up, then switch those options back to 'libsvm' and 'classification_kernel' and try scaling the data. The reason is that the classifier likes input data that is somewhat in a small range, and perhaps LSS and LSA don’t do that for you, but you can achieve it by scaling (e.g. by z-transforming or by scaling between 0 and 1). If it is not scaled, sometimes the classifier can take a long time.

cfg.scale.method = 'z';
cfg.scale.estimation = 'all';

If this improves LSA, this will likely also improve LSS. Maybe not sufficiently, though, because you still have an interesting case: usually the number of dimensions (i.e. voxels, P) is smaller than the number of data points (i.e. beta maps, N), making it super easy for the SVM to find a solution. When P > N, will take longer to find a solution. A simple workaround is to slightly reduce the C parameter to 0.1 or 0.01 or even 0.001, which normally shouldn’t strongly affect your results but can work miracles. It might in this case also make sense to compare the speed for running classification vs. classification_kernel, where kernel is usually faster.

Let me know if any of this helped or whether you are still stuck at slow execution. I think it should in total not take longer than a few hours to run, and there are other strings to pull to speed things up.

Best,
Martin

3

Hi Martin,

Thank you so much! Changing the decoding software to ‘correlation_classifier’ made a dramatic difference, taking the LSA and LSS method analysis down to about 15 minutes. This is not a surprise because when I checked the MATLAB profile viewer, LibSVM was the culprit. I am still kind of curious why the SVM method is taking such a long time, but for now I am happy with the correlation classifier method.

Scaling the data did not appear to significantly change processing time. Lastly, I didn’t try changing the C parameter yet because I was unsure where to change it.

4

Hi Dave,

Glad it helped! I can recommend using LIBSVM, it should generally perform equally well or better. If it’s not related to scaling, then perhaps to the fact that you have more data points than dimensions. There are other SVM implementations out there that could be faster (e.g. Pegasos SVM), and I have implemented an alpha version I’m happy to share with you in case you want to try it (PM me).

For now, I would just give it a try with setting c = 0.01 or 0.001 and seeing what happens. In your script, set:

cfg.decoding.train.classification_kernel.model_parameters = '-s 0 -t 4 -c 0.001 -b 0 -q';

(remove the kernel part if you are using only classification without the kernel speed-up).

Best,
Martin

5

One last question;

I was reading a bit and it seems that LibLinear might be more appropriate for a set of this size. I see that LibLinear appears to be incorporated into TDT to some extent, would you recommend using since libSVM seems untenable?

Thanks again!

6

Sure, give it a try. I’m not sure it will be much faster. For all the options, add TDT to your path and type train into the command window (which will show the help for liblinear). I think the best one would be -s 1, which is also the default.

cfg.decoding.method = 'classification';
cfg.decoding.software = 'liblinear';
cfg.decoding.train.classification.model_parameters = '-s 1 -c 1 -q';

For other classifiers, go to the folder decoding_software, but for the model parameters you need to figure out the software-specific settings in the subfolders.

Best,
Martin

7

Another update:

Changing the C parameter shortened the run time to 16 hours for libSVM when decoding the individual trial data, which is far more reasonable than the previous 1.5 years. However, liblinear took 2 hours on the same dataset. I have not compared classification accuracy for the three different implementations I’ve tried so far, which is the next step. Hopefully this information will be helpful to people in the future trying to run decoding on large, fast event-related designs (which are common in my area of speech perception).

I don’t want to make a separate thread if it isn’t necessary, but I am getting an error that I can’t quite figure out related to TDT writing the results after the decoding finished successfully. I’m getting the following error:

File name or format to be written is neither 
+orig nor +acpc nor +tlrc (filename: , .
Required for writing AFNI files. Unsupported scene data (this is a
field in the BRIK header) for  .../res_accuracy_minus_chance+orig.BRIK

I checked the input betas I’m giving to TDT and the AFNI header file says they’re in original space (+orig). Any ideas about how to troubleshoot this?

Thanks again for your continued help!

1 Like
8

I’m really glad liblinear turned out to be even faster! Thanks for comparing speed, I would go with liblinear in that case, because C = 1 is probably better. I guess all three results will be quite similar, though.

It was quite hard to interface TDT with AFNI, so I kept it in beta mode. For data sets processed the standard way, this should work. But it is possible that your data is somewhat different, e.g. by using LSS. I’m happy to help you troubleshoot, but let’s take this offline.

Martin

9

Ok, problem solved. Result of troubleshooting was that somehow there was a blank added after 'BRIK ', which is obviously difficult to see and made it impossible to write the image. I couldn’t trace the origin of this to TDT or AFNI_matlab (and it normally works well), and Dave checked his code and couldn’t find anything wrong, so possibly it’s related to LSS in AFNI. Anyway, in the most recent version of TDT, I added a check that now solves this problem.