Select the best model for first level analyses on Nilearn (find AIC, BIC?)

Software Support
,
1

Summary of what happened:

Hello all,
I am doing an event related design at 7T and want to optimise the fit of my first level model.
For now, this is how I am doing:

design_matrices = []

for idx, img in enumerate(fmri_imgs):
    events = event_files[idx]
    motion = movements[idx]
    n_scans = img.shape[-1]
    frame_times = np.arange(n_scans) * t_r

    design_matrix = make_first_level_design_matrix(
        frame_times,
        events,
        hrf_model='glover', 
        drift_model='polynomial',
        drift_order =3,
        add_regs=motion.values,           
        add_reg_names=motion.columns.tolist()
    )

    design_matrices.append(design_matrix)

and then I fit my model:

fmri_glm = FirstLevelModel(t_r=1.4, signal_scaling=False, smoothing_fwhm=2)
fmri_glm = fmri_glm.fit(fmri_imgs, design_matrices=design_matrices)

I am trying to know which model I should do to optimize my contrasts.

I don’t really find information on what’s best between glover, or glover+derivatives? I hear all different opinions (maybe derivatives since it is a fast event related design?)
And also, which regressors to put? 6 motion parameters? 12? 24? And WM?

I try to change parameters to optimise my constrats in one subject, and then when I fit the model to another subject it is not optimal. So I try other parameters, but these are not good for other sub.

So, I am wondering, can I change my regressors depending on subjects? Not the hrf model, drift model, and drift order of course, but the movements, CSF and WM?

And, is there a way to see which model fits best (i.e., a AIC or BIC) from the model outputs? If yes, maybe I should fit model with different movement regressors per subject and keep only the ones that fit best (i.e., lower AIC and BIC…?)

Thank you!

Best,

Alexane

2

@Alexane_Leclerc
I edited a bit your post for better rendering of the python snippets.

1 Like
3

Nilearn has no formal way of doing model comparisons.

I have been meaning to at least had an easy way to get a “goodness of fit” in the GLM reports but still have not got to it: [ENH] Include "goodness of fit" image in GLM reports ? · Issue #4525 · nilearn/nilearn · GitHub

As far as I know the only tool that does formal model comparison in fMRI is the MACs toolbox for SPM: GitHub - JoramSoch/MACS: MACS – a new SPM toolbox for model assessment, comparison and selection

I don’t know if AFNI and FSL have things like those.

So for this question, if you ask 6 neuroimagers, you will get 6 different answers… Possibly more if one of them is also a statistician. :person_shrugging:
Even nilearn offers a bunch of different strategies for loading confounds: nilearn.interfaces.fmriprep.load_confounds_strategy - Nilearn

General advice

  1. I am not aware of a confound strategy that works for ALL datasets. And I would not be surprised that most people advises on this are based on a mix “I was taught this way” / “I have always done it this way.” / “I eyeballed the difference between different models on 2 subjects 10 years ago.” rather than on careful systematic investigation. There are defo some sub-field ‘fashions’: I have seen quite a few high-res 7T task based papers that don’t add any motion regressors and barely care about detrending, wheres resting-state folks care A LOT about those things…

  2. Be careful to not overfit! Ideally you would use one dataset to optimize your design and another dataset to get actually estimates. But because it’s fMRI, we can rarely afford to do that, so you can at least optimize on a contrast that is mostly orthogonal to your contrast of interest (something like the button presses of your participants - assuming they are not too correlated with your contrast of interest; or if you are interested on A-B, you can optimize on A+B)

4

I know of some papers look more systematically into this but I can’t recall them from the top of my head.

5

Hi,
Here are some personal takes on this question:

  • Including additional nuisance parameters in the design matrix is in general a good idea. Simply make sure that it remains non-singular
  • AIC, BIC are asymptotic criteria that assume iid data; your data are relatively short time series wuth autocorelation. So AIC/BIC won’t be helpful.
  • the best possible approach would be to run cross-validation if you have repeated runs: take the estimated parameters in one run to see how well it performs in another run. But it complex to set up and expensive, so that very few people do that.
  • I strongly advise to keep the same settings for all subjects: otherwise you need to find a good reason to change them (without overfitting, i.e. picking a posteriori the results you prefer).
    My 2c,
    Bertrand
6

Thank you for your anwer!
Indeed, I talked with another 7T researcher who advised to do the minimum (i.e., only 6 motion regressors, no smoothing, no CSF, no WM, no mask…)
And I will definitely try the second part of point 2: optimise a simple contrast such as button presses. I thought about comparing all my conditions to the null events (which are just grey background without stim to add random jitter) to see if I have all my activations in faces regions, but it is not ideal… I don’t really have a simple control condition. Even happy>sad faces is not done a lot in the literature… I need to think about a simple way

Best,

7

Thank you for your answer!
Good idea, I will check for colinearity in my design matrix. Maybe including too many motion parameters+derivatives is too much!
I take into account your other points!
Best,

8

last version of nilearn has a plotting tool to help viewing this: nilearn.plotting.plot_design_matrix_correlation - Nilearn