Reuse and rerunning with fMRIPrep 25.2.2

[jaetzel]

Summary of what happened:

1: Ran fMRIPrep to produce surface (gifti, fsaverage5) and volume (nii.gz, MNI) outputs.
2: Kept all directories and files (fmriprep_25_2_wf, derivatives, BIDS) in place; made a copy of the derivatives subdirectory.
3: Added the bold, sbref, and fmaps for one additional run to the BIDS (input dataset) directory and ran fMRIPrep again with the same command as before (all previous files still present). The new run had its own pair fieldmaps (with their own unique B0 name); its fieldmaps were not used elsewhere.
4: Examined and compared derivatives and reports from before and after the second fmriprep job.

Since the previous freesurfer and working files were present, I expected fmriprep to reuse them, and only process the new run’s files. The second job finished much, much faster than the first, and as far as I can tell freesurfer wasn’t rerun. I was surprised, though, that nearly all of the derivatives had new timestamps; I expected files for the new run to be added to the existing func and fmap derivatives directory, not for (almost) all of the previous files to be replaced. (“Almost” since the surface anat derivatives appear unchanged.)

Question: Should I have been surprised? (Hunting through the documentation, I saw a few mentions that many pipeline components always run, even when unchanged, which implies I should expect the “normal” amount of fmriprep variability in output.)

Comparing the visual reports and outputs, most were reasonably similar (the normal few decimal places different) … except for one run. For example, the DVARS & FD were much bumpier in the first set of derivatives than the second:

Perhaps the sub-TD_ses-TD7T1_task-rest_run-10_desc-hmc_boldref.json file is a hint of what happened:
first set of derivatives:

{  "Sources": [   "bids:raw:sub-TD/ses-TD7T1/func/sub-TD_ses-TD7T1_task-rest_run-10_echo-1_bold.nii.gz"  ] }

second:

{   "Sources": [ "bids:raw:sub-TD/ses-TD7T1/func/sub-TD_ses-TD7T1_task-rest_run-10_echo-2_bold.nii.gz" ]  }

run-10 is multi-echo, but only two echoes, so the T2starmap wasn’t created. Maybe an echo is chosen at random for use in some steps? The sub-TD_ses-TD7T1_task-rest_run-10_space-MNI152NLin2009cAsym_desc-preproc_bold.json files are identical for the two jobs, both have “bids:raw:sub-TD/ses-TD7T1/func/sub-TD_ses-TD7T1_task-rest_run-10_echo-2_bold.nii.gz” as the first Sources. The preprocessed images are not catastrophically different at first look, but more variable (between the two fmriprep jobs) than the other runs.

Thanks!

Command used (and if a helper script was used, a link to the helper script or the command generated):

/app/.pixi/envs/fmriprep/bin/fmriprep --fs-license-file /tmp/.license/freesurfer/license.txt -w /tmp /tmp/BIDS /tmp/derivatives participant --participant_label TD --output-spaces fsaverage5 MNI152NLin2009cAsym --n-cpus 16 --omp-nthreads 4 --mem-mb 64000 -v

Version:

fMRIPrep version 25.2.2, FreeSurfer 7.3.2

Environment (Docker, Singularity / Apptainer, custom installation):

singularity

Data formatted according to a validatable standard? Please provide the output of the validator:

first fmriprep job:

You are using fMRIPrep-25.2.2, and a newer version of fMRIPrep is available: 25.2.3.
Please check out our documentation about how and when to upgrade:
https://fmriprep.readthedocs.io/en/latest/faq.html#upgrading
251106-09:59:07,528 cli INFO:
	 Telemetry system to collect crashes and errors is enabled - thanks for your feedback!. Use option ``--notrack`` to opt out.
251106-09:59:16,469 cli INFO:
	 Making sure the input data is BIDS compliant (warnings can be ignored in most cases).
bids-validator@1.14.10
(node:39558) Warning: Closing directory handle on garbage collection
(Use `node --trace-warnings ...` to show where the warning was created)
e[32mThis dataset appears to be BIDS compatible.e[39m
        e[34me[4mSummary:e[24me[39m                e[34me[4mAvailable Tasks:e[24me[39m        e[34me[4mAvailable Modalities:e[39me[24m 
        62 Files, 3.58GB        rest                    MRI                   
        1 - Subject                                                           
        1 - Session                                                           
...

251107-13:21:49,238 nipype.workflow IMPORTANT:
	 Works derived from this fMRIPrep execution should include the boilerplate text found in <OUTPUT_PATH>/logs/CITATION.md.

second fmriprep job (same directories as above, plus BIDS bold & sbref, fmaps for one additional run):

You are using fMRIPrep-25.2.2, and a newer version of fMRIPrep is available: 25.2.3.
Please check out our documentation about how and when to upgrade:
https://fmriprep.readthedocs.io/en/latest/faq.html#upgrading
251120-03:48:12,746 cli INFO:
	 Telemetry system to collect crashes and errors is enabled - thanks for your feedback!. Use option ``--notrack`` to opt out.
251120-03:48:22,366 cli INFO:
	 Making sure the input data is BIDS compliant (warnings can be ignored in most cases).
bids-validator@1.14.10
(node:11003) Warning: Closing directory handle on garbage collection
(Use `node --trace-warnings ...` to show where the warning was created)
e[32mThis dataset appears to be BIDS compatible.e[39m
        e[34me[4mSummary:e[24me[39m                e[34me[4mAvailable Tasks:e[24me[39m        e[34me[4mAvailable Modalities:e[39me[24m 
        70 Files, 3.65GB        rest                    MRI                   
        1 - Subject                                                           
        1 - Session 
...

251120-04:56:44,99 nipype.workflow IMPORTANT:
	 Works derived from this fMRIPrep execution should include the boilerplate text found in <OUTPUT_PATH>/logs/CITATION.md.

Relevant log outputs (up to 20 lines):

PASTE LOG OUTPUT HERE

Screenshots / relevant information:

---

[jaetzel]

Revisiting, it looks like part of the answer is that, when given a two-echo dataset, fmriprep randomly picks which echo to process, at least for some steps. The visual report html states this, though I’d missed it last week:

Top image listing echo 1 is from the first time fmriprep was run; lower with echo 2 the second, when fmriprep was run again on the same (including input, output, and working files) directory structure, but after adding BIDS input images for an additional run. (Echo 2 was listed again in a third test; this time in a new directory, providing the freesurfer outputs but no derivatives and no working files.)

I wonder, though, if the echo listed in the Report is actually used for all processing steps. The BOLD summaries are quite different in the “echo 1 processed” and “echo 2 processed” output html (as are the input images from the two echoes), but the three sets of final preprocessed _bold images are very similar (below), and their .jsons are identical, all listing echo-2 in the sources:

sub-TD_ses-TD7T1_task-rest_run-10_hemi-L_space-fsaverage5_bold.json

{
  "AcquisitionDuration": 607.062,
  "DelayTime": 0.08999999999999986,
  "RepetitionTime": 1.15,
  "SliceTimingCorrected": true,
  "Sources": [
    "bids:raw:sub-TD/ses-TD7T1/func/sub-TD_ses-TD7T1_task-rest_run-10_echo-2_bold.nii.gz",
    "bids::sub-TD/ses-TD7T1/func/sub-TD_ses-TD7T1_task-rest_run-10_from-orig_to-boldref_mode-image_desc-hmc_xfm.txt",
    "bids::sub-TD/ses-TD7T1/func/sub-TD_ses-TD7T1_task-rest_run-10_from-boldref_to-T1w_mode-image_desc-coreg_xfm.txt",
    "bids::sub-TD/ses-TD7T1/anat/sub-TD_ses-TD7T1_from-fsnative_to-T1w_mode-image_xfm.txt"
  ],
  "StartTime": 0.53,
  "TaskName": "rest"
}

Shouldn’t run-10_echo-1 be listed as a source for the “only echo 1 processed” job?

Frame 200 of sub-TD_ses-TD7T1_task-rest_run-10_space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz from the three rounds of preprocessing, with matching color intensity ranges. The far left (with pink 1) is from the job listing echo 1 in the Report; the other two listed echo 2. All three images are quite similar:

And here is frame 200 of the two input images. As expected, the second echo is darker than the first, with different contrast:

We will run tests including only echo 1 or only echo 2 in the BIDS input, to see if the output images look more or less similar to these.

[tsalo]

I think two-echo fMRI data is largely unsupported in fMRIPrep, both because of its rarity and the fact that it can’t be processed like multi-echo data with three or more echoes.

Can I ask how you would expect the data to be preprocessed? In fMRIPrep, all of the registration steps should use just the first echo. Would that work for your data? Other than writing out both preprocessed echoes, would you expect fMRIPrep do anything else to them (e.g., combine them in some way)?

I am trying to figure out where it is randomly selecting the echo. It’s definitely a bug, but I’m having trouble tracking it down. I will try to reproduce the bug with some modified multi-echo data so that I can look through the working directory, but with the holiday, I might not get around to that until next week.

[jaetzel]

I’d expected fmriprep to make an error message and simply not process the two-echo run. When it finished without difficulty I was surprised, and started poking at the output to see what happened. (It looks like some of the tedana multi-echo pipeline ran (e.g., the preprocessed images are masked), but not all of it (e.g., no T2starmap).) I can share files & details if that’d be helpful, but in my opinion an error would be the better “solution”, since it’d make it clear that two-echo runs aren’t supported.

(I hadn’t wanted a two-echo acquisition, but missed that the MR physicist programmed one into a pilot session, so have been looking at it anyway.)

[jaetzel]

Thanks for updating the proper github issue, @tsalo! (Someday perhaps I’ll get active on there …)

I’ll mark this neurostars thread as “solved”, with the answers that 1) fmriprep recreates many (most?) output files every time it is rerun, even if the relevant input, derivatives, and working files are all unchanged, and 2) unreliable things happen when preprocessing a run with only two echoes; don’t attempt it, even though (fmriprep 25.2.2) doesn’t make an error.