Issue with fMRIPrep’s -d Flag: Missing Head Motion Correction Files

Yongning_Lei · February 28, 2025, 12:43am

Summary of what happened:

Hi fMRIPrep experts,

I wanted to test the new --level flag in fMRIPrep, which allows passing the bold mask to subsequent processing steps. To do this, I first ran fMRIPrep with --level minimal, and then attempted to run fMRIPrep again while passing the -d flag to reference the precomputed derivatives. It is test for me, so I wasn’t changing any files in the minimal run, I simply separated fMRIprep into 2 steps and wanted to see if it works well.

However, the second run failed at the node: bold_confounds_wf.fdisp. Upon inspecting the working directory, I noticed that the required head motion correction files were missing as inputs to this node.

I did some systematic testing to investigate the issue:

Full fMRIPrep Pipeline Run:

I ran the full fMRIPrep pipeline on my dataset to ensure it completed successfully, which it did.
The output was stored under derivatives/fmriprep-runall, with the work directory at /scratch/tlei/.cache/fmriprep.

Testing Different Work Directory Combinations:
I suspected that the issue might be related to using different working directories. To test this, I tried three different combinations:

Test 1:
- Minimal run work directory: /scratch/tlei/fmriprep_minimal/.cache/fmriprep
- Full run work directory: /scratch/tlei/fmriprep_final/.cache/fmriprep
Test 2:
- Minimal run work directory: /scratch/tlei/fmriprep_minimal/.cache/fmriprep
- Full run work directory: /scratch/tlei/fmriprep_minimal/.cache/fmriprep
Test 3:
- Minimal run work directory: /scratch/tlei/fmriprep_minimal/.cache/fmriprep
- Full run work directory: /scratch/tlei/.cache/fmriprep

Unfortunately, all three attempts resulted in the same error.

Would appreciate any insights into what might be causing this issue and how to resolve it.

Thank you in advance for your help!

Command used (and if a helper script was used, a link to the helper script or the command generated):

Since all three work directory combinations produced the same issue, I am sharing one example:

Step 1: Running fMRIPrep with --level minimal

module load Singularity/3.5.3-GCC-8.3.0 && unset PYTHONPATH && singularity run --cleanenv --no-home --writable-tmpfs -B /scratch:/scratch -B /scratch/tlei/VOTCLOC/BIDS:/base -B /scratch/tlei/.cache/templateflow:/templateflow -B /scratch/tlei/.cache/fmriprep:/work /scratch/tlei/containers/fmriprep_24.1.1.sif /base /scratch/tlei/VOTCLOC/BIDS/derivatives/fmriprep-minimal participant --participant-label 01 -w /work/ -vv --fs-license-file /base/.license --omp-nthreads 20 --nthreads 20 --mem_mb 80000 --skip-bids-validation --force-bbr --level minimal --stop-on-first-crash --bids-filter-file /base/code/bids_filter.json --fs-subjects-dir /base/derivatives/freesurfer --output-spaces T1w func MNI152NLin2009cAsym fsnative fsaverage

Step 2: Running fMRIPrep with --level full

module load Singularity/3.5.3-GCC-8.3.0 && unset PYTHONPATH && singularity run --cleanenv --no-home --writable-tmpfs -B /scratch:/scratch -B /scratch/tlei/VOTCLOC/BIDS:/base -B /scratch/tlei/.cache/templateflow:/templateflow -B /scratch/tlei/.cache/fmriprep:/work /scratch/tlei/containers/fmriprep_24.1.1.sif /base /scratch/tlei/VOTCLOC/BIDS/derivatives/fmriprep-finalorig participant -d minimal=/scratch/tlei/VOTCLOC/BIDS/derivatives/fmriprep-minimal --participant-label 01 -w /work/ -vv --fs-license-file /base/.license --omp-nthreads 20 --nthreads 20 --mem_mb 80000 --skip-bids-validation --force-bbr --level full --stop-on-first-crash --bids-filter-file /base/code/bids_filter.json --fs-subjects-dir /base/derivatives/freesurfer --output-spaces T1w func MNI152NLin2009cAsym fsnative fsaverage

Version:

fMRIPrep Version: 24.1.1

Environment (Docker, Singularity / Apptainer, custom installation):

Environment: Singularity (version 3.5.3-GCC-8.3.0)
HPC Scheduler: SLURM

Data formatted according to a validatable standard? Please provide the output of the validator:

Relevant log outputs (up to 20 lines):

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.11/concurrent/futures/process.py", line 261, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/plugins/multiproc.py", line 70, in run_node
    result["result"] = node.result
                       ^^^^^^^^^^^
  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 223, in result
    return _load_resultfile(
           ^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/utils.py", line 291, in load_resultfile
    raise FileNotFoundError(results_file)
FileNotFoundError: /work/fmriprep_24_1_wf/sub_01_wf/bold_ses_10_task_fLoc_run_01_wf/bold_confounds_wf/fdisp/result_fdisp.pklz
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.11/concurrent/futures/_base.py", line 340, in _invoke_callbacks
    callback(self)
  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
    result = args.result()
             ^^^^^^^^^^^^^
  File "/opt/conda/envs/fmriprep/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/fmriprep/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
FileNotFoundError: /work/fmriprep_24_1_wf/sub_01_wf/bold_ses_10_task_fLoc_run_01_wf/bold_confounds_wf/fdisp/result_fdisp.pklz
Sentry is attempting to send 1 pending events
Waiting up to 2 seconds
Press Ctrl-C to quit

Screenshots / relevant information:

Below I will paste the crash file here:

I also paste the workdir of the failed step 2

Also here I pasted the fMRIprep-minimal session folder contains all the derivatives

├── anat
│   ├── sub-01_ses-10_run-01_desc-brain_mask.json
│   ├── sub-01_ses-10_run-01_desc-brain_mask.nii.gz
│   ├── sub-01_ses-10_run-01_desc-preproc_T1w.json
│   ├── sub-01_ses-10_run-01_desc-preproc_T1w.nii.gz
│   ├── sub-01_ses-10_run-01_desc-ribbon_mask.json
│   ├── sub-01_ses-10_run-01_desc-ribbon_mask.nii.gz
│   ├── sub-01_ses-10_run-01_dseg.nii.gz
│   ├── sub-01_ses-10_run-01_from-fsnative_to-T1w_mode-image_xfm.txt
│   ├── sub-01_ses-10_run-01_from-MNI152NLin2009cAsym_to-T1w_mode-image_xfm.h5
│   ├── sub-01_ses-10_run-01_from-T1w_to-fsnative_mode-image_xfm.txt
│   ├── sub-01_ses-10_run-01_from-T1w_to-MNI152NLin2009cAsym_mode-image_xfm.h5
│   ├── sub-01_ses-10_run-01_hemi-L_midthickness.surf.gii
│   ├── sub-01_ses-10_run-01_hemi-L_pial.surf.gii
│   ├── sub-01_ses-10_run-01_hemi-L_space-fsaverage_desc-reg_sphere.surf.gii
│   ├── sub-01_ses-10_run-01_hemi-L_space-fsLR_desc-msmsulc_sphere.surf.gii
│   ├── sub-01_ses-10_run-01_hemi-L_space-fsLR_desc-reg_sphere.surf.gii
│   ├── sub-01_ses-10_run-01_hemi-L_sphere.surf.gii
│   ├── sub-01_ses-10_run-01_hemi-L_sulc.shape.gii
│   ├── sub-01_ses-10_run-01_hemi-L_thickness.shape.gii
│   ├── sub-01_ses-10_run-01_hemi-L_white.surf.gii
│   ├── sub-01_ses-10_run-01_hemi-R_midthickness.surf.gii
│   ├── sub-01_ses-10_run-01_hemi-R_pial.surf.gii
│   ├── sub-01_ses-10_run-01_hemi-R_space-fsaverage_desc-reg_sphere.surf.gii
│   ├── sub-01_ses-10_run-01_hemi-R_space-fsLR_desc-msmsulc_sphere.surf.gii
│   ├── sub-01_ses-10_run-01_hemi-R_space-fsLR_desc-reg_sphere.surf.gii
│   ├── sub-01_ses-10_run-01_hemi-R_sphere.surf.gii
│   ├── sub-01_ses-10_run-01_hemi-R_sulc.shape.gii
│   ├── sub-01_ses-10_run-01_hemi-R_thickness.shape.gii
│   ├── sub-01_ses-10_run-01_hemi-R_white.surf.gii
│   ├── sub-01_ses-10_run-01_label-CSF_probseg.nii.gz
│   ├── sub-01_ses-10_run-01_label-GM_probseg.nii.gz
│   └── sub-01_ses-10_run-01_label-WM_probseg.nii.gz
├── fmap
│   ├── sub-01_ses-10_acq-fMRI_run-1_fmapid-auto00000_desc-coeff_fieldmap.nii.gz
│   ├── sub-01_ses-10_acq-fMRI_run-1_fmapid-auto00000_desc-epi_fieldmap.nii.gz
│   ├── sub-01_ses-10_acq-fMRI_run-1_fmapid-auto00000_desc-preproc_fieldmap.json
│   ├── sub-01_ses-10_acq-fMRI_run-1_fmapid-auto00000_desc-preproc_fieldmap.nii.gz
│   ├── sub-01_ses-10_acq-fMRI_run-2_fmapid-auto00001_desc-coeff_fieldmap.nii.gz
│   ├── sub-01_ses-10_acq-fMRI_run-2_fmapid-auto00001_desc-epi_fieldmap.nii.gz
│   ├── sub-01_ses-10_acq-fMRI_run-2_fmapid-auto00001_desc-preproc_fieldmap.json
│   └── sub-01_ses-10_acq-fMRI_run-2_fmapid-auto00001_desc-preproc_fieldmap.nii.gz
└── func
    ├── sub-01_ses-10_task-fLoc_run-01_desc-brain_mask.json
    ├── sub-01_ses-10_task-fLoc_run-01_desc-brain_mask.nii.gz
    ├── sub-01_ses-10_task-fLoc_run-01_desc-coreg_boldref.json
    ├── sub-01_ses-10_task-fLoc_run-01_desc-coreg_boldref.nii.gz
    ├── sub-01_ses-10_task-fLoc_run-01_desc-hmc_boldref.json
    ├── sub-01_ses-10_task-fLoc_run-01_desc-hmc_boldref.nii.gz
    ├── sub-01_ses-10_task-fLoc_run-01_from-boldref_to-auto00000_mode-image_xfm.json
    ├── sub-01_ses-10_task-fLoc_run-01_from-boldref_to-auto00000_mode-image_xfm.txt
    ├── sub-01_ses-10_task-fLoc_run-01_from-boldref_to-T1w_mode-image_desc-coreg_xfm.json
    ├── sub-01_ses-10_task-fLoc_run-01_from-boldref_to-T1w_mode-image_desc-coreg_xfm.txt
    ├── sub-01_ses-10_task-fLoc_run-01_from-orig_to-boldref_mode-image_desc-hmc_xfm.json
    ├── sub-01_ses-10_task-fLoc_run-01_from-orig_to-boldref_mode-image_desc-hmc_xfm.txt
    ├── sub-01_ses-10_task-fLoc_run-02_desc-brain_mask.json
    ├── sub-01_ses-10_task-fLoc_run-02_desc-brain_mask.nii.gz
    ├── sub-01_ses-10_task-fLoc_run-02_desc-coreg_boldref.json
    ├── sub-01_ses-10_task-fLoc_run-02_desc-coreg_boldref.nii.gz
    ├── sub-01_ses-10_task-fLoc_run-02_desc-hmc_boldref.json
    ├── sub-01_ses-10_task-fLoc_run-02_desc-hmc_boldref.nii.gz
    ├── sub-01_ses-10_task-fLoc_run-02_from-boldref_to-auto00000_mode-image_xfm.json
    ├── sub-01_ses-10_task-fLoc_run-02_from-boldref_to-auto00000_mode-image_xfm.txt
    ├── sub-01_ses-10_task-fLoc_run-02_from-boldref_to-T1w_mode-image_desc-coreg_xfm.json
    ├── sub-01_ses-10_task-fLoc_run-02_from-boldref_to-T1w_mode-image_desc-coreg_xfm.txt
    ├── sub-01_ses-10_task-fLoc_run-02_from-orig_to-boldref_mode-image_desc-hmc_xfm.json
    ├── sub-01_ses-10_task-fLoc_run-02_from-orig_to-boldref_mode-image_desc-hmc_xfm.txt
    ├── sub-01_ses-10_task-fLoc_run-03_desc-brain_mask.json
    ├── sub-01_ses-10_task-fLoc_run-03_desc-brain_mask.nii.gz
    ├── sub-01_ses-10_task-fLoc_run-03_desc-coreg_boldref.json
    ├── sub-01_ses-10_task-fLoc_run-03_desc-coreg_boldref.nii.gz
    ├── sub-01_ses-10_task-fLoc_run-03_desc-hmc_boldref.json
    ├── sub-01_ses-10_task-fLoc_run-03_desc-hmc_boldref.nii.gz
    ├── sub-01_ses-10_task-fLoc_run-03_from-boldref_to-auto00000_mode-image_xfm.json
    ├── sub-01_ses-10_task-fLoc_run-03_from-boldref_to-auto00000_mode-image_xfm.txt
    ├── sub-01_ses-10_task-fLoc_run-03_from-boldref_to-T1w_mode-image_desc-coreg_xfm.json
    ├── sub-01_ses-10_task-fLoc_run-03_from-boldref_to-T1w_mode-image_desc-coreg_xfm.txt
    ├── sub-01_ses-10_task-fLoc_run-03_from-orig_to-boldref_mode-image_desc-hmc_xfm.json
    ├── sub-01_ses-10_task-fLoc_run-03_from-orig_to-boldref_mode-image_desc-hmc_xfm.txt
    ├── sub-01_ses-10_task-fLoc_run-04_desc-brain_mask.json
    ├── sub-01_ses-10_task-fLoc_run-04_desc-brain_mask.nii.gz
    ├── sub-01_ses-10_task-fLoc_run-04_desc-coreg_boldref.json
    ├── sub-01_ses-10_task-fLoc_run-04_desc-coreg_boldref.nii.gz
    ├── sub-01_ses-10_task-fLoc_run-04_desc-hmc_boldref.json
    ├── sub-01_ses-10_task-fLoc_run-04_desc-hmc_boldref.nii.gz
    ├── sub-01_ses-10_task-fLoc_run-04_from-boldref_to-auto00000_mode-image_xfm.json
    ├── sub-01_ses-10_task-fLoc_run-04_from-boldref_to-auto00000_mode-image_xfm.txt
    ├── sub-01_ses-10_task-fLoc_run-04_from-boldref_to-T1w_mode-image_desc-coreg_xfm.json
    ├── sub-01_ses-10_task-fLoc_run-04_from-boldref_to-T1w_mode-image_desc-coreg_xfm.txt
    ├── sub-01_ses-10_task-fLoc_run-04_from-orig_to-boldref_mode-image_desc-hmc_xfm.json
    ├── sub-01_ses-10_task-fLoc_run-04_from-orig_to-boldref_mode-image_desc-hmc_xfm.txt
    ├── sub-01_ses-10_task-retFF_run-01_desc-brain_mask.json
    ├── sub-01_ses-10_task-retFF_run-01_desc-brain_mask.nii.gz
    ├── sub-01_ses-10_task-retFF_run-01_desc-coreg_boldref.json
    ├── sub-01_ses-10_task-retFF_run-01_desc-coreg_boldref.nii.gz
    ├── sub-01_ses-10_task-retFF_run-01_desc-hmc_boldref.json
    ├── sub-01_ses-10_task-retFF_run-01_desc-hmc_boldref.nii.gz
    ├── sub-01_ses-10_task-retFF_run-01_from-boldref_to-auto00001_mode-image_xfm.json
    ├── sub-01_ses-10_task-retFF_run-01_from-boldref_to-auto00001_mode-image_xfm.txt
    ├── sub-01_ses-10_task-retFF_run-01_from-boldref_to-T1w_mode-image_desc-coreg_xfm.json
    ├── sub-01_ses-10_task-retFF_run-01_from-boldref_to-T1w_mode-image_desc-coreg_xfm.txt
    ├── sub-01_ses-10_task-retFF_run-01_from-orig_to-boldref_mode-image_desc-hmc_xfm.json
    ├── sub-01_ses-10_task-retFF_run-01_from-orig_to-boldref_mode-image_desc-hmc_xfm.txt
    ├── sub-01_ses-10_task-retRW_run-01_desc-brain_mask.json
    ├── sub-01_ses-10_task-retRW_run-01_desc-brain_mask.nii.gz
    ├── sub-01_ses-10_task-retRW_run-01_desc-coreg_boldref.json
    ├── sub-01_ses-10_task-retRW_run-01_desc-coreg_boldref.nii.gz
    ├── sub-01_ses-10_task-retRW_run-01_desc-hmc_boldref.json
    ├── sub-01_ses-10_task-retRW_run-01_desc-hmc_boldref.nii.gz
    ├── sub-01_ses-10_task-retRW_run-01_from-boldref_to-auto00001_mode-image_xfm.json
    ├── sub-01_ses-10_task-retRW_run-01_from-boldref_to-auto00001_mode-image_xfm.txt
    ├── sub-01_ses-10_task-retRW_run-01_from-boldref_to-T1w_mode-image_desc-coreg_xfm.json
    ├── sub-01_ses-10_task-retRW_run-01_from-boldref_to-T1w_mode-image_desc-coreg_xfm.txt
    ├── sub-01_ses-10_task-retRW_run-01_from-orig_to-boldref_mode-image_desc-hmc_xfm.json
    └── sub-01_ses-10_task-retRW_run-01_from-orig_to-boldref_mode-image_desc-hmc_xfm.txt

effigies · February 28, 2025, 2:56pm

This will be fixed in the next release. You can test out the nipreps/fmriprep:unstable docker container in the meantime.

Yongning_Lei · February 28, 2025, 4:00pm

Thanks for your reply! I will test the -d flag using fmriprep:unstable.

Actually, I wasn’t planning to bring this up here, but I discovered the -d flag bug while investigating another issue—the BOLD mask output from fMRIPrep appears incorrect, even though the corresponding xfm.txt is correct. See the screenshots below:

(Blue represents the mask output by fMRIPrep, while purple is the mask I calculated using the same xfm.txt and the T1w mask.)

After reading these posts (Brain mask generation in fMRIPrep, fMRIPrep BOLD brain mask issue), it seems that others have managed to replace incorrect brain masks in intermediate steps and then re-run fMRIPrep.

So, I attempted to use --level minimal, generate the brain mask myself, and finally use -d to bring the new bold mask, but now I’m encountering this bug.

Do you have any thoughts on the bold mask issue? Or should I open a new topic and tag you there?

effigies · February 28, 2025, 4:28pm

Probably worth a new topic, but also probably worth verifying that it is not resolved in the unstable version.

Yongning_Lei · February 28, 2025, 4:34pm

Will do both! Thank you! I will tag you in the new topic if the unstable version still got wrong mask

Yongning_Lei · March 3, 2025, 11:02am

Hi Chris, I ran the unstable version of fMRIprep, and the boldmask problem is being solved. I can run my subjects using just 1 step.

As for the -d flag, it was stuck on fmriprep_25_0_wf.sub_01_wf.bold_ses_10_task_retRW_run_01_wf.bold_native_wf.boldref_fmap for a long time, and the cluster times up, I will launch it again to see if it still stuck at that same place.

effigies · March 3, 2025, 12:57pm

I’m currently working on getting fieldmaps to work with the --derivatives flag.

Yongning_Lei · March 3, 2025, 1:23pm

Hi Chirs, I attached the .o and .e of my last try, briefly, it crashed on ReconstructFieldmap, however, the crash file wasn’t generated.

Please find the files in this link: fmriprep_log – Google Drive