fMRIPrep 23.0.2 hangs during anatomical normalization with multiple sessions

Summary of what happened:

I am running fMRIPrep v23.0.2 (Docker) and encountering a problem during the anatomical normalization step:

fmriprep_23_0_wf.single_subject_<subject_id>_wf.anat_preproc_wf.anat_norm_wf.registration

When the input dataset contains multiple sessions for a subject, the workflow consistently gets stuck at this step. The process does not crash, but appears to hang indefinitely.

Observed behavior:

  • CPU usage drops to < 1 core

  • Memory usage remains stable (~12 GB)

  • No further progress is made, even after several days

  • No explicit error messages are produced

Control test:

  • When I run the exact same subject but include only a single session in the input data, the workflow completes successfully without any issues.

Additional notes:

  • The issue appears specifically tied to multi-session datasets

  • The hang occurs reproducibly at the same anatomical normalization/registration step

Questions:

  1. Is this a known issue with multi-session processing in fMRIPrep 23.0.2?

  2. Are there recommended workarounds (e.g., flags, session-wise processing)?

Thank you for your help!

Command used (and if a helper script was used, a link to the helper script or the command generated):

runai submit --name fmriprep-times \
-i registry.rcp.epfl.ch/rcp-runai-uphummel-hudry/fmriprep_caas:23.0.2 \
--existing-pvc claimname=uphummel-scratch,path=/data \
--run-as-gid 82789 --run-as-uid 312106 – \
/data/gbaron/TiMeS \
/data/gbaron/TiMeS/derivatives/fmriprep \
participant \
--participant-label TiMeSWP11001 \
--fs-license-file /data/gbaron/ReverseStroke/fmristroke_utils/license_freesurfer_antonin.txt \
--fs-subjects-dir 
/data/gbaron/TiMeS/derivatives/fmriprep/sourcedata/freesurfer_TiMeSWP11001 \
--work-dir /data/gbaron/TiMeSWP11001_work \
--output-layout bids \
--write-graph \
--return-all-components \
--force-no-bbr \
--bids-database-dir tmp/process_01 \
--output-spaces MNI152NLin2009cAsym anat func \
--verbose \
--skip_bids_validation

Version:

23.0.2

Environment (Docker, Singularity / Apptainer, custom installation):

Docker

Data formatted according to a validatable standard? Please provide the output of the validator:

PASTE VALIDATOR OUTPUT HERE

Relevant log outputs (up to 20 lines):

PASTE LOG OUTPUT HERE

Screenshots / relevant information:


Hi @Giorgia_Baron,

Does the error persist on the newest fmriprep version? How much memory are you letting Docker access? Also, runai submit is not the typical way of running Docker jobs; is it possible this is limiting your resources?

Best,

Steven

Hello @Steven

Thank you for your reply.

Unfortunately, I am not able to test the newest fMRIPrep version, as the denoising tool I plan to use (fmristroke: https://github.com/alixlam/fmristroke) is currently built specifically on fMRIPrep 23.0.2.

Regarding resources, I am not explicitly limiting memory allocation. The job is being executed on my institution’s research computing cluster using runai submit, which is the standard workflow provided there. I will follow up with our IT team to confirm whether any resource limits might be affecting the job.

I will update you as soon as I have more information, thank you!

Giorgia

Hello @Steven

Thank you again for your suggestions.

I checked with our IT team regarding the RunAI setup. They confirmed that, by default, jobs do not have explicit CPU or memory limits and are able to access all available resources on the node.

Given this, it seems unlikely that the issue is caused by an explicit CPU or memory cap on the cluster side. Do you have additional hints about the possible cause?

Thank you and best,
Giorgia

Hi @Giorgia_Baron ,

I don’t know enough about what RunAI is doing under the hood to rule cluster issues out. I would recommend testing updating fmriprep to rule version differences out.

Best,

Steven