fMRIPrep seems to get stuck

Software Support
1

Summary of what happened:

I am running fmriprep v24.1.1 to process some BOLD fMRI data on hpc and after some processing steps, it seems to get stuck. I don’t see any particular error so am unsure how to go about this. I am specifying --ntasks=1 --cpus-per-task=8 in my slurm file. I’ve specified --mem 64G and 128G but encounter the same problem in both cases.

Version:

24.1.1

Command Used

PASTE CODE HERE

Environment (Docker, Singularity / Apptainer, custom installation):

Singularity

Relevant log outputs (up to 20 lines):

The last 5 lines of the code are:2025-05-05 10:03:13,871 [    INFO] [Node] Outdated cache found for "fmriprep_24_1_wf.sub_003_wf.bold_ses_1_task_LOCALISER5_wf.bold_confounds_wf.tcc_metadata_fmt".
2025-05-05 10:03:13,885 [    INFO] [Node] Executing "tcc_metadata_fmt" <niworkflows.interfaces.utility.TSV2JSON>
2025-05-05 10:03:14,167 [    INFO] [Node] Finished "tcc_metadata_fmt", elapsed time 0.282167s.
2025-05-05 10:03:14,932 [    INFO] [Node] Finished "dvars", elapsed time 37.012805s.
2025-05-05 10:03:16,825 [    INFO] [Node] Finished "dvars", elapsed time 36.881818s.
2025-05-05 10:03:37,825 [    INFO] [Node] Finished "dvars", elapsed time 45.900756s.

I also tried running it with more verbosity and the log shows:
2025-05-07 14:23:21,017 [   DEBUG] Cannot allocate job 1812 (0.20GB, 1 threads).
2025-05-07 14:23:21,017 [   DEBUG] Cannot allocate job 1813 (0.20GB, 1 threads).
2025-05-07 14:23:21,028 [    INFO] [Node] Executing "_autorecon_surfs0" <smriprep.interfaces.freesurfer.ReconAll>
2025-05-07 14:23:21,059 [    INFO] resume recon-all : recon-all -autorecon-hemi lh -noparcstats -noparcstats2 -noparcstats3 -nohyporelabel -nobalabels -lh-only -openmp 8 -subjid sub-003 -sd /scratch/users/gajawell/fmriprep/sourcedata/freesurfer -notessellate -nosmooth1 -noinflate1 -noqsphere
2025-05-07 14:23:21,060 [    INFO] resume recon-all : recon-all -autorecon-hemi lh -noparcstats -noparcstats2 -noparcstats3 -nohyporelabel -nobalabels -lh-only -openmp 8 -subjid sub-003 -sd /scratch/users/gajawell/fmriprep/sourcedata/freesurfer -notessellate -nosmooth1 -noinflate1 -noqsphere
2025-05-07 14:23:22,857 [   DEBUG] Progress: 1816 jobs, 430/1/15 (done/running/ready), 1/1385 (pending_tasks/waiting).
2025-05-07 14:23:22,858 [   DEBUG] Tasks currently running: 1. Pending: 1.
2025-05-07 14:23:22,860 [    INFO] [MultiProc] Running 1 tasks, and 15 jobs ready. Free memory (GB): 221.14/226.14, Free processors: 0/1.
                     Currently running:
                       * _autorecon_surfs0
2025-05-07 14:23:22,860 [   DEBUG] No resources available
2025-05-07 14:23:24,859 [   DEBUG] No resources available

The last 2 lines continue until the time limit is reached.

Thank you!

2

Hi @nihasan01,

Please include the command you are using to run fMRIprep, as requested by the software support template. What time limit are you imposing? How any subjects are handled by a single slurm job? Have you run seff <jobID> to see the resource usage of the job?

Best,
Steven

3

Hi Steven,

Thank you and sorry I missed that. Here is the slurm script I am using.


#!/bin/bash -l
#SBATCH --array 1
#SBATCH --job-name rerun1
#SBATCH --mem 64G
#SBATCH --mail-type=ALL
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --begin=now
#SBATCH --time 48:00:00
#SBATCH -o %x-%A-%a.out
#SBATCH -e %x-%A-%a.err
#SBATCH -p saggar,owners

LINE_NUM=$( expr $SLURM_ARRAY_TASK_ID )
SUBJ=$(awk "NR==$LINE_NUM" /scratch/users/gajawell/test/sublist.txt)

module load system

BIDS_DIR=/scratch/users/gajawell/bids
OUT_DIR=/scratch/users/gajawell/fmriprep
WORK_DIR=/scratch/users/gajawell/fmriprep_working_dir
SING_IMG=/home/groups/saggar/simg/fmriprep-24.1.1.simg

FS_LICENSE=/home/groups/saggar/licenses/freesurfer.txt
FMRIPREP_OPTS="--output-spaces MNI152NLin6Asym:res-2 MNI152NLin2009cAsym:res-2 anat fsaverage:den-10k --cifti-output --fs-license-file ${FS_LICENSE} --nthreads 1 --omp-nthreads 8 -vvv"

unset PYTHONPATH
export FS_LICENSE=${FS_LICENSE}
singularity run --cleanenv ${SING_IMG} ${BIDS_DIR} ${OUT_DIR} participant --participant-label ${SUBJ} -w ${WORK_DIR} ${FMRIPREP_OPTS}

When I got the error, I was running only a single subject. My time limit is 48 hours. I have not done seff but I will try that.

Thank you.
Niharika

4

Hi @nihasan01,

You are only giving fmriprep 1 cpu in the fmriprep command. Increasing that will decrease runtime.

Best,
Steven

5

Thank you Steven. I removed that and increased the memory and it seems to work now.