Failed fMRIPrep during Resample

Summary of what happened:

I’m running a fMRIPrep singularity container (v22.1.0) on my university cluster and running into an error during the “resample” node.

This is only happening for 2/40 participants.

Command used (and if a helper script was used, a link to the helper script or the command generated):

sbatch << EOT
#!/usr/bin/bash

#SBATCH --account=p30952
#SBATCH --partition=normal
#SBATCH --time=10:00:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=16
#SBATCH --mem-per-cpu=4G
#SBATCH --job-name=${JOB_NAME}

# Outputs ----------------------------------
#SBATCH --mail-user=${email}
#SBATCH --mail-type=FAIL
#SBATCH --output=${BIDS_DIR}/code/slurm_logs/out.${JOB_NAME}
#SBATCH --error=${BIDS_DIR}/code/slurm_logs/err.${JOB_NAME}
# ------------------------------------------

# Load modules
module purge
module load singularity
echo "Modules loaded" 

# Remove IsRunning files from Freesurfer
[ -f  ${SUBJECTS_DIR}/$subject/scripts/IsRunning.lh+rh ] && rm -rf ${SUBJECTS_DIR}/$subject/scripts/IsRunning.lh+rh

# Do singularity run
echo "Begin Preprocessing"

singularity run \
        --cleanenv -B /projects \
        /projects/p30952/environment/singularity_images/${FMRIPREP_VER}.simg \
        ${BIDS_DIR} \
        ${FMRIPREP_DIR} \
        participant --participant-label ${subject} \
        -w ${WORK_DIR} \
        --omp-nthreads 8 \
        --nthreads 12 \
        --mem_mb 24000 \
        --fs-license-file ${FS_LICENSE} \
	--fs-subjects-dir ${FMRIPREP_DIR}/sourcedata/freesurfer \
        --output-spaces anat fsnative MNI152NLin2009cAsym fsLR fsaverage \
        --cifti-output 91k \
        --fd-spike-threshold 0.4 \
        --dvars-spike-threshold 1.5 \
	--use-syn-sdc warn \
	--dummy-scans 4
exit 0

EOT

Version:

22.1.0

Environment (Docker, Singularity, custom installation):

Singularity

Data formatted according to a validatable standard? Please provide the output of the validator:

Yes

Relevant log outputs (up to 20 lines):

	 [Node] Error on "fmriprep_22_1_wf.single_subject_SA175_wf.func_preproc_ses_b_task_rest_run_1_wf.unwarp_wf.resample" (/projects/b1145/bram/work/fmriprep_22_1_wf/single_subject_SA175_wf/func_preproc_ses_b_task_rest_run_1_wf/unwarp_wf/resample)
230209-15:41:56,18 nipype.workflow ERROR:
	 Node resample failed to run on host qnode8113.
230209-15:41:56,28 nipype.workflow ERROR:
	 Saving crash info to /projects/b1145/bram/sa_rsfmri/derivatives/fmriprep-22.1.0/sub-SA175/log/20230209-153320_3eea714b-ea06-4b49-89b4-407bae110c00/crash-20230209-154156-brd2241-resample-349fdf07-532d-468e-b61f-ae20fa4c8753.txt
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node resample.

Traceback:
	concurrent.futures.process._RemoteTraceback: 
	"""
	Traceback (most recent call last):
	  File "/opt/conda/lib/python3.9/concurrent/futures/process.py", line 243, in _process_worker
	    r = call_item.fn(*call_item.args, **call_item.kwargs)
	  File "/opt/conda/lib/python3.9/concurrent/futures/process.py", line 202, in _process_chunk
	    return [fn(*args) for args in chunk]
	  File "/opt/conda/lib/python3.9/concurrent/futures/process.py", line 202, in <listcomp>
	    return [fn(*args) for args in chunk]
	  File "/opt/conda/lib/python3.9/site-packages/sdcflows/interfaces/bspline.py", line 630, in _b0_resampler
	    unwarped_img = unwarp.apply(distorted_img, ro_time=ro, pe_dir=pe)
	  File "/opt/conda/lib/python3.9/site-packages/sdcflows/transform.py", line 171, in apply
	    self.fit(spatialimage)
	  File "/opt/conda/lib/python3.9/site-packages/sdcflows/transform.py", line 99, in fit
	    wmat = grid_bspline_weights(spatialimage, moved_cs)
	  File "/opt/conda/lib/python3.9/site-packages/sdcflows/transform.py", line 417, in grid_bspline_weights
	    colloc_ax = BSpline.design_matrix(x, t, 3)[:, (2 + pad_left):-(2 + pad_right)]
	  File "/opt/conda/lib/python3.9/site-packages/scipy/interpolate/_bsplines.py", line 416, in design_matrix
	    raise ValueError(f'Out of bounds w/ x = {x}.')
	ValueError: Out of bounds w/ x = [ 4.19654081  4.29093281  4.38532481  4.47971681  4.57410882  4.66850082
	  4.76289282  4.85728482  4.95167682  5.04606882  5.14046082  5.23485283
	  5.32924483  5.42363683  5.51802883  5.61242083  5.70681283  5.80120483
	  5.89559684  5.98998884  6.08438084  6.17877284  6.27316484  6.36755684
	  6.46194884  6.55634085  6.65073285  6.74512485  6.83951685  6.93390885
	  7.02830085  7.12269285  7.21708486  7.31147686  7.40586886  7.50026086
	  7.59465286  7.68904486  7.78343686  7.87782887  7.97222087  8.06661287
	  8.16100487  8.25539687  8.34978887  8.44418088  8.53857288  8.63296488
	  8.72735688  8.82174888  8.91614088  9.01053288  9.10492489  9.19931689
	  9.29370889  9.38810089  9.48249289  9.57688489  9.67127689  9.7656689
	  9.8600609   9.9544529  10.0488449  10.1432369  10.2376289  10.3320209
	 10.42641291 10.52080491 10.61519691 10.70958891 10.80398091 10.89837291
	 10.99276491 11.08715692 11.18154892 11.27594092 11.37033292 11.46472492
	 11.55911692 11.65350892 11.74790093 11.84229293 11.93668493 12.03107693
	 12.12546893 12.21986093 12.31425293 12.40864494 12.50303694 12.59742894
	 12.69182094 12.78621294 12.88060494 12.97499694 13.06938895 13.16378095
	 13.25817295 13.35256495 13.44695695 13.54134895 13.63574095 13.73013296
	 13.82452496 13.91891696 14.01330896 14.10770096 14.20209296 14.29648496
	 14.39087697 14.48526897 14.57966097 14.67405297 14.76844497 14.86283697
	 14.95722897 15.05162098 15.14601298 15.24040498 15.33479698 15.42918898].
	"""

	The above exception was the direct cause of the following exception:

	Traceback (most recent call last):
	  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 398, in run
	    runtime = self._run_interface(runtime)
	  File "/opt/conda/lib/python3.9/site-packages/sdcflows/interfaces/bspline.py", line 373, in _run_interface
	    ) = zip(*outputs)
	  File "/opt/conda/lib/python3.9/concurrent/futures/process.py", line 559, in _chain_from_iterable_of_lists
	    for element in iterable:
	  File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 609, in result_iterator
	    yield fs.pop().result()
	  File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 439, in result
	    return self.__get_result()
	  File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
	    raise self._exception
	ValueError: Out of bounds w/ x = [ 4.19654081  4.29093281  4.38532481  4.47971681  4.57410882  4.66850082
	  4.76289282  4.85728482  4.95167682  5.04606882  5.14046082  5.23485283
	  5.32924483  5.42363683  5.51802883  5.61242083  5.70681283  5.80120483
	  5.89559684  5.98998884  6.08438084  6.17877284  6.27316484  6.36755684
	  6.46194884  6.55634085  6.65073285  6.74512485  6.83951685  6.93390885
	  7.02830085  7.12269285  7.21708486  7.31147686  7.40586886  7.50026086
	  7.59465286  7.68904486  7.78343686  7.87782887  7.97222087  8.06661287
	  8.16100487  8.25539687  8.34978887  8.44418088  8.53857288  8.63296488
	  8.72735688  8.82174888  8.91614088  9.01053288  9.10492489  9.19931689
	  9.29370889  9.38810089  9.48249289  9.57688489  9.67127689  9.7656689
	  9.8600609   9.9544529  10.0488449  10.1432369  10.2376289  10.3320209
	 10.42641291 10.52080491 10.61519691 10.70958891 10.80398091 10.89837291
	 10.99276491 11.08715692 11.18154892 11.27594092 11.37033292 11.46472492
	 11.55911692 11.65350892 11.74790093 11.84229293 11.93668493 12.03107693
	 12.12546893 12.21986093 12.31425293 12.40864494 12.50303694 12.59742894
	 12.69182094 12.78621294 12.88060494 12.97499694 13.06938895 13.16378095
	 13.25817295 13.35256495 13.44695695 13.54134895 13.63574095 13.73013296
	 13.82452496 13.91891696 14.01330896 14.10770096 14.20209296 14.29648496
	 14.39087697 14.48526897 14.57966097 14.67405297 14.76844497 14.86283697
	 14.95722897 15.05162098 15.14601298 15.24040498 15.33479698 15.42918898].


230209-15:41:58,16 nipype.workflow ERROR:
	 could not run node: fmriprep_22_1_wf.single_subject_SA175_wf.func_preproc_ses_b_task_rest_run_1_wf.unwarp_wf.resample
230209-15:41:58,87 nipype.workflow CRITICAL:
	 fMRIPrep failed: Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node resample.

Traceback:
	concurrent.futures.process._RemoteTraceback: 
	"""
	Traceback (most recent call last):
	  File "/opt/conda/lib/python3.9/concurrent/futures/process.py", line 243, in _process_worker
	    r = call_item.fn(*call_item.args, **call_item.kwargs)
	  File "/opt/conda/lib/python3.9/concurrent/futures/process.py", line 202, in _process_chunk
	    return [fn(*args) for args in chunk]
	  File "/opt/conda/lib/python3.9/concurrent/futures/process.py", line 202, in <listcomp>
	    return [fn(*args) for args in chunk]
	  File "/opt/conda/lib/python3.9/site-packages/sdcflows/interfaces/bspline.py", line 630, in _b0_resampler
	    unwarped_img = unwarp.apply(distorted_img, ro_time=ro, pe_dir=pe)
	  File "/opt/conda/lib/python3.9/site-packages/sdcflows/transform.py", line 171, in apply
	    self.fit(spatialimage)
	  File "/opt/conda/lib/python3.9/site-packages/sdcflows/transform.py", line 99, in fit
	    wmat = grid_bspline_weights(spatialimage, moved_cs)
	  File "/opt/conda/lib/python3.9/site-packages/sdcflows/transform.py", line 417, in grid_bspline_weights
	    colloc_ax = BSpline.design_matrix(x, t, 3)[:, (2 + pad_left):-(2 + pad_right)]
	  File "/opt/conda/lib/python3.9/site-packages/scipy/interpolate/_bsplines.py", line 416, in design_matrix
	    raise ValueError(f'Out of bounds w/ x = {x}.')
	ValueError: Out of bounds w/ x = [ 4.19654081  4.29093281  4.38532481  4.47971681  4.57410882  4.66850082
	  4.76289282  4.85728482  4.95167682  5.04606882  5.14046082  5.23485283
	  5.32924483  5.42363683  5.51802883  5.61242083  5.70681283  5.80120483
	  5.89559684  5.98998884  6.08438084  6.17877284  6.27316484  6.36755684
	  6.46194884  6.55634085  6.65073285  6.74512485  6.83951685  6.93390885
	  7.02830085  7.12269285  7.21708486  7.31147686  7.40586886  7.50026086
	  7.59465286  7.68904486  7.78343686  7.87782887  7.97222087  8.06661287
	  8.16100487  8.25539687  8.34978887  8.44418088  8.53857288  8.63296488
	  8.72735688  8.82174888  8.91614088  9.01053288  9.10492489  9.19931689
	  9.29370889  9.38810089  9.48249289  9.57688489  9.67127689  9.7656689
	  9.8600609   9.9544529  10.0488449  10.1432369  10.2376289  10.3320209
	 10.42641291 10.52080491 10.61519691 10.70958891 10.80398091 10.89837291
	 10.99276491 11.08715692 11.18154892 11.27594092 11.37033292 11.46472492
	 11.55911692 11.65350892 11.74790093 11.84229293 11.93668493 12.03107693
	 12.12546893 12.21986093 12.31425293 12.40864494 12.50303694 12.59742894
	 12.69182094 12.78621294 12.88060494 12.97499694 13.06938895 13.16378095
	 13.25817295 13.35256495 13.44695695 13.54134895 13.63574095 13.73013296
	 13.82452496 13.91891696 14.01330896 14.10770096 14.20209296 14.29648496
	 14.39087697 14.48526897 14.57966097 14.67405297 14.76844497 14.86283697
	 14.95722897 15.05162098 15.14601298 15.24040498 15.33479698 15.42918898].
	"""

	The above exception was the direct cause of the following exception:

	Traceback (most recent call last):
	  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 398, in run
	    runtime = self._run_interface(runtime)
	  File "/opt/conda/lib/python3.9/site-packages/sdcflows/interfaces/bspline.py", line 373, in _run_interface
	    ) = zip(*outputs)
	  File "/opt/conda/lib/python3.9/concurrent/futures/process.py", line 559, in _chain_from_iterable_of_lists
	    for element in iterable:
	  File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 609, in result_iterator
	    yield fs.pop().result()
	  File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 439, in result
	    return self.__get_result()
	  File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
	    raise self._exception
	ValueError: Out of bounds w/ x = [ 4.19654081  4.29093281  4.38532481  4.47971681  4.57410882  4.66850082
	  4.76289282  4.85728482  4.95167682  5.04606882  5.14046082  5.23485283
	  5.32924483  5.42363683  5.51802883  5.61242083  5.70681283  5.80120483
	  5.89559684  5.98998884  6.08438084  6.17877284  6.27316484  6.36755684
	  6.46194884  6.55634085  6.65073285  6.74512485  6.83951685  6.93390885
	  7.02830085  7.12269285  7.21708486  7.31147686  7.40586886  7.50026086
	  7.59465286  7.68904486  7.78343686  7.87782887  7.97222087  8.06661287
	  8.16100487  8.25539687  8.34978887  8.44418088  8.53857288  8.63296488
	  8.72735688  8.82174888  8.91614088  9.01053288  9.10492489  9.19931689
	  9.29370889  9.38810089  9.48249289  9.57688489  9.67127689  9.7656689
	  9.8600609   9.9544529  10.0488449  10.1432369  10.2376289  10.3320209
	 10.42641291 10.52080491 10.61519691 10.70958891 10.80398091 10.89837291
	 10.99276491 11.08715692 11.18154892 11.27594092 11.37033292 11.46472492
	 11.55911692 11.65350892 11.74790093 11.84229293 11.93668493 12.03107693
	 12.12546893 12.21986093 12.31425293 12.40864494 12.50303694 12.59742894
	 12.69182094 12.78621294 12.88060494 12.97499694 13.06938895 13.16378095
	 13.25817295 13.35256495 13.44695695 13.54134895 13.63574095 13.73013296
	 13.82452496 13.91891696 14.01330896 14.10770096 14.20209296 14.29648496
	 14.39087697 14.48526897 14.57966097 14.67405297 14.76844497 14.86283697
	 14.95722897 15.05162098 15.14601298 15.24040498 15.33479698 15.42918898].


230209-15:42:01,245 cli ERROR:
	 Preprocessing did not finish successfully. Errors occurred while processing data from participants: SA175 (1). Check the HTML reports for details.
230209-15:42:01,245 cli ERROR:
	 Preprocessing did not finish successfully. Errors occurred while processing data from participants: SA175 (1). Check the HTML reports for details.

Screenshots / relevant information:

N/A

Hi @bramdiamond,

This was fixed in 22.1.1 (see here NodeExecutionError: · Issue #2933 · nipreps/fmriprep · GitHub).

Best,
Steven

Thanks @Steven!

A few small questions:

  1. If I rerun fMRIPrep with an updated version (i.e., v22.1.0 → v22.1.1), is it best practice to run all participants through the updated version or should the update not make a difference to successful runs? Is there a rule for this, such as only re-running all other participants if moving from 22.1.X → 22.2.X or 22.X.X → 23.X.X?
  2. Will re-running fMRIPrep with a new version remove any manually created files in the fMRIPrep directory?

Cheers,
Bram

In general, it is good to have all subjects ran on the same version.

I don’t think so, but it might be good to test with a single subject first where you backup anything you’re worried about losing.

Best,
Steven