fMRIPrep 24.1.0 failed during `bs_filter` (Extreme value 2.21e+04 detected in spline coefficients)

yibeichen · September 30, 2024, 3:16pm

Summary of what happened:

I’m using fMRIPrep 24.1.0 on a subset (N=40) of the Narratives dataset, there are two subjects failed due to the same reason:

Extreme value 2.21e+04 detected in spline coefficients

when running "fmriprep_24_1_wf.sub_095_wf.fmap_preproc_wf.wf_auto_00004.bs_filter"

Command used (and if a helper script was used, a link to the helper script or the command generated):

singularity run --cleanenv \
    -B /om2/scratch/tmp/yibei/fmriprep_compute/job-38898175-sub-095/ds \
    -B /nese/mit/group/sig/projects/hbn/hbn_bids/code/license.txt:/SGLR/FREESURFER_HOME/license.txt \
    containers/.datalad/environments/fmriprep-24-1-0/image \
    inputs/data/BIDS \
    outputs/fmriprep \
    participant \
    -w /om2/scratch/tmp/yibei/fmriprep_compute/job-38898175-sub-095/ds/.git/tmp/wkdir 
    --nthreads 16 \
    --omp-nthreads 8 \
    --mem-mb 40000 \
    --low-mem --stop-on-first-crash \
    --fs-license-file /SGLR/FREESURFER_HOME/license.txt \
    --skip-bids-validation \
    --force-bbr \
    --fd-spike-threshold 0.9 \
    --dvars-spike-threshold 5 \
    --cifti-output 91k \
    --output-spaces MNI152NLin6Asym:res-2 MNI152NLin2009cAsym \
    --use-syn-sdc \
    --notrack \
    --medial-surface-nan \
    -v -v \
    --participant-label sub-095

Version: fMRIPrep 24.1.0

Environment (Docker, Singularity / Apptainer, custom installation): Singularity (within BABS)

Data formatted according to a validatable standard? Please provide the output of the validator: BIDS

Relevant log outputs (up to 20 lines):

Working directory: /om2/scratch/tmp/yibei/fmriprep_compute/job-38898175-sub-095/ds/.git/tmp/wkdir/fmriprep_24_1_wf/sub_095_wf/fmap_preproc_wf/wf_auto_00004/bs_filter

Node inputs:

bs_spacing = [(100.0, 100.0, 40.0), (16.0, 16.0, 10.0)]
debug = False
extrapolate = True
in_data = <undefined>
in_mask = <undefined>
recenter = False
ridge_alpha = 0.0001
zooms_min = 1.0

Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node bs_filter.

Traceback:
        Traceback (most recent call last):
          File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 397, in run
            runtime = self._run_interface(runtime)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/sdcflows/interfaces/bspline.py", line 225, in _run_interface
            raise RuntimeError(
        RuntimeError: Spline fit of input file /om2/scratch/tmp/yibei/fmriprep_compute/job-38898175-sub-095/ds/.git/tmp/wkdir/fmriprep_24_1_wf/sub_095_wf/fmap_preproc_wf/wf_auto_00004/zooms_field/fmap_syn0Warp_Hz_trans.nii.gz failed. Extreme value 2.21e+04 detected in spline coefficients.

Screenshots / relevant information:

effigies · September 30, 2024, 7:11pm

Does rerunning help? You should be able to use the same working directory, and just rerun the spline filter. Since the regression is stochastic, there’s a decent chance a second try would succeed.

yibeichen · September 30, 2024, 7:25pm

Yes, I’m rerunning it now with everything. I also removed other irrelevant tasks. I’ll close this one if it succeeds. Thanks!

const · October 22, 2024, 1:46pm

I encountered the same error while using fMRIPrep 24.1.1 with fmriprep-docker on both Windows 11 (WSL2) and Ubuntu.
Rerunning didn’t resolve the issue in my case. Everything works well when the --use-syn-sdc flag is removed, although I’m not sure to what extent this will affect the final results.

yibeichen · October 22, 2024, 2:00pm

Hi, rerunning (HPC, CentOS) solved my problem. Based on the output, can you identify which tasks/runs having the problem?

const · October 23, 2024, 9:25am

Thanks for your information. I couldn’t identify which specific task or run encountered the error based on the fmriprep output. Below is the crash information (similar to what you have).

Node: fmriprep_24_1_wf.sub_107_wf.fmap_preproc_wf.wf_auto_00002.bs_filter
Working directory: /scratch/fmriprep_24_1_wf/sub_107_wf/fmap_preproc_wf/wf_auto_00002/bs_filter

Node inputs:

bs_spacing = [(100.0, 100.0, 40.0), (16.0, 16.0, 10.0)]
debug = False
extrapolate = True
in_data = <undefined>
in_mask = <undefined>
recenter = False
ridge_alpha = 0.0001
zooms_min = 1.0

Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node bs_filter.

Traceback:
	Traceback (most recent call last):
	  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 397, in run
	    runtime = self._run_interface(runtime)
	              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/sdcflows/interfaces/bspline.py", line 225, in _run_interface
	    raise RuntimeError(
	RuntimeError: Spline fit of input file /scratch/fmriprep_24_1_wf/sub_107_wf/fmap_preproc_wf/wf_auto_00002/zooms_field/fmap_syn0Warp_Hz_trans.nii.gz failed. Extreme value 1.81e+05 detected in spline coefficients.

My command looks simply like this:

fmriprep-docker "${FMRI_BIDS_DIR}" "${FMRI_PREPROC_DIR}" participant --participant-label "$pid" --use-syn-sdc --fs-license-file "${LICENSE_PATH}" --stop-on-first-crash | tee "${log_path}" &

I previously ran it successfully with other versions of fMRIPrep, but with this latest version (2024.1.1), the error occurs for all participants. Therefore, I lean towards ruling out stochastic regression as a potential random cause.

dmd · November 3, 2024, 1:10pm

Chiming in - we are also getting this exact error. Always the same handful of participants, and 100% of the time on those particular participants.

Using fmriprep 23.2 fixes the problem.

dmd · December 1, 2024, 10:31pm

Any ideas what to do here other than go back to using 23? Rerunning does NOT help - for us, anyone who fails on this always fails, and rerunning using 23 always succeeds.