fMRIPrep 24.1.0 failed during `bs_filter` (Extreme value 2.21e+04 detected in spline coefficients)

Summary of what happened:

I’m using fMRIPrep 24.1.0 on a subset (N=40) of the Narratives dataset, there are two subjects failed due to the same reason:

Extreme value 2.21e+04 detected in spline coefficients

when running "fmriprep_24_1_wf.sub_095_wf.fmap_preproc_wf.wf_auto_00004.bs_filter"

Command used (and if a helper script was used, a link to the helper script or the command generated):

singularity run --cleanenv \
    -B /om2/scratch/tmp/yibei/fmriprep_compute/job-38898175-sub-095/ds \
    -B /nese/mit/group/sig/projects/hbn/hbn_bids/code/license.txt:/SGLR/FREESURFER_HOME/license.txt \
    containers/.datalad/environments/fmriprep-24-1-0/image \
    inputs/data/BIDS \
    outputs/fmriprep \
    participant \
    -w /om2/scratch/tmp/yibei/fmriprep_compute/job-38898175-sub-095/ds/.git/tmp/wkdir 
    --nthreads 16 \
    --omp-nthreads 8 \
    --mem-mb 40000 \
    --low-mem --stop-on-first-crash \
    --fs-license-file /SGLR/FREESURFER_HOME/license.txt \
    --skip-bids-validation \
    --force-bbr \
    --fd-spike-threshold 0.9 \
    --dvars-spike-threshold 5 \
    --cifti-output 91k \
    --output-spaces MNI152NLin6Asym:res-2 MNI152NLin2009cAsym \
    --use-syn-sdc \
    --notrack \
    --medial-surface-nan \
    -v -v \
    --participant-label sub-095

Version: fMRIPrep 24.1.0

Environment (Docker, Singularity / Apptainer, custom installation): Singularity (within BABS)

Data formatted according to a validatable standard? Please provide the output of the validator: BIDS

Relevant log outputs (up to 20 lines):

Working directory: /om2/scratch/tmp/yibei/fmriprep_compute/job-38898175-sub-095/ds/.git/tmp/wkdir/fmriprep_24_1_wf/sub_095_wf/fmap_preproc_wf/wf_auto_00004/bs_filter

Node inputs:

bs_spacing = [(100.0, 100.0, 40.0), (16.0, 16.0, 10.0)]
debug = False
extrapolate = True
in_data = <undefined>
in_mask = <undefined>
recenter = False
ridge_alpha = 0.0001
zooms_min = 1.0

Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node bs_filter.

Traceback:
        Traceback (most recent call last):
          File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 397, in run
            runtime = self._run_interface(runtime)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/sdcflows/interfaces/bspline.py", line 225, in _run_interface
            raise RuntimeError(
        RuntimeError: Spline fit of input file /om2/scratch/tmp/yibei/fmriprep_compute/job-38898175-sub-095/ds/.git/tmp/wkdir/fmriprep_24_1_wf/sub_095_wf/fmap_preproc_wf/wf_auto_00004/zooms_field/fmap_syn0Warp_Hz_trans.nii.gz failed. Extreme value 2.21e+04 detected in spline coefficients.

Screenshots / relevant information:


Does rerunning help? You should be able to use the same working directory, and just rerun the spline filter. Since the regression is stochastic, there’s a decent chance a second try would succeed.

1 Like

Yes, I’m rerunning it now with everything. I also removed other irrelevant tasks. I’ll close this one if it succeeds. Thanks!

I encountered the same error while using fMRIPrep 24.1.1 with fmriprep-docker on both Windows 11 (WSL2) and Ubuntu.
Rerunning didn’t resolve the issue in my case. Everything works well when the --use-syn-sdc flag is removed, although I’m not sure to what extent this will affect the final results.

Hi, rerunning (HPC, CentOS) solved my problem. Based on the output, can you identify which tasks/runs having the problem?

Thanks for your information. I couldn’t identify which specific task or run encountered the error based on the fmriprep output. Below is the crash information (similar to what you have).

Node: fmriprep_24_1_wf.sub_107_wf.fmap_preproc_wf.wf_auto_00002.bs_filter
Working directory: /scratch/fmriprep_24_1_wf/sub_107_wf/fmap_preproc_wf/wf_auto_00002/bs_filter

Node inputs:

bs_spacing = [(100.0, 100.0, 40.0), (16.0, 16.0, 10.0)]
debug = False
extrapolate = True
in_data = <undefined>
in_mask = <undefined>
recenter = False
ridge_alpha = 0.0001
zooms_min = 1.0

Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node bs_filter.

Traceback:
	Traceback (most recent call last):
	  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 397, in run
	    runtime = self._run_interface(runtime)
	              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/sdcflows/interfaces/bspline.py", line 225, in _run_interface
	    raise RuntimeError(
	RuntimeError: Spline fit of input file /scratch/fmriprep_24_1_wf/sub_107_wf/fmap_preproc_wf/wf_auto_00002/zooms_field/fmap_syn0Warp_Hz_trans.nii.gz failed. Extreme value 1.81e+05 detected in spline coefficients.

My command looks simply like this:

fmriprep-docker "${FMRI_BIDS_DIR}" "${FMRI_PREPROC_DIR}" participant --participant-label "$pid" --use-syn-sdc --fs-license-file "${LICENSE_PATH}" --stop-on-first-crash | tee "${log_path}" &

I previously ran it successfully with other versions of fMRIPrep, but with this latest version (2024.1.1), the error occurs for all participants. Therefore, I lean towards ruling out stochastic regression as a potential random cause.

Chiming in - we are also getting this exact error. Always the same handful of participants, and 100% of the time on those particular participants.

Using fmriprep 23.2 fixes the problem.

Any ideas what to do here other than go back to using 23? Rerunning does NOT help - for us, anyone who fails on this always fails, and rerunning using 23 always succeeds.

Hi, we have the same problem (when running with docker on Ubuntu), re-running did not fix the issue, were you able to solve the issue on version 24?

Summary of what happened:

Good afternoon,

I am trying to preprocess a dataset where gradient fieldmaps are not available and I was hoping to get the SyN distortion correction to run using the --use-syn-sdc flag. The subjects contain various sessions, and when running fMRIPrep the SyN algorithm is able to generate fieldmaps for only a few of the sessions out of all of them. This results in no output generated. Moreover, this is not consistent and rerunning only results in successful and unsuccessful attempts across some of the sessions (some of them work sometimes, they will not work however other times and other sessions don’t work at all in any runs).

I have tried to run just one session where the SyN has successfully created a fieldmap and output was generated as a consequence. My goal however is to get SyN running across all sessions so that I can get the full Output in order to complete the research (at least some of the sessions are a must and I am still not able to get the desired sessions working with SyN).

I highly doubt data quality is an issue since the T1w anatomical image is of good resolution and the funcional image is standard 3x3x3mm voxel image. I have read in Methods and implementation - sdcflows 0+unknown documentation that SyN is using skull-strippet anatomical image so I don’t really know if using the --skull-strip-t1w flag will do any good.

Any other solution I can try? Thank you very much in advance.

Doneka

Command used (and if a helper script was used, a link to the helper script or the command generated):

singularity run --cleanenv ./sandbox/fmriprep-24.1.1.simg/ ./data/ ./Output participant --participant-label astronaut03 --fs-license-file /home/dloin01/license.txt --output-spaces T1w MNI152NLin2009cAsym func --use-syn-sdc warn --verbose --notrack --ignore slicetiming fieldmaps --md-only-boilerplate --stop-on-first-crash

Version:

fMRIPrep version 24.1.1

Environment (Docker, Singularity / Apptainer, custom installation):

Singularity container

Data formatted according to a validatable standard? Please provide the output of the validator:

Compliant with BIDS standard

Relevant log outputs (up to 20 lines):

RuntimeError: Spline fit of input file /home/dloin01/syn_nolong_preproc_tryout/work/fmriprep_24_1_wf/sub_astronaut03_wf/fmap_preproc_wf/wf_auto_00000/zooms_field/fmap_syn0Warp_Hz_trans.nii.gz failed. Extreme value 1.43e+04 detected in spline coefficients.