Summary of what happened:
I’m running a fMRIPrep singularity container (v22.1.0) on my university cluster and running into an error during the “resample” node.
This is only happening for 2/40 participants.
Command used (and if a helper script was used, a link to the helper script or the command generated):
sbatch << EOT
#!/usr/bin/bash
#SBATCH --account=p30952
#SBATCH --partition=normal
#SBATCH --time=10:00:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=16
#SBATCH --mem-per-cpu=4G
#SBATCH --job-name=${JOB_NAME}
# Outputs ----------------------------------
#SBATCH --mail-user=${email}
#SBATCH --mail-type=FAIL
#SBATCH --output=${BIDS_DIR}/code/slurm_logs/out.${JOB_NAME}
#SBATCH --error=${BIDS_DIR}/code/slurm_logs/err.${JOB_NAME}
# ------------------------------------------
# Load modules
module purge
module load singularity
echo "Modules loaded"
# Remove IsRunning files from Freesurfer
[ -f ${SUBJECTS_DIR}/$subject/scripts/IsRunning.lh+rh ] && rm -rf ${SUBJECTS_DIR}/$subject/scripts/IsRunning.lh+rh
# Do singularity run
echo "Begin Preprocessing"
singularity run \
--cleanenv -B /projects \
/projects/p30952/environment/singularity_images/${FMRIPREP_VER}.simg \
${BIDS_DIR} \
${FMRIPREP_DIR} \
participant --participant-label ${subject} \
-w ${WORK_DIR} \
--omp-nthreads 8 \
--nthreads 12 \
--mem_mb 24000 \
--fs-license-file ${FS_LICENSE} \
--fs-subjects-dir ${FMRIPREP_DIR}/sourcedata/freesurfer \
--output-spaces anat fsnative MNI152NLin2009cAsym fsLR fsaverage \
--cifti-output 91k \
--fd-spike-threshold 0.4 \
--dvars-spike-threshold 1.5 \
--use-syn-sdc warn \
--dummy-scans 4
exit 0
EOT
Version:
22.1.0
Environment (Docker, Singularity, custom installation):
Singularity
Data formatted according to a validatable standard? Please provide the output of the validator:
Yes
Relevant log outputs (up to 20 lines):
[Node] Error on "fmriprep_22_1_wf.single_subject_SA175_wf.func_preproc_ses_b_task_rest_run_1_wf.unwarp_wf.resample" (/projects/b1145/bram/work/fmriprep_22_1_wf/single_subject_SA175_wf/func_preproc_ses_b_task_rest_run_1_wf/unwarp_wf/resample)
230209-15:41:56,18 nipype.workflow ERROR:
Node resample failed to run on host qnode8113.
230209-15:41:56,28 nipype.workflow ERROR:
Saving crash info to /projects/b1145/bram/sa_rsfmri/derivatives/fmriprep-22.1.0/sub-SA175/log/20230209-153320_3eea714b-ea06-4b49-89b4-407bae110c00/crash-20230209-154156-brd2241-resample-349fdf07-532d-468e-b61f-ae20fa4c8753.txt
Traceback (most recent call last):
File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
result["result"] = node.run(updatehash=updatehash)
File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
result = self._run_interface(execute=True)
File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
return self._run_command(execute)
File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node resample.
Traceback:
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/opt/conda/lib/python3.9/concurrent/futures/process.py", line 243, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/opt/conda/lib/python3.9/concurrent/futures/process.py", line 202, in _process_chunk
return [fn(*args) for args in chunk]
File "/opt/conda/lib/python3.9/concurrent/futures/process.py", line 202, in <listcomp>
return [fn(*args) for args in chunk]
File "/opt/conda/lib/python3.9/site-packages/sdcflows/interfaces/bspline.py", line 630, in _b0_resampler
unwarped_img = unwarp.apply(distorted_img, ro_time=ro, pe_dir=pe)
File "/opt/conda/lib/python3.9/site-packages/sdcflows/transform.py", line 171, in apply
self.fit(spatialimage)
File "/opt/conda/lib/python3.9/site-packages/sdcflows/transform.py", line 99, in fit
wmat = grid_bspline_weights(spatialimage, moved_cs)
File "/opt/conda/lib/python3.9/site-packages/sdcflows/transform.py", line 417, in grid_bspline_weights
colloc_ax = BSpline.design_matrix(x, t, 3)[:, (2 + pad_left):-(2 + pad_right)]
File "/opt/conda/lib/python3.9/site-packages/scipy/interpolate/_bsplines.py", line 416, in design_matrix
raise ValueError(f'Out of bounds w/ x = {x}.')
ValueError: Out of bounds w/ x = [ 4.19654081 4.29093281 4.38532481 4.47971681 4.57410882 4.66850082
4.76289282 4.85728482 4.95167682 5.04606882 5.14046082 5.23485283
5.32924483 5.42363683 5.51802883 5.61242083 5.70681283 5.80120483
5.89559684 5.98998884 6.08438084 6.17877284 6.27316484 6.36755684
6.46194884 6.55634085 6.65073285 6.74512485 6.83951685 6.93390885
7.02830085 7.12269285 7.21708486 7.31147686 7.40586886 7.50026086
7.59465286 7.68904486 7.78343686 7.87782887 7.97222087 8.06661287
8.16100487 8.25539687 8.34978887 8.44418088 8.53857288 8.63296488
8.72735688 8.82174888 8.91614088 9.01053288 9.10492489 9.19931689
9.29370889 9.38810089 9.48249289 9.57688489 9.67127689 9.7656689
9.8600609 9.9544529 10.0488449 10.1432369 10.2376289 10.3320209
10.42641291 10.52080491 10.61519691 10.70958891 10.80398091 10.89837291
10.99276491 11.08715692 11.18154892 11.27594092 11.37033292 11.46472492
11.55911692 11.65350892 11.74790093 11.84229293 11.93668493 12.03107693
12.12546893 12.21986093 12.31425293 12.40864494 12.50303694 12.59742894
12.69182094 12.78621294 12.88060494 12.97499694 13.06938895 13.16378095
13.25817295 13.35256495 13.44695695 13.54134895 13.63574095 13.73013296
13.82452496 13.91891696 14.01330896 14.10770096 14.20209296 14.29648496
14.39087697 14.48526897 14.57966097 14.67405297 14.76844497 14.86283697
14.95722897 15.05162098 15.14601298 15.24040498 15.33479698 15.42918898].
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 398, in run
runtime = self._run_interface(runtime)
File "/opt/conda/lib/python3.9/site-packages/sdcflows/interfaces/bspline.py", line 373, in _run_interface
) = zip(*outputs)
File "/opt/conda/lib/python3.9/concurrent/futures/process.py", line 559, in _chain_from_iterable_of_lists
for element in iterable:
File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 609, in result_iterator
yield fs.pop().result()
File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 439, in result
return self.__get_result()
File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
ValueError: Out of bounds w/ x = [ 4.19654081 4.29093281 4.38532481 4.47971681 4.57410882 4.66850082
4.76289282 4.85728482 4.95167682 5.04606882 5.14046082 5.23485283
5.32924483 5.42363683 5.51802883 5.61242083 5.70681283 5.80120483
5.89559684 5.98998884 6.08438084 6.17877284 6.27316484 6.36755684
6.46194884 6.55634085 6.65073285 6.74512485 6.83951685 6.93390885
7.02830085 7.12269285 7.21708486 7.31147686 7.40586886 7.50026086
7.59465286 7.68904486 7.78343686 7.87782887 7.97222087 8.06661287
8.16100487 8.25539687 8.34978887 8.44418088 8.53857288 8.63296488
8.72735688 8.82174888 8.91614088 9.01053288 9.10492489 9.19931689
9.29370889 9.38810089 9.48249289 9.57688489 9.67127689 9.7656689
9.8600609 9.9544529 10.0488449 10.1432369 10.2376289 10.3320209
10.42641291 10.52080491 10.61519691 10.70958891 10.80398091 10.89837291
10.99276491 11.08715692 11.18154892 11.27594092 11.37033292 11.46472492
11.55911692 11.65350892 11.74790093 11.84229293 11.93668493 12.03107693
12.12546893 12.21986093 12.31425293 12.40864494 12.50303694 12.59742894
12.69182094 12.78621294 12.88060494 12.97499694 13.06938895 13.16378095
13.25817295 13.35256495 13.44695695 13.54134895 13.63574095 13.73013296
13.82452496 13.91891696 14.01330896 14.10770096 14.20209296 14.29648496
14.39087697 14.48526897 14.57966097 14.67405297 14.76844497 14.86283697
14.95722897 15.05162098 15.14601298 15.24040498 15.33479698 15.42918898].
230209-15:41:58,16 nipype.workflow ERROR:
could not run node: fmriprep_22_1_wf.single_subject_SA175_wf.func_preproc_ses_b_task_rest_run_1_wf.unwarp_wf.resample
230209-15:41:58,87 nipype.workflow CRITICAL:
fMRIPrep failed: Traceback (most recent call last):
File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
result["result"] = node.run(updatehash=updatehash)
File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
result = self._run_interface(execute=True)
File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
return self._run_command(execute)
File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node resample.
Traceback:
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/opt/conda/lib/python3.9/concurrent/futures/process.py", line 243, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/opt/conda/lib/python3.9/concurrent/futures/process.py", line 202, in _process_chunk
return [fn(*args) for args in chunk]
File "/opt/conda/lib/python3.9/concurrent/futures/process.py", line 202, in <listcomp>
return [fn(*args) for args in chunk]
File "/opt/conda/lib/python3.9/site-packages/sdcflows/interfaces/bspline.py", line 630, in _b0_resampler
unwarped_img = unwarp.apply(distorted_img, ro_time=ro, pe_dir=pe)
File "/opt/conda/lib/python3.9/site-packages/sdcflows/transform.py", line 171, in apply
self.fit(spatialimage)
File "/opt/conda/lib/python3.9/site-packages/sdcflows/transform.py", line 99, in fit
wmat = grid_bspline_weights(spatialimage, moved_cs)
File "/opt/conda/lib/python3.9/site-packages/sdcflows/transform.py", line 417, in grid_bspline_weights
colloc_ax = BSpline.design_matrix(x, t, 3)[:, (2 + pad_left):-(2 + pad_right)]
File "/opt/conda/lib/python3.9/site-packages/scipy/interpolate/_bsplines.py", line 416, in design_matrix
raise ValueError(f'Out of bounds w/ x = {x}.')
ValueError: Out of bounds w/ x = [ 4.19654081 4.29093281 4.38532481 4.47971681 4.57410882 4.66850082
4.76289282 4.85728482 4.95167682 5.04606882 5.14046082 5.23485283
5.32924483 5.42363683 5.51802883 5.61242083 5.70681283 5.80120483
5.89559684 5.98998884 6.08438084 6.17877284 6.27316484 6.36755684
6.46194884 6.55634085 6.65073285 6.74512485 6.83951685 6.93390885
7.02830085 7.12269285 7.21708486 7.31147686 7.40586886 7.50026086
7.59465286 7.68904486 7.78343686 7.87782887 7.97222087 8.06661287
8.16100487 8.25539687 8.34978887 8.44418088 8.53857288 8.63296488
8.72735688 8.82174888 8.91614088 9.01053288 9.10492489 9.19931689
9.29370889 9.38810089 9.48249289 9.57688489 9.67127689 9.7656689
9.8600609 9.9544529 10.0488449 10.1432369 10.2376289 10.3320209
10.42641291 10.52080491 10.61519691 10.70958891 10.80398091 10.89837291
10.99276491 11.08715692 11.18154892 11.27594092 11.37033292 11.46472492
11.55911692 11.65350892 11.74790093 11.84229293 11.93668493 12.03107693
12.12546893 12.21986093 12.31425293 12.40864494 12.50303694 12.59742894
12.69182094 12.78621294 12.88060494 12.97499694 13.06938895 13.16378095
13.25817295 13.35256495 13.44695695 13.54134895 13.63574095 13.73013296
13.82452496 13.91891696 14.01330896 14.10770096 14.20209296 14.29648496
14.39087697 14.48526897 14.57966097 14.67405297 14.76844497 14.86283697
14.95722897 15.05162098 15.14601298 15.24040498 15.33479698 15.42918898].
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 398, in run
runtime = self._run_interface(runtime)
File "/opt/conda/lib/python3.9/site-packages/sdcflows/interfaces/bspline.py", line 373, in _run_interface
) = zip(*outputs)
File "/opt/conda/lib/python3.9/concurrent/futures/process.py", line 559, in _chain_from_iterable_of_lists
for element in iterable:
File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 609, in result_iterator
yield fs.pop().result()
File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 439, in result
return self.__get_result()
File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
ValueError: Out of bounds w/ x = [ 4.19654081 4.29093281 4.38532481 4.47971681 4.57410882 4.66850082
4.76289282 4.85728482 4.95167682 5.04606882 5.14046082 5.23485283
5.32924483 5.42363683 5.51802883 5.61242083 5.70681283 5.80120483
5.89559684 5.98998884 6.08438084 6.17877284 6.27316484 6.36755684
6.46194884 6.55634085 6.65073285 6.74512485 6.83951685 6.93390885
7.02830085 7.12269285 7.21708486 7.31147686 7.40586886 7.50026086
7.59465286 7.68904486 7.78343686 7.87782887 7.97222087 8.06661287
8.16100487 8.25539687 8.34978887 8.44418088 8.53857288 8.63296488
8.72735688 8.82174888 8.91614088 9.01053288 9.10492489 9.19931689
9.29370889 9.38810089 9.48249289 9.57688489 9.67127689 9.7656689
9.8600609 9.9544529 10.0488449 10.1432369 10.2376289 10.3320209
10.42641291 10.52080491 10.61519691 10.70958891 10.80398091 10.89837291
10.99276491 11.08715692 11.18154892 11.27594092 11.37033292 11.46472492
11.55911692 11.65350892 11.74790093 11.84229293 11.93668493 12.03107693
12.12546893 12.21986093 12.31425293 12.40864494 12.50303694 12.59742894
12.69182094 12.78621294 12.88060494 12.97499694 13.06938895 13.16378095
13.25817295 13.35256495 13.44695695 13.54134895 13.63574095 13.73013296
13.82452496 13.91891696 14.01330896 14.10770096 14.20209296 14.29648496
14.39087697 14.48526897 14.57966097 14.67405297 14.76844497 14.86283697
14.95722897 15.05162098 15.14601298 15.24040498 15.33479698 15.42918898].
230209-15:42:01,245 cli ERROR:
Preprocessing did not finish successfully. Errors occurred while processing data from participants: SA175 (1). Check the HTML reports for details.
230209-15:42:01,245 cli ERROR:
Preprocessing did not finish successfully. Errors occurred while processing data from participants: SA175 (1). Check the HTML reports for details.
Screenshots / relevant information:
N/A