Summary of what happened:
I tried to run fMRIprep to preprocess functional data for about 30 subjects. After running about 1 day, it showed the following error and I do not know why.
Command used (and if a helper script was used, a link to the helper script or the command generated):
#!/bin/bash
#User inputs:
HOME=/share/home/candy_group
bids_root_dir=/share/home/candy_group/ZJJ/data/SUDMEX_CONN/
#subj=015
nthreads=30
mem=100 #gb
container=singularity #docker or singularity
#Begin:
#Convert virtual memory from gb to mb
mem=`echo "${mem//[!0-9]/}"` #remove gb at end
mem_mb=`echo $(((mem*1000)-5000))` #reduce some memory for buffer space during pre-processing
#export TEMPLATEFLOW_HOME=$HOME/.cache/templateflow
FREESURFER_HOME=$HOME/yzheng/software/FreeSurferLicense
TEMPLATEFLOW_HOME=/share/home/candy_group/yzheng/software/templateflow
#Run fmriprep
singularity run --cleanenv -B $TEMPLATEFLOW_HOME:/templateflow $HOME/yzheng/software/my_images/fmriprep-latest.simg \
$bids_root_dir $HOME/yzheng/project5/preprocessing_volume/derivatives2 \
participant \
--participant-label 061 062 064 065 066 068 069 071 072 073 076 078 080 081 082 084 085 086 087 088 091 092 093 095 097 098 099 100 101 102 104 106 107 109 110 \
--skip-bids-validation \
--md-only-boilerplate \
--fs-license-file $FREESURFER_HOME/license.txt \
--dummy-scans 5 \
--fs-no-reconall \
--output-spaces MNI152NLin6Asym:res-2 \
--nthreads $nthreads \
--stop-on-first-crash \
--mem_mb $mem_mb \
-w $HOME/yzheng/project5/preprocessing_volume/work2
Version:
latest
Environment (Docker, Singularity / Apptainer, custom installation):
singularity
Data formatted according to a validatable standard? Please provide the output of the validator:
yes BIDS
Relevant log outputs (up to 20 lines):
241013-23:48:23,121 nipype.workflow WARNING:
[Node] Error on "fmriprep_24_1_wf.sub_061_wf.bold_task_rest_wf.bold_native_wf.boldref_fmap" (/share/home/candy_group/yzheng/project5/preprocessing_volume/work2/fmriprep_24_1_wf/sub_061_wf/bold_task_rest_wf/bold_native_wf/boldref_fmap)
2024-10-13 23:48:23,121 [ WARNING] [Node] Error on "fmriprep_24_1_wf.sub_061_wf.bold_task_rest_wf.bold_native_wf.boldref_fmap" (/share/home/candy_group/yzheng/project5/preprocessing_volume/work2/fmriprep_24_1_wf/sub_061_wf/bold_task_rest_wf/bold_native_wf/boldref_fmap)
241013-23:48:23,349 nipype.workflow ERROR:
Node boldref_fmap failed to run on host c1n02.
2024-10-13 23:48:23,349 [ ERROR] Node boldref_fmap failed to run on host c1n02.
241013-23:48:23,353 nipype.workflow ERROR:
Saving crash info to /share/home/candy_group/yzheng/project5/preprocessing_volume/derivatives2/sub-061/log/20241012-181225_4f4edb9b-ed47-4a4f-a31c-147bec71517c/crash-20241013-234823-candy_group-boldref_fmap-3bad4ef3-6a74-404c-b19b-8e9546198e96.txt
Traceback (most recent call last):
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
result["result"] = node.run(updatehash=updatehash)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
result = self._run_interface(execute=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
return self._run_command(execute)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node boldref_fmap.
Traceback:
Traceback (most recent call last):
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 397, in run
runtime = self._run_interface(runtime)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/fmriprep/interfaces/resampling.py", line 171, in _run_interface
fieldmap = reconstruct_fieldmap(
^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/fmriprep/interfaces/resampling.py", line 674, in reconstruct_fieldmap
raise ValueError('Reference passed is not aligned with spline grids')
ValueError: Reference passed is not aligned with spline grids
2024-10-13 23:48:23,353 [ ERROR] Saving crash info to /share/home/candy_group/yzheng/project5/preprocessing_volume/derivatives2/sub-061/log/20241012-181225_4f4edb9b-ed47-4a4f-a31c-147bec71517c/crash-20241013-234823-candy_group-boldref_fmap-3bad4ef3-6a74-404c-b19b-8e9546198e96.txt
Traceback (most recent call last):
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
result["result"] = node.run(updatehash=updatehash)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
result = self._run_interface(execute=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
return self._run_command(execute)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node boldref_fmap.
Traceback:
Traceback (most recent call last):
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 397, in run
runtime = self._run_interface(runtime)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/fmriprep/interfaces/resampling.py", line 171, in _run_interface
fieldmap = reconstruct_fieldmap(
^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/fmriprep/interfaces/resampling.py", line 674, in reconstruct_fieldmap
raise ValueError('Reference passed is not aligned with spline grids')
ValueError: Reference passed is not aligned with spline grids
2024-10-13 23:48:23,386 [ WARNING] WARNING: sentry_sdk.push_scope is deprecated and will be removed in the next major version. Please consult our migration guide to learn how to migrate to the new API: https://docs.sentry.io/platforms/python/migration/1.x-to-2.x#scope-pushing
241013-23:48:23,423 nipype.workflow CRITICAL:
fMRIPrep failed: Traceback (most recent call last):
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
result["result"] = node.run(updatehash=updatehash)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
result = self._run_interface(execute=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
return self._run_command(execute)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node boldref_fmap.
Traceback:
Traceback (most recent call last):
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 397, in run
runtime = self._run_interface(runtime)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/fmriprep/interfaces/resampling.py", line 171, in _run_interface
fieldmap = reconstruct_fieldmap(
^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/fmriprep/interfaces/resampling.py", line 674, in reconstruct_fieldmap
raise ValueError('Reference passed is not aligned with spline grids')
ValueError: Reference passed is not aligned with spline grids
2024-10-13 23:48:23,423 [CRITICAL] fMRIPrep failed: Traceback (most recent call last):
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
result["result"] = node.run(updatehash=updatehash)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
result = self._run_interface(execute=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
return self._run_command(execute)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node boldref_fmap.
Traceback:
Traceback (most recent call last):
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 397, in run
runtime = self._run_interface(runtime)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/fmriprep/interfaces/resampling.py", line 171, in _run_interface
fieldmap = reconstruct_fieldmap(
^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/fmriprep/interfaces/resampling.py", line 674, in reconstruct_fieldmap
raise ValueError('Reference passed is not aligned with spline grids')
ValueError: Reference passed is not aligned with spline grids