Summary of what happened:
I’m using fmriprep (23.1.4) on a movie-watching dataset (N=25, n_run = 5). almost all subjects failed at the node midthickness0
Command used (and if a helper script was used, a link to the helper script or the command generated):
I’m running fmriprep thru BABS on HPC (slurm); the whole BABS setup was successful.
here is the fmriprep setup in BABS
singularity run --cleanenv \
-B ${PWD} \
-B /om2/user/$(whoami)/images/license.txt:/SGLR/FREESURFER_HOME/license.txt \
containers/.datalad/environments/fmriprep-23-1-4/image \
inputs/data/BIDS \
outputs/fmriprep \
participant \
-w ${PWD}/.git/tmp/wkdir \
--stop-on-first-crash \
--nthreads 16 \
--omp-nthreads 8 \
--mem-mb 40000 \
--fd-spike-threshold 0.9 \
--dvars-spike-threshold 5 \
--fs-license-file /SGLR/FREESURFER_HOME/license.txt \
--skip-bids-validation \
--output-layout legacy \
--force-bbr \
--notrack \
--cifti-output 91k \
-v -v \
--participant-label "${subid}"
Version:
fmriprep 23.1.4
Environment (Docker, Singularity, custom installation):
Data formatted according to a validatable standard? Please provide the output of the validator:
The input data is BIDS formated see here
Relevant log outputs (up to 20 lines):
here is the crash file from fmriprep
Node: _midthickness0
Working directory: /om2/scratch/Sun/yibei/budapest/output/job-32194722-sub-sid000030/ds/.git/tmp/wkdir/fmriprep_23_1_wf/single_subject_sid000030_wf/anat_preproc_wf/surface_recon_wf/gifti_surface_wf/midthickness/mapflow/_midthickness0
Node inputs:
args = <undefined>
distance = 0.5
dt = <undefined>
environ = {'SUBJECTS_DIR': '/opt/freesurfer/subjects'}
graymid = <undefined>
in_file = /om2/scratch/Sun/yibei/budapest/output/job-32194722-sub-sid000030/ds/outputs/fmriprep/freesurfer/sub-sid000030/surf/lh.white
nsurfaces = <undefined>
out_name = midthickness
pial = <undefined>
smooth_averages = <undefined>
sphere = sphere
spring = <undefined>
subjects_dir = /opt/freesurfer/subjects
thickness = True
thickness_name = <undefined>
write_iterations = <undefined>
Traceback (most recent call last):
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
result["result"] = node.run(updatehash=updatehash)
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
result = self._run_interface(execute=True)
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
return self._run_command(execute)
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node _midthickness0.
Cmdline:
mris_expand -pial /om2/scratch/Sun/yibei/budapest/output/job-32194722-sub-sid000030/ds/.git/tmp/wkdir/fmriprep_23_1_wf/single_subject_sid000030_wf/anat_preproc_wf/surface_recon_wf/gifti_surface_wf/midthickness/mapflow/_midthickness0/lh.pial -thickness -thickness_name /om2/scratch/Sun/yibei/budapest/output/job-32194722-sub-sid000030/ds/.git/tmp/wkdir/fmriprep_23_1_wf/single_subject_sid000030_wf/anat_preproc_wf/surface_recon_wf/gifti_surface_wf/midthickness/mapflow/_midthickness0/lh.thickness /om2/scratch/Sun/yibei/budapest/output/job-32194722-sub-sid000030/ds/.git/tmp/wkdir/fmriprep_23_1_wf/single_subject_sid000030_wf/anat_preproc_wf/surface_recon_wf/gifti_surface_wf/midthickness/mapflow/_midthickness0/lh.white 0.5 /om2/scratch/Sun/yibei/budapest/output/job-32194722-sub-sid000030/ds/.git/tmp/wkdir/fmriprep_23_1_wf/single_subject_sid000030_wf/anat_preproc_wf/surface_recon_wf/gifti_surface_wf/midthickness/mapflow/_midthickness0/lh.midthickness
Stdout:
reading pial surface from /om2/scratch/Sun/yibei/budapest/output/job-32194722-sub-sid000030/ds/.git/tmp/wkdir/fmriprep_23_1_wf/single_subject_sid000030_wf/anat_preproc_wf/surface_recon_wf/gifti_surface_wf/midthickness/mapflow/_midthickness0/lh.pial
using distance as a % of thickness
using thickness file /om2/scratch/Sun/yibei/budapest/output/job-32194722-sub-sid000030/ds/.git/tmp/wkdir/fmriprep_23_1_wf/single_subject_sid000030_wf/anat_preproc_wf/surface_recon_wf/gifti_surface_wf/midthickness/mapflow/_midthickness0/lh.thickness
expanding surface /om2/scratch/Sun/yibei/budapest/output/job-32194722-sub-sid000030/ds/.git/tmp/wkdir/fmriprep_23_1_wf/single_subject_sid000030_wf/anat_preproc_wf/surface_recon_wf/gifti_surface_wf/midthickness/mapflow/_midthickness0/lh.white by 50.0% of thickness and writing it to /om2/scratch/Sun/yibei/budapest/output/job-32194722-sub-sid000030/ds/.git/tmp/wkdir/fmriprep_23_1_wf/single_subject_sid000030_wf/anat_preproc_wf/surface_recon_wf/gifti_surface_wf/midthickness/mapflow/_midthickness0/lh.midthickness
reading thickness...
Stderr:
*** buffer overflow detected ***: terminated
Aborted (core dumped)
Traceback:
RuntimeError: subprocess exited with code 134.
Screenshots / relevant information:
here is the information about this folder
/fmriprep_23_1_wf/single_subject_sid000030_wf/anat_preproc_wf/surface_recon_wf/gifti_surface_wf/midthickness/mapflow/_midthickness0/
ls -ls
total 5
1 -rw-r--r-- 1 yibei root 954 Aug 23 22:27 command.txt
1 -rw-r--r-- 1 yibei root 264 Aug 23 22:27 _inputs.pklz
0 lrwxrwxrwx 1 yibei root 123 Aug 23 22:27 lh.pial -> /om2/scratch/Sun/yibei/budapest/output/job-32194716-sub-sid000010/ds/outputs/fmriprep/freesurfer/sub-sid000010/surf/lh.pial
0 lrwxrwxrwx 1 yibei root 125 Aug 23 22:27 lh.sphere -> /om2/scratch/Sun/yibei/budapest/output/job-32194716-sub-sid000010/ds/outputs/fmriprep/freesurfer/sub-sid000010/surf/lh.sphere
0 lrwxrwxrwx 1 yibei root 128 Aug 23 22:27 lh.thickness -> /om2/scratch/Sun/yibei/budapest/output/job-32194716-sub-sid000010/ds/outputs/fmriprep/freesurfer/sub-sid000010/surf/lh.thickness
0 lrwxrwxrwx 1 yibei root 124 Aug 23 22:27 lh.white -> /om2/scratch/Sun/yibei/budapest/output/job-32194716-sub-sid000010/ds/outputs/fmriprep/freesurfer/sub-sid000010/surf/lh.white
2 -rw-r--r-- 1 yibei root 1443 Aug 23 22:27 _node.pklz
0 drwxr-sr-x 2 yibei root 4096 Aug 23 22:27 _report
2 -rw-r--r-- 1 yibei root 2008 Aug 23 22:27 result__midthickness0.pklz
full command
mris_expand -pial /om2/scratch/Sun/yibei/budapest/output/job-32194716-sub-sid000010/ds/.git/tmp/wkdir/fmriprep_23_1_wf/single_subject_sid000010_wf/anat_preproc_wf/surface_recon_wf/gifti_surface_wf/midthickness/mapflow/_midthickness0/lh.pial -thickness -thickness_name /om2/scratch/Sun/yibei/budapest/output/job-32194716-sub-sid000010/ds/.git/tmp/wkdir/fmriprep_23_1_wf/single_subject_sid000010_wf/anat_preproc_wf/surface_recon_wf/gifti_surface_wf/midthickness/mapflow/_midthickness0/lh.thickness /om2/scratch/Sun/yibei/budapest/output/job-32194716-sub-sid000010/ds/.git/tmp/wkdir/fmriprep_23_1_wf/single_subject_sid000010_wf/anat_preproc_wf/surface_recon_wf/gifti_surface_wf/midthickness/mapflow/_midthickness0/lh.white 0.5 /om2/scratch/Sun/yibei/budapest/output/job-32194716-sub-sid000010/ds/.git/tmp/wkdir/fmriprep_23_1_wf/single_subject_sid000010_wf/anat_preproc_wf/surface_recon_wf/gifti_surface_wf/midthickness/mapflow/_midthickness0/lh.midthickness