fMRIPrep FreeSurfer Segmentation Fault

Shlok_Purani · March 27, 2023, 7:26pm

Summary of what happened:

I am running fMRIPrep with Singularity

Command used (and if a helper script was used, a link to the helper script or the command generated):

singularity exec --bind ~/pzero_outs/102109/001/bids/23.3.1/:/data/bids:ro --bind ~/fmriprep_outs:/data/out \
                 --bind ~/containers-main/containers/fmriprep/workdir:/data/workdir fmriprep_2101.simg \
                 fmriprep_adapter /data/bids /data/out --subject-id="102109" --session-id="001" --skip-bids-validation --force-bbr \
                 --output-layout "bids" --output-spaces "MNI152NLin6Asym:res-2" --task-id "rest" --work-dir /data/workdir

Version:

21.0.1

Environment (Docker, Singularity, custom installation):

Singularity

Data formatted according to a validatable standard? Please provide the output of the validator:

Relevant log outputs (up to 20 lines):

Here is the crash log

Node: fmriprep_wf.single_subject_102109_wf.anat_preproc_wf.surface_recon_wf.autorecon1
Working directory: /global/home/hpc5172/containers-main/containers/fmriprep/work/fmriprep_wf/single_subject_102109_wf/anat_preproc_wf/surface_recon_wf/autorecon1

Node inputs:

FLAIR_file = <undefined>
T1_files = ['/global/home/hpc5172/containers-main/containers/fmriprep/work/fmriprep_wf/single_subject_102109_wf/anat_preproc_wf/anat_validate/sub-102109_ses-001_T1w_ras_valid.nii.gz']
T2_file = /data/bids/sub-102109/ses-001/anat/sub-102109_ses-001_T2w.nii.gz
args = <undefined>
big_ventricles = <undefined>
brainstem = <undefined>
directive = autorecon1
environ = {}
expert = <undefined>
flags = ['-noskullstrip']
hemi = <undefined>
hippocampal_subfields_T1 = <undefined>
hippocampal_subfields_T2 = <undefined>
hires = True
mprage = <undefined>
mri_aparc2aseg = <undefined>
mri_ca_label = <undefined>
mri_ca_normalize = <undefined>
mri_ca_register = <undefined>
mri_edit_wm_with_aseg = <undefined>
mri_em_register = <undefined>
mri_fill = <undefined>
mri_mask = <undefined>
mri_normalize = <undefined>
mri_pretess = <undefined>
mri_remove_neck = <undefined>
mri_segment = <undefined>
mri_segstats = <undefined>
mri_tessellate = <undefined>
mri_watershed = <undefined>
mris_anatomical_stats = <undefined>
mris_ca_label = <undefined>
mris_fix_topology = <undefined>
mris_inflate = -n 50
mris_make_surfaces = <undefined>
mris_register = <undefined>
mris_smooth = <undefined>
mris_sphere = <undefined>
mris_surf2vol = <undefined>
mrisp_paint = <undefined>
openmp = 8
parallel = <undefined>
steps = <undefined>
subject_id = sub-102109
subjects_dir = /data/workdir/sourcedata/freesurfer
talairach = <undefined>
use_FLAIR = <undefined>
use_T2 = <undefined>
xopts = <undefined>

Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/opt/conda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 428, in run
    runtime = self._run_interface(runtime)
  File "/opt/conda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 822, in _run_interface
    self.raise_exception(runtime)
  File "/opt/conda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 749, in raise_exception
    raise RuntimeError(
RuntimeError: Command:
recon-all -autorecon1 -T2 /data/bids/sub-102109/ses-001/anat/sub-102109_ses-001_T2w.nii.gz -noskullstrip -hires -openmp 8 -subjid sub-102109 -sd /data/workdir/sourcedata/freesurfer -xopts-use -nomotioncor
Standard output:
INFO: hi-res volumes are conformed to the min voxel size
Subject Stamp: freesurfer-Linux-centos6_x86_64-stable-pub-v6.0.1-f53a55a
Current Stamp: freesurfer-Linux-centos6_x86_64-stable-pub-v6.0.1-f53a55a
INFO: SUBJECTS_DIR is /data/workdir/sourcedata/freesurfer
Actual FREESURFER_HOME /opt/freesurfer
-rw-rw-r-- 1 hpc5172 hpcg2016 21894 Mar 27 11:22 /data/workdir/sourcedata/freesurfer/sub-102109/scripts/recon-all.log
Linux cac030 3.10.0-862.9.1.el7.x86_64 #1 SMP Mon Jul 16 16:29:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
'/opt/freesurfer/bin/recon-all' -> '/data/workdir/sourcedata/freesurfer/sub-102109/scripts/recon-all.local-copy'
#--------------------------------------------
#@# T2/FLAIR Input Mon Mar 27 13:40:20 EDT 2023
/data/workdir/sourcedata/freesurfer/sub-102109

 mri_convert --no_scale 1 /data/bids/sub-102109/ses-001/anat/sub-102109_ses-001_T2w.nii.gz /data/workdir/sourcedata/freesurfer/sub-102109/mri/orig/T2raw.mgz

mri_convert.bin --no_scale 1 /data/bids/sub-102109/ses-001/anat/sub-102109_ses-001_T2w.nii.gz /data/workdir/sourcedata/freesurfer/sub-102109/mri/orig/T2raw.mgz
$Id: mri_convert.c,v 1.226 2016/02/26 16:15:24 mreuter Exp $
reading from /data/bids/sub-102109/ses-001/anat/sub-102109_ses-001_T2w.nii.gz...
TR=3200.00, TE=0.00, TI=0.00, flip angle=0.00
i_ras = (-0.998011, -0.0521218, -0.0354659)
j_ras = (-0.0560184, 0.99124, 0.119602)
k_ras = (-0.0289213, -0.121351, 0.992188)
writing to /data/workdir/sourcedata/freesurfer/sub-102109/mri/orig/T2raw.mgz...
#--------------------------------------------
#@# Talairach Mon Mar 27 13:40:35 EDT 2023
/data/workdir/sourcedata/freesurfer/sub-102109/mri

 mri_nu_correct.mni --no-rescale --i orig.mgz --o orig_nu.mgz --n 1 --proto-iters 1000 --distance 50

/data/workdir/sourcedata/freesurfer/sub-102109/mri
/opt/freesurfer/bin/mri_nu_correct.mni
--no-rescale --i orig.mgz --o orig_nu.mgz --n 1 --proto-iters 1000 --distance 50
nIters 1
$Id: mri_nu_correct.mni,v 1.27 2016/02/26 16:19:49 mreuter Exp $
Linux cac030 3.10.0-862.9.1.el7.x86_64 #1 SMP Mon Jul 16 16:29:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Mon Mar 27 13:40:36 EDT 2023
Program nu_correct, built from:
Package MNI N3, version 1.12.0, compiled by nicks@terrier (x86_64-unknown-linux-gnu) on 2015-06-19 at 01:25:34
/bin/bc
tmpdir is ./tmp.mri_nu_correct.mni.25890
/data/workdir/sourcedata/freesurfer/sub-102109/mri
mri_convert orig.mgz ./tmp.mri_nu_correct.mni.25890/nu0.mnc -odt float
mri_convert.bin orig.mgz ./tmp.mri_nu_correct.mni.25890/nu0.mnc -odt float
/opt/freesurfer/bin/mri_convert: line 3: 25925 Segmentation fault      (core dumped) mri_convert.bin "$@"
ERROR: converting to minc
Linux cac030 3.10.0-862.9.1.el7.x86_64 #1 SMP Mon Jul 16 16:29:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

recon-all -s sub-102109 exited with ERRORS at Mon Mar 27 13:41:23 EDT 2023

For more details, see the log file /data/workdir/sourcedata/freesurfer/sub-102109/scripts/recon-all.log
To report a problem, see http://surfer.nmr.mgh.harvard.edu/fswiki/BugReporting

Standard error:

Return code: 1

Screenshots / relevant information:

HPC System configuration is
NAME=“CentOS Linux”
VERSION=“7 (Core)”
RAM = 200GB
CPUs = 128

Any thoughts about what might have caused it?

Steven · March 27, 2023, 9:06pm

Hi @Shlok_Purani,

I have reformatted your post with the template provided by the Software Support category. For future posts related to software issues, please use category and provide the info in the pre-populated template.

Some other info that may help us debug:

What is fmriprep-adatper?

That’s a lot of resoures you are just running one participant. Try something like 32GB and 16 CPUs.

Does the error persist on the most recent fmriprep version (23.0.1 at this time)?
Is this error subject specific, or does it happen with everyone?
Does the log at /data/workdir/sourcedata/freesurfer/sub-102109/scripts/recon-all.log contain helpful information?

Best,
Steven

Shlok_Purani · March 28, 2023, 12:24pm

Hello @Steven ,

Thanks for your putting the post into the right category.

fmriprep-adapter is just a wrapper which executes the fMRIPrep pipeline
I am using fMRIPrep 21.0.1
crashes at the same point for every subject
/data/workdir/sourcedata/freesurfer/sub-102109/scripts/recon-all.log is very much similar to crash.log
Note: When I am running fMRIPrep without FreeSurfer on same subject it works fine. No errors or warnings. It only crashes with FreeSurfer
Thanks

Steven · March 28, 2023, 12:38pm

Can you try on the latest version,
Making sure to use a fresh working directory and not having any previous Freesurfer outputs available to fmriprep?

If that doesn’t work, can you run recon-all outside of fMRIprep?