Hello,
Trying to run fmriprep 23.1.4 on one subject, (5 sessions, 4 runs each) on one of our pretty beefy compute nodes. I can’t seem to get passed autorecon2. I have tried rerunning with different resource parameters, deleting my sourcedata/freesurfer, and deleting my /scratch/$USER/fmriprep-work/work-SID001907/fmriprep_23_1_wf/single_subject_SID001907_wf/anat_preproc_wf/ directories. Nothing has helped so far. Here’s the fmriprep command:
singularity run
–cleanenv
-B {MAINDIR}:{MAINDIR}
-B {BIDSDIR},{PREPROCDIR},{SCRATCHDIR}
-B /optnfs/freesurfer:/optnfs/freesurfer {IMAGE}
{BIDSDIR} {OUTDIR} participant
–participant_label {SUBJ}
–ignore slicetiming
–resource-monitor
–bold2t1w-dof 9
–dummy-scans 6
–notrack
–nprocs 32
–omp-nthreads 5
–nthreads 32
–mem_mb 280000
–fs-license-file /optnfs/freesurfer/6.0.0/license.txt
–cifti-output 91k
–output-spaces T1w MNI152NLin2009cAsym fsnative fsaverage6
–skull-strip-fixed-seed
-w ${WORKDIR}
–random-seed 42
–stop-on-first-crash
–force-syn
–use-syn-sdc
–skip_bids_validation
And here’s the error:
230907-17:49:44,71 nipype.workflow WARNING:
[Node] Error on “fmriprep_23_1_wf.single_subject_SID001907_wf.anat_preproc_wf.surface_recon_wf.autorecon_resume_wf.autorecon2_vol” (/scratch/f003z4j/fmriprep-work/work-SID001907/fmriprep_23_1_wf/single_subject_SID001907_wf/anat_preproc_wf/surface_recon_wf/autorecon_resume_wf/autorecon2_vol)
230907-17:49:45,242 nipype.workflow ERROR:
Node autorecon2_vol failed to run on host polaris.dartmouth.edu.
230907-17:49:45,252 nipype.workflow ERROR:
Saving crash info to /dartfs-hpc/rc/lab/C/CANlab/labdata/data/WASABI/derivatives/fmriprep/sub-SID001907/log/20230907-173247_748ac22b-af54-4059-a405-4131d9f54e71/crash-20230907-174945-f003z4j-autorecon2_vol-7473cbf9-4ea8-4ffd-9e75-77f4ca25a551.txt
Traceback (most recent call last):
File “/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py”, line 67, in run_node
result[“result”] = node.run(updatehash=updatehash)
File “/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py”, line 527, in run
result = self._run_interface(execute=True)
File “/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py”, line 645, in _run_interface
return self._run_command(execute)
File “/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py”, line 771, in _run_command
raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node autorecon2_vol.Cmdline:
recon-all -autorecon2-volonly -openmp 5 -subjid sub-SID001907 -sd /dartfs-hpc/rc/lab/C/CANlab/labdata/data/WASABI/derivatives/fmriprep/sourcedata/freesurfer -nogcareg
Stdout:
fs-check-version --s sub-SID001907 --o /tmp/tmp.2gUI7k
Thu Sep 7 17:49:39 EDT 2023setenv SUBJECTS_DIR /dartfs-hpc/rc/lab/C/CANlab/labdata/data/WASABI/derivatives/fmriprep/sourcedata/freesurfer cd /scratch/f003z4j/fmriprep-work/work-SID001907/fmriprep_23_1_wf/single_subject_SID001907_wf/anat_preproc_wf/surface_recon_wf/autorecon_resume_wf/autorecon2_vol /opt/freesurfer/bin/fs-check-version --s sub-SID001907 --o /tmp/tmp.2gUI7k -rwxrwxr-x. 1 f003z4j rc-users 18565 Aug 4 2022 /opt/freesurfer/bin/fs-check-version freesurfer-linux-ubuntu22_x86_64-7.3.2-20220804-6354275 $Id$ Linux polaris.dartmouth.edu 3.10.0-1160.95.1.el7.x86_64 #1 SMP Fri Jun 23 08:44:55 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux pid 29371 Current FS Version freesurfer-linux-ubuntu22_x86_64-7.3.2-20220804-6354275 bstampfile exists /dartfs-hpc/rc/lab/C/CANlab/labdata/data/WASABI/derivatives/fmriprep/sourcedata/freesurfer/sub-SID001907/scripts/build-stamp.txt Subject FS Version: freesurfer-linux-ubuntu22_x86_64-7.3.2-20220804-6354275 No constraints on version because REQ=UnSet and FsVerFile=NotThere #@#% fs-check-version match = 1 fs-check-version Done INFO: SUBJECTS_DIR is /dartfs-hpc/rc/lab/C/CANlab/labdata/data/WASABI/derivatives/fmriprep/sourcedata/freesurfer Actual FREESURFER_HOME /opt/freesurfer -rwxrwx---. 1 f003z4j nogroup 137474 Sep 7 17:26 /dartfs-hpc/rc/lab/C/CANlab/labdata/data/WASABI/derivatives/fmriprep/sourcedata/freesurfer/sub-SID001907/scripts/recon-all.log Linux polaris.dartmouth.edu 3.10.0-1160.95.1.el7.x86_64 #1 SMP Fri Jun 23 08:44:55 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux ERROR: Executable 'mris_curvature -all-info failed! Is it missing from the distribution?'Stderr:
Killed
Traceback:
RuntimeError: subprocess exited with code 1.230907-17:49:45,267 nipype.workflow CRITICAL:
fMRIPrep failed: Traceback (most recent call last):
File “/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py”, line 67, in run_node
result[“result”] = node.run(updatehash=updatehash)
File “/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py”, line 527, in run
result = self._run_interface(execute=True)
File “/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py”, line 645, in _run_interface
return self._run_command(execute)
File “/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py”, line 771, in _run_command
raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node autorecon2_vol.Cmdline:
recon-all -autorecon2-volonly -openmp 5 -subjid sub-SID001907 -sd /dartfs-hpc/rc/lab/C/CANlab/labdata/data/WASABI/derivatives/fmriprep/sourcedata/freesurfer -nogcareg
Stdout:
fs-check-version --s sub-SID001907 --o /tmp/tmp.2gUI7k
Thu Sep 7 17:49:39 EDT 2023setenv SUBJECTS_DIR /dartfs-hpc/rc/lab/C/CANlab/labdata/data/WASABI/derivatives/fmriprep/sourcedata/freesurfer cd /scratch/f003z4j/fmriprep-work/work-SID001907/fmriprep_23_1_wf/single_subject_SID001907_wf/anat_preproc_wf/surface_recon_wf/autorecon_resume_wf/autorecon2_vol /opt/freesurfer/bin/fs-check-version --s sub-SID001907 --o /tmp/tmp.2gUI7k -rwxrwxr-x. 1 f003z4j rc-users 18565 Aug 4 2022 /opt/freesurfer/bin/fs-check-version freesurfer-linux-ubuntu22_x86_64-7.3.2-20220804-6354275 $Id$ Linux polaris.dartmouth.edu 3.10.0-1160.95.1.el7.x86_64 #1 SMP Fri Jun 23 08:44:55 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux pid 29371 Current FS Version freesurfer-linux-ubuntu22_x86_64-7.3.2-20220804-6354275 bstampfile exists /dartfs-hpc/rc/lab/C/CANlab/labdata/data/WASABI/derivatives/fmriprep/sourcedata/freesurfer/sub-SID001907/scripts/build-stamp.txt Subject FS Version: freesurfer-linux-ubuntu22_x86_64-7.3.2-20220804-6354275 No constraints on version because REQ=UnSet and FsVerFile=NotThere #@#% fs-check-version match = 1 fs-check-version Done INFO: SUBJECTS_DIR is /dartfs-hpc/rc/lab/C/CANlab/labdata/data/WASABI/derivatives/fmriprep/sourcedata/freesurfer Actual FREESURFER_HOME /opt/freesurfer -rwxrwx---. 1 f003z4j nogroup 137474 Sep 7 17:26 /dartfs-hpc/rc/lab/C/CANlab/labdata/data/WASABI/derivatives/fmriprep/sourcedata/freesurfer/sub-SID001907/scripts/recon-all.log Linux polaris.dartmouth.edu 3.10.0-1160.95.1.el7.x86_64 #1 SMP Fri Jun 23 08:44:55 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux ERROR: Executable 'mris_curvature -all-info failed! Is it missing from the distribution?'Stderr:
Killed
Traceback:
RuntimeError: subprocess exited with code 1.230907-17:49:49,760 nipype.workflow INFO:
[Node] Finished “fsnative2t1w_xfm”, elapsed time 78.783807s.
230907-17:49:54,84 cli ERROR:
Preprocessing did not finish successfully. Errors occurred while processing data from participants: SID001907 (1). Check the HTML reports for details.