I’m getting a recon-all exception in 7/20 participants.
Running on a singularity image (version 20.2.6) on our HPC, separate job for each participant.
error below:
211111-21:53:44,798 nipype.workflow WARNING:
[Node] Error on "_autorecon_surfs1" (/scratch/fmriprep/8006/fmriprep_wf/single_subject_8006_wf/anat_preproc_wf/surface_recon_wf/autorecon_resume_wf/autorecon_surfs/mapflow/_autorecon_surfs1)
211111-21:53:45,554 nipype.workflow ERROR:
Node _autorecon_surfs1 failed to run on host node1328.oscar.ccv.brown.edu.
211111-21:53:45,568 nipype.workflow ERROR:
Saving crash info to /data/bids/derivatives/fmriprep-20.2.6/fmriprep/sub-8006/log/20211111-202450_506b0092-e097-4b28-814b-0ed154e67b8e/crash-20211111-215345-hritz-_autorecon_surfs1-7397157e-8795-4036-ac02-255011767b99.txt
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
result["result"] = node.run(updatehash=updatehash)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 521, in run
result = self._run_interface(execute=True)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 639, in _run_interface
return self._run_command(execute)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 751, in _run_command
f"Exception raised while executing Node {self.name}.\n\n{result.runtime.traceback}"
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node _autorecon_surfs1.
RuntimeError: subprocess exited with code 1.
211111-21:53:45,586 nipype.workflow CRITICAL:
fMRIPrep failed: Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
result["result"] = node.run(updatehash=updatehash)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 521, in run
result = self._run_interface(execute=True)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 639, in _run_interface
return self._run_command(execute)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 751, in _run_command
f"Exception raised while executing Node {self.name}.\n\n{result.runtime.traceback}"
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node _autorecon_surfs1.
RuntimeError: subprocess exited with code 1.
211111-21:53:45,686 nipype.workflow INFO:
[Node] Finished "_autorecon_surfs0", elapsed time 1.698105s.
211111-21:53:45,688 nipype.workflow WARNING:
[Node] Error on "_autorecon_surfs0" (/scratch/fmriprep/8006/fmriprep_wf/single_subject_8006_wf/anat_preproc_wf/surface_recon_wf/autorecon_resume_wf/autorecon_surfs/mapflow/_autorecon_surfs0)
211111-21:54:01,204 cli ERROR:
Preprocessing did not finish successfully. Errors occurred while processing data from participants: 8006 (1). Check the HTML reports for details.
Issue hasn’t been resolved (difficult to diagnose from error message), would really appreciate any guidance. @effigies?
fmriprep call:
singularity run --cleanenv \
--bind ${bids_root_dir}/${investigator}/study-${study_label}:/data \
--bind /gpfs/scratch/hritz:/scratch \
--bind /gpfs/data/bnc/licenses:/licenses \
/gpfs/data/bnc/simgs/nipreps/fmriprep-${fmriprep_version}.sif \
/data/bids \
/data/bids/derivatives/fmriprep-${fmriprep_version} \
participant \
--participant-label ${SUBJ_LABEL} \
--fs-license-file /licenses/freesurfer-license.txt \
-w /scratch/fmriprep/${SUBJ_LABEL} \
--output-spaces MNI152NLin6Asym:res-2 \
--nthreads ${nThreads} \
--write-graph \
--dummy-scans ${nDummy} \
--stop-on-first-crash
What were the contents of /data/bids/derivatives/fmriprep-20.2.6/fmriprep/sub-8006/log/20211111-202450_506b0092-e097-4b28-814b-0ed154e67b8e/crash-20211111-215345-hritz-_autorecon_surfs1-7397157e-8795-4036-ac02-255011767b99.txt?
Well that sure is uninformative. I would look in scripts/recon-all-rh.log in your FreeSurfer directory. If that doesn’t give any indication of what went wrong, I would just make sure there aren’t any IsRunning files in scripts/ and try again.
ya didn’t make much of the crash log either. weird thing is that other participants work (including subsequent runs). I’ll try your recc, would love to take advantage of ANTs. If this doesn’t work, I’ll post a bug to github
If you got this far, you’ve already used the ANTs skull-strip. You should be able to resume on your own with recon-all -autorecon2 -autorecon3 -sd $SUBJECT_DIR -s sub-8006. That will at least give you a better error message.
1 Like
There is a little more info here, esp ‘Probably trying to use a scalar data file as a surface’.
The same error came up a couple of years ago in nipype, but looks resolved: https://groups.google.com/g/cmtk-users/c/AbUu-ChAueY
error:
mris_ca_label: could not parse 96348th line ‘102498 47.746 18.541 -8.365’ in label file
ERROR:
MRISreadVertexPositions(…/surf/lh.sphere.reg): surfaces differ. Main: 139204 verts 278404 faces, …/surf/lh.sphere.reg: 0 verts 0 faces
Surfaces may be out-of-date
rh log file
using smoothwm curvature for final alignment
cwd /data/bids/derivatives/fmriprep-20.2.6/freesurfer/sub-8004/scripts
cmdline mris_register -curv -rusage /data/bids/derivatives/fmriprep-20.2.6/freesurfer/sub-8004/touch/rusage.mris_register.rh.dat …/surf/rh.sphere /opt/freesurfer/average/rh.folding.atlas.acfb40.noaparc.i12.2016-08-02.tif …/surf/rh.sphere.reg
0 inflated.H
1 sulc
2 smoothwm (computed)
Id: mris_register.c,v 1.63 2016/01/20 23:43:04 greve Exp
Id: mrisurf.c,v 1.781.2.6 2016/12/27 16:47:14 zkaufman Exp
reading surface from …/surf/rh.sphere…
mrisFindNeighbors: …/surf/rh.sphere: face[224074].v[0] = 112919, but face 224074 not in vertex 112919 face list
lh log file
using smoothwm curvature for final alignment
cwd /data/bids/derivatives/fmriprep-20.2.6/freesurfer/sub-8004/scripts
cmdline mris_register -curv -rusage /data/bids/derivatives/fmriprep-20.2.6/freesurfer/sub-8004/touch/rusage.mris_register.lh.dat …/surf/lh.sphere /opt/freesurfer/average/lh.folding.atlas.acfb40.noaparc.i12.2016-08-02.tif …/surf/lh.sphere.reg
0 inflated.H
1 sulc
2 smoothwm (computed)
Id: mris_register.c,v 1.63 2016/01/20 23:43:04 greve Exp
Id: mrisurf.c,v 1.781.2.6 2016/12/27 16:47:14 zkaufman Exp
reading surface from …/surf/lh.sphere…
nquads=1572865, nvertices=438
ERROR: MRISread: file ‘…/surf/lh.sphere’ has many more faces than vertices!
Probably trying to use a scalar data file as a surface!
error.txt (273 Bytes) recon-all-rh.txt (165.5 KB)
recon-all-lh.txt (166.3 KB)
though re-running hadn’t been helping, deleted the deriv/20.2.6/freesurfer folder, and this time it worked fine.
Thanks for your help @effigies !