Hi NeuroStars!
We recently updated the version of fmriprep on our cluster to fmriprep-1.4.1.sif from fmriprep-1.3.2.sif and are now struggling to get the majority of our subjects through the pipeline.
We’re noticing two main types of errors.
The first seems to be due to failure to retrieve a file in a hidden cache folder in my home directory:
traits.trait_errors.TraitError: The trait ‘reference_image’ of an ApplyTransformsInputSpec instance is an existing file name, but the path ‘/home/ac6976/.cache/templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-01_T1w.nii.gz’ does not exist.
Error setting node input:
Node: tpl_moving
input: reference_image
results_file: /beegfs/work/hartley/projects/MotER/work/fmriprep_wf/single_subject_500_wf/anat_preproc_wf/anat_norm_wf/_template_MNI152NLin2009cAsym/tpl_select/result_tpl_select.pklz
value: /home/ac6976/.cache/templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-01_T1w.nii.gz
190822-21:10:05,296 nipype.workflow ERROR:
Node tpl_seg.a0 failed to run on host c19-01.
190822-21:10:05,307 nipype.workflow ERROR:
Saving crash info to /beegfs/work/hartley/projects/MotER/MotER500/latestderivs/fmriprep/sub-500/log/20190822-143024_5598415f-62ff-4949-8ee5-d7a37f158718/crash-20190822-211005-mmg469-tpl_seg.a0-0580d69e-babb-44c5-b6a7-25140f93a69e.txt
This file exists, the permissions have been changed to try and allow others to be able to access to the file, but the crashes still happen. I don’t believe this type of error occurred in the earlier fmriprep implementation.
The other error, which is not new to us, is the following:
mghRead(/beegfs/work/hartley/projects/MotER/MotER513/latestderivs/freesurfer/sub-513/mri/T2.prenorm.mgz): could not read 262144 bytes at slice 141
using Gaussian smoothing of bias field, sigma=0.500
disabling nonmaximum suppression
retaining points that are at least 1.000mm from the boundary
using segmentation for initial intensity normalization
reading from /beegfs/work/hartley/projects/MotER/MotER513/latestderivs/freesurfer/sub-513/mri/T2.prenorm.mgz…
mri_normalize: could not open source file /beegfs/work/hartley/projects/MotER/MotER513/latestderivs/freesurfer/sub-513/mri/T2.prenorm.mgz
Linux c27-14 3.10.0-514.10.2.el7.x86_64 #1 SMP Fri Mar 3 00:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
recon-all -s sub-513 exited with ERRORS at Thu Aug 22 19:15:35 EDT 2019
For more details, see the log file /beegfs/work/hartley/projects/MotER/MotER513/latestderivs/freesurfer/sub-513/scripts/recon-all.log
To report a problem, see http://surfer.nmr.mgh.harvard.edu/fswiki/BugReporting
Standard error:
Return code: 1
190822-21:11:12,877 nipype.workflow INFO:
[Node] Finished “_autorecon31”.
190822-21:11:16,52 nipype.workflow ERROR:
could not run node: fmriprep_wf.single_subject_513_wf.anat_preproc_wf.surface_recon_wf.autorecon_resume_wf.autorecon3
fMRIPrep failed: Workflow did not execute cleanly. Check log for details
Preprocessing did not finish successfully. Errors occurred while processing data from participants: 513 (1). Check the HTML reports for details.
No errors are written into the recon-all.log and there don’t seem to be issues with the T2.prenorm.mgz file. We’ve previously “solved” this by deleting the output directory and re-running fmriprep, but that no longer seems to do the trick.
Any advice as to how to proceed would be greatly appreciated.
Thanks!