Fmriprep node hash + trait errors

alioco · October 22, 2018, 1:57pm

Hi NeuroStars,

We’ve been successfully running fmriprep-1.1.8.simg on our cluster on some multiband and multiecho pilot data without errors, but have suddenly run into the errors below on a multiecho pilot subject (this is a different subject than the one I posted about earlier, just in case emdupre sees this ). The dataset passed BIDS validation and I saw an earlier thread on trait errors that were fixed in earlier releases… Any thoughts/suggestions?

Thank you!

fmriprep output

References

[WARNING] This document format requires a nonempty element.
Please specify either ‘title’ or ‘pagetitle’ in the metadata.
Falling back to ‘CITATION’
/usr/local/miniconda/lib/python3.6/site-packages/bids/grabbids/init.py:6: FutureWarning: grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8
warnings.warn(“grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8”, FutureWarning)
181022-09:41:04,107 nipype.workflow WARNING:
Error while checking node hash, forcing re-run. Although this error may not prevent the workflow from running, it could indicate a major problem. Please report a new issue at https://github.com/nipy/nipype/issues adding the following information:

    Node: fmriprep_wf.single_subject_08ME_wf.summary
    Interface: fmriprep.interfaces.reports.SubjectSummary
    Traceback:

Traceback (most recent call last):

File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/base.py”, line 338, in _local_hash_check
cached, updated = self.procs[jobid].is_cached()

File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py”, line 294, in is_cached
hashed_inputs, hashvalue = self._get_hashval()

File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py”, line 488, in _get_hashval
self._get_inputs()

File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py”, line 515, in _get_inputs
results = loadpkl(results_file)

File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py”, line 668, in loadpkl
raise e

File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py”, line 659, in loadpkl
unpkl = pickle.load(pkl_file)

File “/usr/local/miniconda/lib/python3.6/site-packages/traits/has_traits.py”, line 1440, in setstate
self.trait_set( trait_change_notify = trait_change_notify, **state )

File “/usr/local/miniconda/lib/python3.6/site-packages/traits/has_traits.py”, line 1543, in trait_set
setattr( self, name, value )

File “/usr/local/miniconda/lib/python3.6/site-packages/traits/trait_types.py”, line 1586, in validate
self.error( object, name, value )

File “/usr/local/miniconda/lib/python3.6/site-packages/traits/trait_handlers.py”, line 172, in error
value )

traits.trait_errors.TraitError: The ‘subjects_dir’ trait of a BIDSFreeSurferDirOutputSpec instance must be a directory name, but a value of ‘/beegfs/work/hartley/projects/seqTest/Out_SeqTest_ME08/freesurfer’ <class ‘str’> was specified.

Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py”, line 308, in _send_procs_to_workers
self.procs[jobid].run(updatehash=updatehash)
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py”, line 408, in run
cached, updated = self.is_cached()
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py”, line 294, in is_cached
hashed_inputs, hashvalue = self._get_hashval()
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py”, line 488, in _get_hashval
self._get_inputs()
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py”, line 515, in _get_inputs
results = loadpkl(results_file)
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py”, line 668, in loadpkl
raise e
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py”, line 659, in loadpkl
unpkl = pickle.load(pkl_file)
File “/usr/local/miniconda/lib/python3.6/site-packages/traits/has_traits.py”, line 1440, in setstate
self.trait_set( trait_change_notify = trait_change_notify, **state )
File “/usr/local/miniconda/lib/python3.6/site-packages/traits/has_traits.py”, line 1543, in trait_set
setattr( self, name, value )
File “/usr/local/miniconda/lib/python3.6/site-packages/traits/trait_types.py”, line 1586, in validate
self.error( object, name, value )
File “/usr/local/miniconda/lib/python3.6/site-packages/traits/trait_handlers.py”, line 172, in error
value )
traits.trait_errors.TraitError: The ‘subjects_dir’ trait of a BIDSFreeSurferDirOutputSpec instance must be a directory name, but a value of ‘/beegfs/work/hartley/projects/seqTest/Out_SeqTest_ME08/freesurfer’ <class ‘str’> was specified.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/miniconda/bin/fmriprep”, line 11, in
sys.exit(main())
File “/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/cli/run.py”, line 343, in main
fmriprep_wf.run(**plugin_settings)
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/workflows.py”, line 595, in run
runner.run(execgraph, updatehash=updatehash, config=self.config)
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/base.py”, line 184, in run
self._send_procs_to_workers(updatehash=updatehash, graph=graph)
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py”, line 316, in _send_procs_to_workers
‘traceback’: traceback
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/base.py”, line 224, in _clean_queue
raise RuntimeError("".join(result[‘traceback’]))
RuntimeError: Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py”, line 308, in _send_procs_to_workers
self.procs[jobid].run(updatehash=updatehash)
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py”, line 408, in run
cached, updated = self.is_cached()
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py”, line 294, in is_cached
hashed_inputs, hashvalue = self._get_hashval()
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py”, line 488, in _get_hashval
self._get_inputs()
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py”, line 515, in _get_inputs
results = loadpkl(results_file)
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py”, line 668, in loadpkl
raise e
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py”, line 659, in loadpkl
unpkl = pickle.load(pkl_file)
File “/usr/local/miniconda/lib/python3.6/site-packages/traits/has_traits.py”, line 1440, in setstate
self.trait_set( trait_change_notify = trait_change_notify, **state )
File “/usr/local/miniconda/lib/python3.6/site-packages/traits/has_traits.py”, line 1543, in trait_set
setattr( self, name, value )
File “/usr/local/miniconda/lib/python3.6/site-packages/traits/trait_types.py”, line 1586, in validate
self.error( object, name, value )
File “/usr/local/miniconda/lib/python3.6/site-packages/traits/trait_handlers.py”, line 172, in error
value )
traits.trait_errors.TraitError: The ‘subjects_dir’ trait of a BIDSFreeSurferDirOutputSpec instance must be a directory name, but a value of ‘/beegfs/work/hartley/projects/seqTest/Out_SeqTest_ME08/freesurfer’ <class ‘str’> was specified.

Sentry is attempting to send 1 pending error messages
Waiting up to 10 seconds
Press Ctrl-C to quit

fmriprep call
singularity exec --cleanenv
–bind /beegfs/work/hartley/projects/seqTest/scripts/pyReports/reports.py:/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/viz/reports.py
/beegfs/work/public/singularity/fmriprep-1.1.8.simg fmriprep ${BIDS_Dir} ${Out_Dir} participant
–fs-license-file /beegfs/work/hartley/projects/seqTest/scripts/fsLicense/license.txt
–template-resampling-grid “native”
–participant_label ${SUBJ_ID}
–t2s-coreg

& because I read online this may be related to memory issues, we’ve been requesting 8 cpus per task and 50 GB of memory

effigies · October 22, 2018, 3:02pm

This error text doesn’t make it very clear, but what it means is that /beegfs/work/hartley/projects/seqTest/Out_SeqTest_ME08/freesurfer either cannot be found or is not a directory.

Assuming the path exists and is a directory, then the fMRIPrep instance could not see it when it tried to look for it. This could happen for various reasons, including filesystem errors (e.g., the directory was created in one environment and was looked for in another, before the change propagated) and permissions issues (e.g. if the user fMRIPrep was run as could not run ls /beegfs/work/hartley/projects/seqTest/Out_SeqTest_ME08/freesurfer). This latter also includes quirks of Singularity configurations, in which some directories are not accessible from within a Singularity container.

You should be able to check this with:

$> singularity shell /beegfs/work/public/singularity/fmriprep-1.1.8.simg
$> ls /beegfs/work/hartley/projects/seqTest/Out_SeqTest_ME08/freesurfer

If that works, I would try re-running. If it was a synchronization issue, it may have resolved itself.

alioco · October 22, 2018, 3:57pm

thanks for the speedy reply @effigies ! The directory did not exist, so I created it within the singularity and shell and now it throws out more errors (below). It seems that for some reason the subdirectories and/or the freesurfer outputs aren’t being made… would this be a synchronization issue? And if so, are there any resources anyone may be able to point us to on how to troubleshoot? It’s especially strange that this container seemed to work on Friday, but stopped working over the weekend…

References

[WARNING] This document format requires a nonempty element.
Please specify either ‘title’ or ‘pagetitle’ in the metadata.
Falling back to ‘CITATION’
/usr/local/miniconda/lib/python3.6/site-packages/bids/grabbids/init.py:6: FutureWarning: grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8
warnings.warn(“grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8”, FutureWarning)
181022-11:39:37,708 nipype.workflow WARNING:
Error while checking node hash, forcing re-run. Although this error may not prevent the workflow from running, it could indicate a major problem. Please report a new issue at https://github.com/nipy/nipype/issues adding the following information:

    Node: fmriprep_wf.single_subject_08ME_wf.anat_preproc_wf.surface_recon_wf.fsnative_2_t1_xfm
    Interface: fmriprep.interfaces.freesurfer.PatchedRobustRegister
    Traceback: