Fmriprep 1.5.0 consistent error in anat_preproc_wf.anat_norm_wf.registration

1

Hello! I have been having some trouble with using 1.5.0 in a singularity container. Subjects that ran through fine in 1.4.0 now all consistently fail at anat_preproc_wf.anat_norm_wf.registration. The log file looks like this:

    Node: fmriprep_wf.single_subject_1026_wf.anat_preproc_wf.anat_norm_wf.registration
Working directory: /pine/scr/t/r/trhenry/working/fmriprep_wf/single_subject_1026_wf/anat_preproc_wf/anat_norm_wf/_template_MNI152NLin2009cAsym/registration

Node inputs:

explicit_masking = True
flavor = precise
float = True
initial_moving_transform = <undefined>
lesion_mask = <undefined>
moving = T1w
moving_image = /pine/scr/t/r/trhenry/working/fmriprep_wf/single_subject_1026_wf/anat_preproc_wf/anat_norm_wf/_template_MNI152NLin2009cAsym/trunc_mov/sub-1026_ses-01_T1w_template_corrected_xform_maths.nii.gz
moving_mask = /pine/scr/t/r/trhenry/working/fmriprep_wf/single_subject_1026_wf/anat_preproc_wf/surface_recon_wf/refine/sub-1026_ses-01_T1w_template_corrected_xform_rbrainmask.nii.gz
num_threads = 7
orientation = RAS
reference = T1w
reference_image = <undefined>
reference_mask = <undefined>
settings = <undefined>
template = MNI152NLin2009cAsym
template_resolution = <undefined>
template_spec = {}

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 473, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 564, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 649, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 376, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/mni.py", line 127, in _run_interface
    ants_args = self._get_ants_args()
  File "/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/mni.py", line 378, in _get_ants_args
    "fixed_masked.nii.gz")
  File "/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/mni.py", line 422, in mask
    data = in_nii.get_data()
  File "/usr/local/miniconda/lib/python3.7/site-packages/nibabel/dataobj_images.py", line 203, in get_data
    data = np.asanyarray(self._dataobj)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/core/numeric.py", line 553, in asanyarray
    return array(a, dtype, copy=False, order=order, subok=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nibabel/arrayproxy.py", line 355, in __array__
    raw_data = self.get_unscaled()
  File "/usr/local/miniconda/lib/python3.7/site-packages/nibabel/arrayproxy.py", line 350, in get_unscaled
    mmap=self._mmap)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nibabel/volumeutils.py", line 535, in array_from_file
    getattr(infile, 'name', 'object')))
OSError: Expected 17060042 bytes, got 6607696 bytes from object
 - could the file be damaged?

Any ideas?

Thanks!
Teague

2

Hi @trhenry,

I suspect you were running several subjects at the same time. fMRIPrep relies on a library called TemplateFlow to allow the flexibility of changing templates.

I think you would need to clean up the TemplateFlow directory on the host, prefetch the templates you’ll need before execution and then retry running fMRIPrep.

Please have a look into this section of the documentation - https://fmriprep.readthedocs.io/en/latest/singularity.html

Let us know if we are missing important details that deviate in your settings from what is explained there.

3

Oh I see!
Yes, we are submitting a large number of subject level jobs using SLURM.

This is great, I will look into this. It appears as though I need to set the bind point for the template flow directory. I take it that does not utilize the specified working directory?

Thanks!
Teague

4

It doesn’t by default. You can point it to the working directory, but still you need to solve the initial burst of processes trying to fetch templates for first (or have each process download the templates in separate folders, which is not very efficient).

5

I see, that makes sense. Thanks for this! I’ll post any solution I find!

6

The solution in the link @oesteban posted above for prefetching the templates worked quite well. We prefetch based on project needs once during project setup, which then allows fmriprep to be run once the data is in BIDS format.