AttributeError and warnings during fmriprep for last few subjects

kdestasio · September 5, 2018, 10:28pm

The last ~ 10% of our participants’ fmriprep jobs do not run to completion. The same pipeline was used on all of the project’s data, most of which ran successfully. What strikes us as unusual is that of all the subjects, the failed subjects are the last 5 participant IDs. We would greatly appreciate any insight anyone may have as to why these jobs are failing. Thanks!

Jobs are submitted with SLURM on a Linux HPC. We are using a singularity container for fmriprep: poldracklab_fmriprep_latest-2017-12-07

All error logs for the failed indicate one either an AttributeError or Casting warning.
Here is the first error output from the log file:

/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/fsl/preprocess.py:1597: UserWarning: 
This has not been fully tested. Please report any failures.
  warn('This has not been fully tested. Please report any failures.')
/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/fsl/preprocess.py:1597: UserWarning: 
This has not been fully tested. Please report any failures.
  warn('This has not been fully tested. Please report any failures.')
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/fsl/preprocess.py:1597: UserWarning: 
This has not been fully tested. Please report any failures.
  warn('This has not been fully tested. Please report any failures.')
/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/fsl/preprocess.py:1597: UserWarning: 
This has not been fully tested. Please report any failures.
  warn('This has not been fully tested. Please report any failures.')
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))

Here is the second type of issue encountered, the casting warning:

/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/fsl/preprocess.py:1597: UserWarning: 
This has not been fully tested. Please report any failures.
  warn('This has not been fully tested. Please report any failures.')
/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/fsl/preprocess.py:1597: UserWarning: 
This has not been fully tested. Please report any failures.
  warn('This has not been fully tested. Please report any failures.')
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/fsl/preprocess.py:1597: UserWarning: 
This has not been fully tested. Please report any failures.
  warn('This has not been fully tested. Please report any failures.')
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/fsl/preprocess.py:1597: UserWarning: 
This has not been fully tested. Please report any failures.
  warn('This has not been fully tested. Please report any failures.')
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int
16 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))

ChrisGorgolewski · September 5, 2018, 10:36pm

All I see are warnings so it’s hard to debug. The HTML report for those subjects should have a better description of the errors you encountered. Could you check there and report back or ideally share the whole report (with the figures which are stored in a separate subfolder).

Judging from the file name of your singularity image you might be using a version of FMRIPREP which is 9 months old. I would recommend upgrading.

jguassimoreira · September 5, 2018, 11:08pm

Out of curiosity, have you tried re-running just those subjects on their own? I had a somewhat similar issue come up recently, and it ended up being an issue with how much memory docker wanted vs how much was available (I was running the job locally).

kdestasio · September 5, 2018, 11:08pm

@ChrisGorgolewski, here’s a link to the HTML reports and the images for the relevant subjects: https://drive.google.com/drive/folders/1D6D2r2brukO5FgbwnVYGOLd2pnedFMvn?usp=sharing

I’ll also go ahead and update our version of fmriprep. Thanks for looking at this.

kdestasio · September 5, 2018, 11:09pm

@jguassimoreira, yes, at this point we’ve tried running them individually multiple times. And it’s being run on a high performance cluster that successfully ran the first 100+ jobs, so that shouldn’t be the issue.

jguassimoreira · September 5, 2018, 11:13pm

So strange! Good luck!

ChrisGorgolewski · September 5, 2018, 11:23pm

Ok so one of the errors listed in the reports is

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 404, in run
    self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 508, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 639, in _run_command
    result = self._interface.run()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base/core.py", line 486, in run
    outputs = self.aggregate_outputs(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base/core.py", line 574, in aggregate_outputs
    raise error
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base/core.py", line 568, in aggregate_outputs
    setattr(outputs, key, val)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base/traits_extension.py", line 383, in validate
    value = super(MultiPath, self).validate(object, name, newvalue)
  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/trait_types.py", line 2336, in validate
    return TraitListObject( self, object, name, value )
  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/trait_handlers.py", line 2313, in __init__
    raise excp
  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/trait_handlers.py", line 2305, in __init__
    value = [ validate( object, name, val ) for val in value ]
  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/trait_handlers.py", line 2305, in 
    value = [ validate( object, name, val ) for val in value ]
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base/traits_extension.py", line 109, in validate
    self.info_text, value))
traits.trait_errors.TraitError: The trait 'out_mats' of an IntraModalMergeOutputSpec instance is an existing file name, but the path  '/projects/sanlab/shared/REV/bids_data/derivatives/working/fmriprep_wf/single_subject_REV138_wf/func_preproc_ses_wave2_task_react_acq_3_run_02_wf/phdiff_wf/magmrg/sub-REV138_ses-wave2_run-01_magnitude1_merged_mcf.nii.gz.mat/MAT_0000' does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/multiproc.py", line 62, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 408, in run
    os.remove(hashfile_unfinished)
FileNotFoundError: [Errno 2] No such file or directory: '/projects/sanlab/shared/REV/bids_data/derivatives/working/fmriprep_wf/single_subject_REV138_wf/func_preproc_ses_wave2_task_react_acq_3_run_02_wf/phdiff_wf/magmrg/_0x87642785fac8a81f7ba96925eff0b2d6_unfinished.json'

I would try the following things:

Cleaning the working directory and using the most recent version of FMRIPREP.
Using a working directory that is not a network filesystem - for example /tmp.

kdestasio · September 5, 2018, 11:24pm

Thanks! I’ll try that now.

kdestasio · September 6, 2018, 7:30pm

@ChrisGorgolewski implementing your suggestions worked. We were able to successfully run the participants through fmriprep. Thanks!