Fmriprep 1.5.0 semi-inconsistent errors

,
1

Hello - I have run one subject through fmriprep 1.5.0 multiple times, all of which have failed. What is odd is that the errors I’ve received are inconsistent to some extent. That is, the specific types or classes of errors are consistent but the runs in which the errors appear are (somewhat) inconsistent. There are many errors, not just one or two.

There are two types of errors that I receive:

  • FileNotFoundError: [Errno 2] No such file or directory: 'result_merger.pklz' or result_transforms.pklz
  • AttributeError: 'NoneType' object has no attribute 'outputs'

Errors for one round of fmriprepping:

Errors
Node Name: fmriprep_wf.single_subject_M16_wf.func_preproc_ses_wordclouds01_task_wordclouds_rec_prospective_run_01_wf.bold_confounds_wf.add_motion_headers
Node Name: fmriprep_wf.single_subject_M16_wf.func_preproc_ses_sentences01_task_sentences_rec_prospective_run_03_wf.func_derivatives_wf.ds_bold_native_ref
Node Name: fmriprep_wf.single_subject_M16_wf.func_preproc_ses_wordclouds01_task_wordclouds_rec_prospective_run_10_wf.bold_reg_wf.bbreg_wf.lta_ras2ras
Node Name: fmriprep_wf.single_subject_M16_wf.func_preproc_ses_wordclouds01_task_wordclouds_rec_prospective_run_03_wf.bold_t1_trans_wf.merge_xforms
Node Name: fmriprep_wf.single_subject_M16_wf.func_preproc_ses_pictures01_task_pictures_rec_prospective_run_05_wf.bold_reg_wf.bbreg_wf.lta_ras2ras
Node Name: fmriprep_wf.single_subject_M16_wf.func_preproc_ses_sentences01_task_sentences_rec_prospective_run_05_wf.bold_t1_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.check_hdr
Node Name: fmriprep_wf.single_subject_M16_wf.func_preproc_ses_sentences01_task_sentences_rec_prospective_run_06_wf.func_derivatives_wf.ds_bold_t1_ref
Node Name: fmriprep_wf.single_subject_M16_wf.func_preproc_ses_sentences01_task_sentences_rec_prospective_run_04_wf.bold_t1_trans_wf.bold_reference_wf.calc_dummy_scans
Node Name: fmriprep_wf.single_subject_M16_wf.func_preproc_ses_pictures01_task_pictures_rec_prospective_run_04_wf.bold_t1_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.check_hdr

Errors for another round of fmriprepping:

Errors
Node Name: fmriprep_wf.single_subject_M16_wf.func_preproc_ses_wordclouds01_task_wordclouds_rec_prospective_run_10_wf.bold_reg_wf.bbreg_wf.lta_ras2ras
Node Name: fmriprep_wf.single_subject_M16_wf.func_preproc_ses_wordclouds01_task_wordclouds_rec_prospective_run_06_wf.bold_reg_wf.bbreg_wf.lta_ras2ras
Node Name: fmriprep_wf.single_subject_M16_wf.func_preproc_ses_wordclouds01_task_wordclouds_rec_prospective_run_02_wf.func_derivatives_wf.ds_bold_aseg_t1
Node Name: fmriprep_wf.single_subject_M16_wf.func_preproc_ses_pictures01_task_pictures_rec_prospective_run_03_wf.bold_reg_wf.bbreg_wf.lta_ras2ras
Node Name: fmriprep_wf.single_subject_M16_wf.func_preproc_ses_wordclouds01_task_wordclouds_rec_prospective_run_01_wf.bold_t1_trans_wf.bold_reference_wf.calc_dummy_scans
Node Name: fmriprep_wf.single_subject_M16_wf.func_preproc_ses_sentences01_task_sentences_rec_prospective_run_03_wf.bold_surf_wf.update_metadata
Node Name: fmriprep_wf.single_subject_M16_wf.func_preproc_ses_pictures01_task_pictures_rec_prospective_run_04_wf.bold_surf_wf.update_metadata
Node Name: fmriprep_wf.single_subject_M16_wf.func_preproc_ses_wordclouds01_task_wordclouds_rec_prospective_run_09_wf.func_derivatives_wf.ds_bold_aparc_std

Here are the crash logs of two sample errors:

Node: fmriprep_wf.single_subject_M16_wf.func_preproc_ses_wordclouds01_task_wordclouds_rec_prospective_run_10_wf.bold_reg_wf.bbreg_wf.lta_ras2ras
Working directory: /scratch/dmoracze/KRNS/wrk/fmriprep_wf/single_subject_M16_wf/func_preproc_ses_wordclouds01_task_wordclouds_rec_prospective_run_10_wf/bold_reg_wf/bbreg_wf/lta_ras2ras

Node inputs:

args = <undefined>
environ = {}
in_fsl = <undefined>
in_itk = <undefined>
in_lta = <undefined>
in_mni = <undefined>
in_niftyreg = <undefined>
in_reg = <undefined>
invert = <undefined>
ltavox2vox = <undefined>
out_fsl = <undefined>
out_itk = <undefined>
out_lta = True
out_mni = <undefined>
out_reg = <undefined>
source_file = <undefined>
target_conform = <undefined>
target_file = <undefined>

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 269, in _send_procs_to_workers
    num_subnodes = self.procs[jobid].num_subnodes()
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 1204, in num_subnodes
    self._get_inputs()
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 1219, in _get_inputs
    super(MapNode, self)._get_inputs()
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 521, in _get_inputs
    outputs = _load_resultfile(results_file)[0].outputs
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/utils.py", line 300, in load_resultfile
    result = loadpkl(results_file)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/utils/filemanip.py", line 687, in loadpkl
    pkl_file = pklopen(infile.name, 'rb')
  File "/usr/local/miniconda/lib/python3.7/gzip.py", line 53, in open
    binary_file = GzipFile(filename, gz_mode, compresslevel)
  File "/usr/local/miniconda/lib/python3.7/gzip.py", line 163, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
FileNotFoundError: [Errno 2] No such file or directory: 'result_transforms.pklz'

Node: fmriprep_wf.single_subject_M16_wf.func_preproc_ses_wordclouds01_task_wordclouds_rec_prospective_run_02_wf.func_derivatives_wf.ds_bold_aseg_t1
Working directory: /scratch/dmoracze/KRNS/wrk/fmriprep_wf/single_subject_M16_wf/func_preproc_ses_wordclouds01_task_wordclouds_rec_prospective_run_02_wf/func_derivatives_wf/ds_bold_aseg_t1

Node inputs:

base_directory = /scratch/dmoracze/KRNS/out
check_hdr = True
compress = <undefined>
desc = aseg
extra_values = <undefined>
in_file = <undefined>
keep_dtype = False
meta_dict = <undefined>
source_file = /scratch/dmoracze/KRNS/rawdata/sub-M16/ses-wordclouds01/func/sub-M16_ses-wordclouds01_task-wordclouds_rec-prospective_run-02_bold.nii.gz
space = T1w
suffix = dseg

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 316, in _send_procs_to_workers
    self.procs[jobid].run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 411, in run
    cached, updated = self.is_cached()
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 297, in is_cached
    hashed_inputs, hashvalue = self._get_hashval()
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 494, in _get_hashval
    self._get_inputs()
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 521, in _get_inputs
    outputs = _load_resultfile(results_file)[0].outputs
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/utils.py", line 311, in load_resultfile
    if resolve and result.outputs:
AttributeError: 'NoneType' object has no attribute 'outputs'

There are many other errors such as this, but they fall into the same classes as the above. Please let me know if you’d like to see the other errors. I have also uploaded the stderr, which is pretty extensive:

KRNS_37795518_0.e.txt (61.7 KB)

Here are the environment preparation and fmriprep commands I am using. The evironment prep and fmriprep commands are given to slurm as one line, I just made it easier to parse by adding returns here.

export TMPDIR=/scratch/dmoracze/data &&
export SINGULARITY_BINDPATH=/gs3,/gs4,/gs5,/gs6,/gs7,/gs8,/gs9,/gs10,/gs11,/spin1,/scratch,/fdb,/data,/lscratch &&    
export SINGULARITYENV_TEMPLATEFLOW_HOME=/home/fmriprep/.cache/templateflow &&     
mkdir -p $TMPDIR/out &&
mkdir -p $TMPDIR/wrk &&     
singularity run --cleanenv
-B /data/DSST/containers/TemplateFlow:/home/fmriprep/.cache/templateflow
-B /home/dmoracze/.cache/fmriprep:/home/fmriprep/.cache/fmriprep
/data/DSST/containers/fmriprep_1.5.0_DN.simg
/scratch/dmoracze/data/rawdata $TMPDIR/out
participant --participant_label sub-M16
-w $TMPDIR/wrk
--nthreads $SLURM_CPUS_PER_TASK 
--mem_mb $SLURM_MEM_PER_NODE
--fs-license-file /home/dmoracze/license.txt 
--output-spaces T1w func fsaverage6 MNI152NLin6Asym
--longitudinal --use-aroma --error-on-aroma-warnings
2

Hi @dmoracze –

Thanks so much for providing such detailed information !

It looks like some of these (such as FileNotFoundError: [Errno 2] No such file or directory: 'result_transforms.pklz') are due to re-using an existing working directory that previously failed. Could you please try to clear your working directory and output directories, re-run fMRIPrep, and report the error ?

Elizabeth

3

Hello @emdupre, thanks for your response. I have cleared $TMPDIR/wrk and $TMPDIR/out multiple times and still get similar errors. I tried once more after your comment and received the same result.

4

Yes, this looks like a problem we’re working on resolving currently. I apologize for the inconvenience and will alert you when a new version is out.

5

Thanks.

You may already be privy to this but I think the issue might be due to my desire to have --output-spaces T1w. Testing now without that option.

6

How this turned out, @dmoracze? Did removing the T1w from --output-spaces work?

I would imagine your /scratch dir is backed by a networking filesystem. Could you try using some local storage for your work directory (instead of $TMPDIR/wrk)?

Finally (and probably unrelated), why --longitudinal?

7

Thanks for checking up on this, @oesteban.

Did removing the T1w from --output-spaces work?

No this did not work. Though I have since cleared out the output and working directories so that I can try something new. I have attached the stdout and the stderr files from an example job.

I am now attempting to run this through 1.4.1, though, from what I understand, I can’t include --output-spaces T1w because the templates are not include in that version.

I would imagine your /scratch dir is backed by a networking filesystem. Could you try using some local storage for your work directory (instead of $TMPDIR/wrk )?

I have tried to use /scratch and the node’s temporary storage /lscratch for the work directory with the same results. I have not tried local storage, though I could try for one subject with only a couple runs. We have lots and lots of data on each subject, which makes the work directory massive (which is why I wanted to use the node’s temporary storage to distribute the storage when running in parallel).

Finally (and probably unrelated), why --longitudinal ?

For this study, each subject has many sessions and runs within sessions, so I used --longitudinal.

Let me know if I can provide any more information.
KRNS_38121342_1.e.txt (66.2 KB)

8

Yes, they are. But I would keep working on 1.5 as you might want to fix this for good and keep using fMRIPrep in the future.

Something is wrong in your settings. Have you checked your attachment?

Preprocessing did not finish successfully. Errors occurred while processing data from participants: M16 (7). Check the HTML reports for details.
/spin1/swarm/dmoracze/mpN6Vnos9V/cmd.1: line 1:  STATUS=1: command not found
chown: cannot access ‘/scratch/dmoracze/derivatives’: No such file or directory

This very much looks like you did not successfully find a stable configuration for your system. Could you post a link to the documentation of your HPC?

As discussed previously (Longitudinal FMRIPrep) the choice of --longitudinal will define how your T1w will be averaged. Unless you are expecting large anatomical changes, I think you are better off without it.

9

Something is wrong in your settings. Have you checked your attachment?

For this specific command I used:

---- COMMAND EXECUTED: ---------------------------------------------------------
(   export TMPDIR=/lscratch/$SLURM_JOB_ID &&     export SINGULARITY_BINDPATH=/gs3,/gs4,/gs5,/gs6,/gs7,/gs8,/gs9,/gs10,/gs11,/spin1,/scratch,/fdb,/data,/lscratch &&    export SINGULARITYENV_TEMPLATEFLOW_HOME=/home/fmriprep/.cache/templateflow &&     mkdir -p $TMPDIR/out &&     mkdir -p $TMPDIR/wrk &&     singularity run --cleanenv     -B /data/DSST/containers/TemplateFlow:/home/fmriprep/.cache/templateflow     -B /home/dmoracze/.cache/fmriprep:/home/fmriprep/.cache/fmriprep     /data/DSST/containers/fmriprep-1.5.0.simg /data/dmoracze/rawdata $TMPDIR/out     participant     --participant_label sub-M16     -w $TMPDIR/wrk     --nthreads $SLURM_CPUS_PER_TASK     --mem_mb $SLURM_MEM_PER_NODE     --fs-license-file /home/dmoracze/license.txt     --output-spaces func fsaverage6 MNI152NLin6Asym --longitudinal --use-aroma --error-on-aroma-warnings ; \ STATUS=$? rsync -ach $TMPDIR/out /scratch/dmoracze/derivatives ; chown -R :MLDSST /scratch/dmoracze/derivatives ;  (exit $STATUS) )

Which uses the node’s temporary scratch (/lscratch), then once the processing is finished it copies the output to /scratch. I’ll check my settings to see what’s going on, but either way the preprocessing still has errors.

This very much looks like you did not successfully find a stable configuration for your system. Could you post a link to the documentation of your HPC?

https://hpc.nih.gov/storage/

As discussed previously (Longitudinal FMRIPrep) the choice of --longitudinal will define how your T1w will be averaged. Unless you are expecting large anatomical changes, I think you are better off without it.

That’s fair. I’d still like the option to make this work if possible.

10

Modified version of your script:

export SINGULARITY_BINDPATH=/gs3,/gs4,/gs5,/gs6,/gs7,/gs8,/gs9,/gs10,/gs11,/spin1,/scratch,/fdb,/data,/lscratch
export SINGULARITYENV_TEMPLATEFLOW_HOME=/home/fmriprep/.cache/templateflow

singularity run --cleanenv -B /home/dmoracze/.cache:/home/fmriprep/.cache /data/DSST/containers/fmriprep-1.5.0.simg /data/dmoracze/rawdata /data/dmoracze/rawdata/derivatives/fmriprep-1.5.0 participant --participant-label M16 -w /lscratch/wrk/sub-M16 --nthreads 12 --omp-nthreads 8 --mem_mb 30 --fs-license-file /home/dmoracze/license.txt --output-spaces func fsaverage6 MNI152NLin6Asym --use-aroma --error-on-aroma-warnings
  • No need to add the final rsync, fMRIPrep can write out to the output folder directly. This will get you more feedback when things go wrong.
  • Bind .cache to simplify.
  • Avoid variables that might not be replaced in runtime (those hard-coded --nthreads --omp-nthreads and --mem-gb are conservative but generally work well for me.
1 Like
11

Very helpful! I’ll try this.

12

I tried this and I’m still getting errors, though they are more consistent now. Of my 19 subjects, only two have finished but both have similar errors.

All of the errors are FileNotFoundError: [Errno 2] No such file or directory: 'result_name_surfs.pklz'. This error does not occur for every run, but a (seemingly) random smattering of them.

Here is a full error sample:

Node Name: fmriprep_wf.single_subject_M16_wf.func_preproc_ses_wordclouds01_task_wordclouds_rec_prospective_run_06_wf.func_derivatives_wf.ds_bold_surfs
File: /gs10/users/DSST/KRNS/bids/derivatives/fmriprep-1.5.0/fmriprep/sub-M16/log/20191016-051653_ee5b1477-a939-4423-b09b-d028b41ae3e6/crash-20191016-181604-dmoracze-ds_bold_surfs-77c98459-d71a-4460-8706-d6ebda7e6810.txt
Working Directory: /lscratch/39376864/wrk/sub-M16/fmriprep_wf/single_subject_M16_wf/func_preproc_ses_wordclouds01_task_wordclouds_rec_prospective_run_06_wf/func_derivatives_wf/ds_bold_surfs
Inputs:
base_directory: /gs10/users/DSST/KRNS/bids/derivatives/fmriprep-1.5.0
check_hdr: True
compress:
extra_values:
in_file:
keep_dtype: False
meta_dict:
source_file:
suffix: ['']
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 269, in _send_procs_to_workers
    num_subnodes = self.procs[jobid].num_subnodes()
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 1204, in num_subnodes
    self._get_inputs()
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 1219, in _get_inputs
    super(MapNode, self)._get_inputs()
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 521, in _get_inputs
    outputs = _load_resultfile(results_file)[0].outputs
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/utils.py", line 300, in load_resultfile
    result = loadpkl(results_file)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/utils/filemanip.py", line 687, in loadpkl
    pkl_file = pklopen(infile.name, 'rb')
  File "/usr/local/miniconda/lib/python3.7/gzip.py", line 53, in open
    binary_file = GzipFile(filename, gz_mode, compresslevel)
  File "/usr/local/miniconda/lib/python3.7/gzip.py", line 163, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
FileNotFoundError: [Errno 2] No such file or directory: 'result_name_surfs.pklz'

Here is the fmriprep command I used, basically what @oesteban suggested with a couple of changes (I have noted why each change in the comments)

# I changed the TMPDIR as per my HPC's documentation. My subjects have lots of runs and I don't have the personal storage for the working files nor do I want to tie down the shared scratch 
export TMPDIR=/lscratch/$SLURM_JOB_ID &&

export SINGULARITY_BINDPATH=/gs3,/gs4,/gs5,/gs6,/gs7,/gs8,/gs9,/gs10,/gs11,/spin1,/scratch,/fdb,/data,/lscratch &&    
export SINGULARITYENV_TEMPLATEFLOW_HOME=/home/fmriprep/.cache/templateflow &&     
mkdir -p $TMPDIR/wrk/sub-P03 &&     

singularity run --cleanenv
-B /home/dmoracze/.cache:/home/fmriprep/.cache
/data/DSST/containers/fmriprep-1.5.0.simg /data/dmoracze/rawdata
/data/DSST/KRNS/bids/derivatives/fmriprep-1.5.0
participant --participant_label sub-P03
-w /lscratch/$SLURM_JOB_ID/wrk/sub-P03
--nthreads 12 --omp-nthreads 8 --mem_mb 30
--fs-license-file /home/dmoracze/license.txt
--output-spaces func fsaverage6 MNI152NLin6Asym 
--use-aroma --error-on-aroma-warnings

I removed --longitudinal and, in a perfect world, I would like to eventually include --output-spaces T1w, but I’m trying to get this working first.

13

Hi @dmoracze, this indeed looks like great progress. Now you are facing a struggle that has become pretty common with fMRIPrep 1.5.0 (you can follow up here https://github.com/poldracklab/fmriprep/issues/1819).

Could you please try the very recently released fMRIPrep 1.5.1rc1? This release candidate tries to alleviate this particular issue. I can’t promise it will fix it, but I’m positive it will maintain these errors under more stringent control.

Please let us know whether that worked for you.

14

Thanks, I’m happy to know it isn’t just me!

Jobs submitted, I’ll update later.

15

I am happy to report that most of my subjects are finished processing through 1.5.1rc1 with no errors to report.

Thanks for all your help!

2 Likes