Fmriprep carpet plot error

#1

Hi fmriprepers!

Please could you advise why I can’t seem to produce carpet plots with fmriprep 1.4.1rc4 and including MNI152NLin2009cAsym as an output option? This was run in Jupyter Notebook and sent to a HPC using sbatch - the command was:

sbatch --mincpus=6 --time=24:00:00 --job-name=fmriprep --output /imaging/jj02/CALM/slurm/fmriprep/sub-001.out singularity exec -C -B /imaging/jj02/CALM:/CALM -B /imaging/projects/cbu/calm/CALM_BIDS_New_2:/CALM_BIDS_New_2 -B /home/jj02/.cache/templateflow:/templateflow -B /tmp:/tmp /imaging/local/software/singularity_images/fmriprep/fmriprep-1.4.1rc4.simg fmriprep /CALM_BIDS_New_2 /CALM_BIDS_New_2/derivatives/fmriprep participant --participant_label 001 -v -w /CALM_BIDS_New_2/derivatives/fmriprepwork --skull-strip-template MNIPediatricAsym:cohort-1 --output-spaces MNIPediatricAsym:cohort-1:res-2 T1w fsaverage MNI152NLin6Asym:res-2 MNI152NLin2009cAsym --fs-license-file /CALM/license.txt --use-aroma --return-all-components --write-graph --fd-spike-threshold 0.5 --dvars-spike-threshold 0.5 --notrack --resource-monitor

The error was:

Node: fmriprep_wf.single_subject_001_wf.func_preproc_task_rest_wf.carpetplot_wf.select_std
Working directory: /CALM_BIDS_New_2/derivatives/fmriprepwork/fmriprep_wf/single_subject_001_wf/func_preproc_task_rest_wf/carpetplot_wf/select_std

Node inputs:

key = MNI152NLin2009cAsym
keys = [‘MNIPediatricAsym’, ‘fsaverage’, ‘MNI152NLin6Asym’, ‘MNI152NLin2009cAsym’]
std2anat_xfm = [’/CALM_BIDS_New_2/derivatives/fmriprepwork/fmriprep_wf/single_subject_001_wf/anat_preproc_wf/anat_norm_wf/_template_MNIPediatricAsym/registration/ants_t1_to_mniInverseComposite.h5’, ‘/CALM_BIDS_New_2/derivatives/fmriprepwork/fmriprep_wf/single_subject_001_wf/anat_preproc_wf/anat_norm_wf/_template_MNI152NLin6Asym/registration/ants_t1_to_mniInverseComposite.h5’, ‘/CALM_BIDS_New_2/derivatives/fmriprepwork/fmriprep_wf/single_subject_001_wf/anat_preproc_wf/anat_norm_wf/_template_MNI152NLin2009cAsym/registration/ants_t1_to_mniInverseComposite.h5’]

Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py”, line 316, in _send_procs_to_workers
self.procs[jobid].run(updatehash=updatehash)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 472, in run
result = self._run_interface(execute=True)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 563, in _run_interface
return self._run_command(execute)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 643, in _run_command
result = self._interface.run(cwd=outdir)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py”, line 377, in run
outputs = self.aggregate_outputs(runtime)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py”, line 455, in aggregate_outputs
predicted_outputs = self._list_outputs()
File “/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/utility.py”, line 152, in _list_outputs
for k in self._fields}
File “/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/utility.py”, line 152, in
for k in self._fields}
IndexError: list index out of range

The full log is attached fmriprep1.4.1rc carpet plot error.txt (642.0 KB)

Many thanks,
Joff

#2

I am having the same problem with 1-4-1rc5, except I usually get an OSError: handle is closed at the end of processing.

#3

Are you reusing the working directory? Looks like a problem with old results. If you don’t want to remove all the cached results, would you first try deleting /CALM_BIDS_New_2/derivatives/fmriprepwork/fmriprep_wf/single_subject_001_wf/func_preproc_task_rest_wf/carpetplot_wf?

That said, I would very strongly recommend upgrading to fMRIPrep 1.4.1. The RC4 (release candidate 4) you are using is a pre-release so it is likely buggy.

#4

I’m not re-using my working directory, using a temporary directory each time, and I’ve heard from @monicathieu that this error persists in stable 1-4-1. Results in no carpetplots despite sometimes showing no errors to report! in the .html report. I have occurences of both types (no errors and errors in carpetplot_wf reported)

#5

Similarly, I ran fmriprep without any pre-existing working directories - so the only thing I can think of is if there is a problem when using multiple templates, perhaps if fmriprep is going to the working directory for the results normalised to the child template?

#6

Okay, thanks for the feedback. It seems like we need to have a deeper look as soon as possible.

EDIT: https://github.com/poldracklab/fmriprep/issues/1721

#7

Just to confirm that the error persists in 1.4.1 and here’s the associated output:

Node Name: fmriprep_wf.single_subject_001_wf.func_preproc_task_rest_wf.carpetplot_wf.select_std

fmriprep1.4.1 carpet plot error.txt (630.9 KB)

Thanks for your help

#8

Wondering @oesteban whether this would affect outputs in some way I should be concerned about? I’m not able to tell specifically from the github issues, though I assumed not.

I’m currently running > 1200 subjects through a cluster and would just hit pause on that if I’m going to have to re-run them anyways, and free up my slots for other jobs!

Edit: something else I just noticed. Lots of jobs are ending is OSError: handle is closed as I posted in a different thread, and I’m getting lots of core dumps, which could only be coming from these jobs.fmriprep_cmd2.sh 2.o7515861.txt (635.5 KB)

#9

Hi @oesteban, thanks for looking into this. I was just wondering if the plan is to patch this in 1.5.0 and if you have an estimated release date for this?

Many thanks,
Joff