Fmriprep carpet plot error

Hi fmriprepers!

Please could you advise why I can’t seem to produce carpet plots with fmriprep 1.4.1rc4 and including MNI152NLin2009cAsym as an output option? This was run in Jupyter Notebook and sent to a HPC using sbatch - the command was:

sbatch --mincpus=6 --time=24:00:00 --job-name=fmriprep --output /imaging/jj02/CALM/slurm/fmriprep/sub-001.out singularity exec -C -B /imaging/jj02/CALM:/CALM -B /imaging/projects/cbu/calm/CALM_BIDS_New_2:/CALM_BIDS_New_2 -B /home/jj02/.cache/templateflow:/templateflow -B /tmp:/tmp /imaging/local/software/singularity_images/fmriprep/fmriprep-1.4.1rc4.simg fmriprep /CALM_BIDS_New_2 /CALM_BIDS_New_2/derivatives/fmriprep participant --participant_label 001 -v -w /CALM_BIDS_New_2/derivatives/fmriprepwork --skull-strip-template MNIPediatricAsym:cohort-1 --output-spaces MNIPediatricAsym:cohort-1:res-2 T1w fsaverage MNI152NLin6Asym:res-2 MNI152NLin2009cAsym --fs-license-file /CALM/license.txt --use-aroma --return-all-components --write-graph --fd-spike-threshold 0.5 --dvars-spike-threshold 0.5 --notrack --resource-monitor

The error was:

Node: fmriprep_wf.single_subject_001_wf.func_preproc_task_rest_wf.carpetplot_wf.select_std
Working directory: /CALM_BIDS_New_2/derivatives/fmriprepwork/fmriprep_wf/single_subject_001_wf/func_preproc_task_rest_wf/carpetplot_wf/select_std

Node inputs:

key = MNI152NLin2009cAsym
keys = [‘MNIPediatricAsym’, ‘fsaverage’, ‘MNI152NLin6Asym’, ‘MNI152NLin2009cAsym’]
std2anat_xfm = [’/CALM_BIDS_New_2/derivatives/fmriprepwork/fmriprep_wf/single_subject_001_wf/anat_preproc_wf/anat_norm_wf/_template_MNIPediatricAsym/registration/ants_t1_to_mniInverseComposite.h5’, ‘/CALM_BIDS_New_2/derivatives/fmriprepwork/fmriprep_wf/single_subject_001_wf/anat_preproc_wf/anat_norm_wf/_template_MNI152NLin6Asym/registration/ants_t1_to_mniInverseComposite.h5’, ‘/CALM_BIDS_New_2/derivatives/fmriprepwork/fmriprep_wf/single_subject_001_wf/anat_preproc_wf/anat_norm_wf/_template_MNI152NLin2009cAsym/registration/ants_t1_to_mniInverseComposite.h5’]

Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py”, line 316, in _send_procs_to_workers
self.procs[jobid].run(updatehash=updatehash)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 472, in run
result = self._run_interface(execute=True)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 563, in _run_interface
return self._run_command(execute)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 643, in _run_command
result = self._interface.run(cwd=outdir)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py”, line 377, in run
outputs = self.aggregate_outputs(runtime)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py”, line 455, in aggregate_outputs
predicted_outputs = self._list_outputs()
File “/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/utility.py”, line 152, in _list_outputs
for k in self._fields}
File “/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/utility.py”, line 152, in
for k in self._fields}
IndexError: list index out of range

The full log is attached fmriprep1.4.1rc carpet plot error.txt (642.0 KB)

Many thanks,
Joff

I am having the same problem with 1-4-1rc5, except I usually get an OSError: handle is closed at the end of processing.

Are you reusing the working directory? Looks like a problem with old results. If you don’t want to remove all the cached results, would you first try deleting /CALM_BIDS_New_2/derivatives/fmriprepwork/fmriprep_wf/single_subject_001_wf/func_preproc_task_rest_wf/carpetplot_wf?

That said, I would very strongly recommend upgrading to fMRIPrep 1.4.1. The RC4 (release candidate 4) you are using is a pre-release so it is likely buggy.

I’m not re-using my working directory, using a temporary directory each time, and I’ve heard from @monicathieu that this error persists in stable 1-4-1. Results in no carpetplots despite sometimes showing no errors to report! in the .html report. I have occurences of both types (no errors and errors in carpetplot_wf reported)

Similarly, I ran fmriprep without any pre-existing working directories - so the only thing I can think of is if there is a problem when using multiple templates, perhaps if fmriprep is going to the working directory for the results normalised to the child template?

Okay, thanks for the feedback. It seems like we need to have a deeper look as soon as possible.

EDIT: https://github.com/poldracklab/fmriprep/issues/1721

Just to confirm that the error persists in 1.4.1 and here’s the associated output:

Node Name: fmriprep_wf.single_subject_001_wf.func_preproc_task_rest_wf.carpetplot_wf.select_std

fmriprep1.4.1 carpet plot error.txt (630.9 KB)

Thanks for your help

Wondering @oesteban whether this would affect outputs in some way I should be concerned about? I’m not able to tell specifically from the github issues, though I assumed not.

I’m currently running > 1200 subjects through a cluster and would just hit pause on that if I’m going to have to re-run them anyways, and free up my slots for other jobs!

Edit: something else I just noticed. Lots of jobs are ending is OSError: handle is closed as I posted in a different thread, and I’m getting lots of core dumps, which could only be coming from these jobs.fmriprep_cmd2.sh 2.o7515861.txt (635.5 KB)

Hi @oesteban, thanks for looking into this. I was just wondering if the plan is to patch this in 1.5.0 and if you have an estimated release date for this?

Many thanks,
Joff

Hi @JoffJones, yes, that is the only outstanding issue holding up 1.5.0. I can’t give you an estimation, but I don’t expect it to take much longer than one week.

Hi @JoffJones, I just pushed a PR addressing this. Meanwhile, could you test whether reordering your --output-spaces would fix the problem? In particular, can you place any surface-template space to the end of the list?

  --output-spaces MNIPediatricAsym:cohort-1:res-2 T1w MNI152NLin6Asym:res-2 MNI152NLin2009cAsym fsaverage

instead of:

  --output-spaces MNIPediatricAsym:cohort-1:res-2 T1w fsaverage MNI152NLin6Asym:res-2 MNI152NLin2009cAsym
1 Like

Hi @oesteban, I never would have thought to try that but it seems to have fixed the problem! I now have a nice carpet plot in my fmriprep report and no errors to report.

Thanks for looking into this!

1 Like

MANAGED BY INCF