fMRIprep Aroma error

lvbrussel · November 8, 2021, 10:20am

Hello,

I’m trying to run fmriprep with --use-aroma, but run into the issue below:

Node: fmriprep_wf.single_subject_02_wf.func_preproc_task_nnip_wf.ica_aroma_wf.ica_aroma
Working directory: /data/leo/working/fmriprep_wf/single_subject_02_wf/func_preproc_task_nnip_wf/ica_aroma_wf/ica_aroma

Node inputs:

TR = 2.14
args = -np
compress_report = auto
denoise_type = nonaggr
dim = <undefined>
environ = {}
feat_dir = <undefined>
fnirt_warp_file = <undefined>
in_file = <undefined>
mask = <undefined>
mat_file = <undefined>
melodic_dir = <undefined>
motion_parameters = <undefined>
out_dir = out
out_report = ica_aroma_reportlet.svg
report_mask = <undefined>

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 530, in aggregate_outputs
    setattr(outputs, key, val)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
    value = super(File, self).validate(objekt, name, value, return_pathlike=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
    self.error(objekt, name, str(value))
  File "/usr/local/miniconda/lib/python3.7/site-packages/traits/trait_handlers.py", line 172, in error
    value )
traits.trait_errors.TraitError: The 'nonaggr_denoised_file' trait of a _ICA_AROMAOutputSpecRPT instance must be a pathlike object or string representing an existing file, but a value of '/data/leo/working/fmriprep_wf/single_subject_02_wf/func_preproc_task_nnip_wf/ica_aroma_wf/ica_aroma/out/denoised_func_data_nonaggr.nii.gz' <class 'str'> was specified.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 429, in run
    runtime = self._post_run_hook(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/segmentation.py", line 211, in _post_run_hook
    outputs = self.aggregate_outputs(runtime=runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 537, in aggregate_outputs
    raise FileNotFoundError(msg)
FileNotFoundError: No such file or directory '/data/leo/working/fmriprep_wf/single_subject_02_wf/func_preproc_task_nnip_wf/ica_aroma_wf/ica_aroma/out/denoised_func_data_nonaggr.nii.gz' for output 'nonaggr_denoised_file' of a ICA_AROMARPT interface

'/data/leo/working/fmriprep_wf/single_subject_02_wf/func_preproc_task_nnip_wf/ica_aroma_wf/ica_aroma/out/denoised_func_data_nonaggr.nii.gz' is indeed nonexistent. But I’m not sure why…

The issue seems similar to 17023, but unfortunately that user didn’t share the solution.

Any help would be much appreciated!

Leo

Steven · November 8, 2021, 2:52pm

Can you share

your fMRIPrep command
your fMRIPrep installation (singularity, docker, baremetal python etc.)
your fMRIPrep version

lvbrussel · November 8, 2021, 3:04pm

Hi Steven,

Yes of course! (And a bit naïve of me to think that the error output would be enough!)

The shell script below has all the info (singularity container; fmriprep-20.2.5)

#!/bin/bash

# To get rid of the 'Fontconfig warning'
LC_ALL=C

INDIR="/data/leo/NNIP/NNIP_Bids2/"
OUTDIR="/data/leo/NNIP/NNIP_Bids2/derivatives/"
WORKDIR="/data/leo/working"
IMAGE="/data/fmriprep/fmriprep-20.2.5.simg" 
LABEL="02"
export FS_LICENSE=$HOME/license.txt

singularity run -B /data -B /run/shm:/run/shm $IMAGE \
     $INDIR $OUTDIR \
    participant --participant-label $LABEL \
    -w $WORKDIR \
    --n_cpus 1\
    --mem-mb 18000\
    --fs-no-reconall\
    --use-aroma

Note: I have successfully processed this data with fmriprep-20.2.0 before, but without aroma. Given that the dataset suffers from quite some motion artifacts, I am now interested in aroma results.

Note 2: Initially I ran the command for multiple subjects (9), which all finished with an error at the ica_aroma_wf.ica_aroma node. Since I did not empty the workdir, I tried running only sub-02, for which I received the error output in my first post.

Steven · November 8, 2021, 3:17pm

Thanks for the info! In your workdir, are there still remnants from your first 20.2.0 run? Can you try to use a new workdir for a single subject (or is that what you already tried)?

lvbrussel · November 8, 2021, 3:22pm

The workdir was empty prior to running 20.2.5 (if the working files from 20.2.0 had still been there, I would have probably decided against upgrading and used Aroma with 20.2.0 )

Maybe my attempt to run 9 subjects was a bit ambitious though, and caused some memory/disk errors… I’ll try your second suggestion!

lvbrussel · November 9, 2021, 2:48pm

Unfortunately setting a new workdir and running fmriprep on a single subject resulted in the same error…

Any other suggestions for what I could try?

Node: fmriprep_wf.single_subject_11_wf.func_preproc_task_nnip_wf.ica_aroma_wf.ica_aroma
Working directory: /data/leo/working_new/fmriprep_wf/single_subject_11_wf/func_preproc_task_nnip_wf/ica_aroma_wf/ica_aroma

Node inputs:

TR = 2.14
args = -np
compress_report = auto
denoise_type = nonaggr
dim = <undefined>
environ = {}
feat_dir = <undefined>
fnirt_warp_file = <undefined>
in_file = <undefined>
mask = <undefined>
mat_file = <undefined>
melodic_dir = <undefined>
motion_parameters = <undefined>
out_dir = out
out_report = ica_aroma_reportlet.svg
report_mask = <undefined>

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 530, in aggregate_outputs
    setattr(outputs, key, val)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
    value = super(File, self).validate(objekt, name, value, return_pathlike=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
    self.error(objekt, name, str(value))
  File "/usr/local/miniconda/lib/python3.7/site-packages/traits/trait_handlers.py", line 172, in error
    value )
traits.trait_errors.TraitError: The 'nonaggr_denoised_file' trait of a _ICA_AROMAOutputSpecRPT instance must be a pathlike object or string representing an existing file, but a value of '/data/leo/working_new/fmriprep_wf/single_subject_11_wf/func_preproc_task_nnip_wf/ica_aroma_wf/ica_aroma/out/denoised_func_data_nonaggr.nii.gz' <class 'str'> was specified.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 429, in run
    runtime = self._post_run_hook(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/segmentation.py", line 211, in _post_run_hook
    outputs = self.aggregate_outputs(runtime=runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 537, in aggregate_outputs
    raise FileNotFoundError(msg)
FileNotFoundError: No such file or directory '/data/leo/working_new/fmriprep_wf/single_subject_11_wf/func_preproc_task_nnip_wf/ica_aroma_wf/ica_aroma/out/denoised_func_data_nonaggr.nii.gz' for output 'nonaggr_denoised_file' of a ICA_AROMARPT interface

Steven · November 10, 2021, 9:05pm

I wonder if there was an error before this node too, e.g. one that was supposed to create the file that is being looked for here. Do you have a full log of the fMRIPrep run (e.g. an sbatch log)?

lvbrussel · November 10, 2021, 9:22pm

Hi Steven,

Not sure if the following is an sbatch log, but this is the full log of a fMRIprep re-run (e.g. retry after the run completed without AROMA). If this is not an sbatch log, could you advice how I could obtain one?

export PATH=/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/bin
ubuntu@packer-Ubuntu-18:~$ cd /data/leo/fmriprep/
ubuntu@packer-Ubuntu-18:/data/leo/fmriprep$ ./run_NNIP.sh
bids-validator@1.4.0

	1: [WARN] The recommended file /README is missing. See Section 03 (Modality agnostic files) of the BIDS specification. (code: 101 - README_FILE_MISSING)

	Please visit https://neurostars.org/search?q=README_FILE_MISSING for existing conversations about this issue.


        Summary:                  Available Tasks:        Available Modalities: 
        184 Files, 18.11GB        nnip                    T1w                   
        36 - Subjects                                     bold                  
        1 - Session                                       events                


	If you have any questions, please post on https://neurostars.org/tags/bids.

211110-12:53:31,547 nipype.workflow WARNING:
	 Previous output generated by version 0+unknown found.
211110-12:53:31,935 nipype.workflow IMPORTANT:
	 
    Running fMRIPREP version 20.2.5:
      * BIDS dataset path: /data/leo/NNIP/NNIP_Bids2.
      * Participant list: ['11'].
      * Run identifier: 20211110-125258_a8585b14-dc56-460b-8fa2-338f35970ee1.
      * Output spaces: MNI152NLin2009cAsym:res-native.
      * Pre-run FreeSurfer's SUBJECTS_DIR: /data/leo/NNIP/NNIP_Bids2/derivatives/freesurfer.
211110-12:53:33,586 nipype.workflow INFO:
	 No single-band-reference found for sub-11_task-nnip_bold.nii.gz.
211110-12:53:35,332 nipype.workflow IMPORTANT:
	 BOLD series will be slice-timing corrected to an offset of 1.05s.
211110-12:53:38,397 nipype.workflow INFO:
	 fMRIPrep workflow graph with 340 nodes built successfully.
Could not generate CITATION.html file:
pandoc -s --bibliography /usr/local/miniconda/lib/python3.7/site-packages/fmriprep/data/boilerplate.bib --filter pandoc-citeproc --metadata pagetitle="fMRIPrep citation boilerplate" /data/leo/NNIP/NNIP_Bids2/derivatives/fmriprep/logs/CITATION.md -o /data/leo/NNIP/NNIP_Bids2/derivatives/fmriprep/logs/CITATION.html
211110-12:54:00,114 nipype.workflow IMPORTANT:
	 fMRIPrep started!
211110-12:59:03,960 nipype.interface WARNING:
	 Changing /data/leo/NNIP/NNIP_Bids2/derivatives/fmriprep/sub-11/anat/sub-11_desc-brain_mask.nii.gz dtype from int16 to uint8
211110-13:00:12,923 nipype.interface WARNING:
	 Changing /data/leo/NNIP/NNIP_Bids2/derivatives/fmriprep/sub-11/anat/sub-11_space-MNI152NLin2009cAsym_dseg.nii.gz dtype from float64 to int16
211110-13:00:15,147 nipype.interface WARNING:
	 Changing /data/leo/NNIP/NNIP_Bids2/derivatives/fmriprep/sub-11/anat/sub-11_space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz dtype from float64 to uint8
211110-13:01:46,255 nipype.interface WARNING:
	 Changing /data/leo/NNIP/NNIP_Bids2/derivatives/fmriprep/sub-11/func/sub-11_task-nnip_space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz dtype from float64 to uint8
211110-13:03:16,991 nipype.workflow INFO:
	 [Node] Setting-up "fmriprep_wf.single_subject_11_wf.func_preproc_task_nnip_wf.ica_aroma_wf.ica_aroma" in "/data/leo/working_new/fmriprep_wf/single_subject_11_wf/func_preproc_task_nnip_wf/ica_aroma_wf/ica_aroma".
211110-13:03:17,88 nipype.workflow INFO:
	 [Node] Running "ica_aroma" ("niworkflows.interfaces.segmentation.ICA_AROMARPT"), a CommandLine Interface with command:
ICA_AROMA.py -tr 2.140 -np -den nonaggr -i /data/leo/working_new/fmriprep_wf/single_subject_11_wf/func_preproc_task_nnip_wf/ica_aroma_wf/smooth/vol0000_xform-00000_merged_cut_smooth.nii.gz -m /data/leo/working_new/fmriprep_wf/single_subject_11_wf/func_preproc_task_nnip_wf/bold_std_trans_wf/_std_target_MNI152NLin6Asym.res2/mask_std_tfm/ref_bold_corrected_brain_mask_maths_trans.nii.gz -meldir /data/leo/working_new/fmriprep_wf/single_subject_11_wf/func_preproc_task_nnip_wf/ica_aroma_wf/melodic -mc /data/leo/working_new/fmriprep_wf/single_subject_11_wf/func_preproc_task_nnip_wf/bold_hmc_wf/normalize_motion/motion_params.txt -o /data/leo/working_new/fmriprep_wf/single_subject_11_wf/func_preproc_task_nnip_wf/ica_aroma_wf/ica_aroma/out
211110-13:38:30,673 nipype.workflow WARNING:
	 Storing result file without outputs
211110-13:38:30,675 nipype.workflow WARNING:
	 [Node] Error on "fmriprep_wf.single_subject_11_wf.func_preproc_task_nnip_wf.ica_aroma_wf.ica_aroma" (/data/leo/working_new/fmriprep_wf/single_subject_11_wf/func_preproc_task_nnip_wf/ica_aroma_wf/ica_aroma)
211110-13:38:31,748 nipype.workflow ERROR:
	 Node ica_aroma failed to run on host packer-Ubuntu-18.
211110-13:38:31,757 nipype.workflow ERROR:
	 Saving crash info to /data/leo/NNIP/NNIP_Bids2/derivatives/fmriprep/sub-11/log/20211110-125258_a8585b14-dc56-460b-8fa2-338f35970ee1/crash-20211110-133831-ubuntu-ica_aroma-ade5fe66-1057-437c-a415-abb0682b6b79.txt
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 530, in aggregate_outputs
    setattr(outputs, key, val)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
    value = super(File, self).validate(objekt, name, value, return_pathlike=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
    self.error(objekt, name, str(value))
  File "/usr/local/miniconda/lib/python3.7/site-packages/traits/trait_handlers.py", line 172, in error
    value )
traits.trait_errors.TraitError: The 'nonaggr_denoised_file' trait of a _ICA_AROMAOutputSpecRPT instance must be a pathlike object or string representing an existing file, but a value of '/data/leo/working_new/fmriprep_wf/single_subject_11_wf/func_preproc_task_nnip_wf/ica_aroma_wf/ica_aroma/out/denoised_func_data_nonaggr.nii.gz' <class 'str'> was specified.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 429, in run
    runtime = self._post_run_hook(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/segmentation.py", line 211, in _post_run_hook
    outputs = self.aggregate_outputs(runtime=runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 537, in aggregate_outputs
    raise FileNotFoundError(msg)
FileNotFoundError: No such file or directory '/data/leo/working_new/fmriprep_wf/single_subject_11_wf/func_preproc_task_nnip_wf/ica_aroma_wf/ica_aroma/out/denoised_func_data_nonaggr.nii.gz' for output 'nonaggr_denoised_file' of a ICA_AROMARPT interface

211110-13:40:52,16 nipype.workflow ERROR:
	 could not run node: fmriprep_wf.single_subject_11_wf.func_preproc_task_nnip_wf.ica_aroma_wf.ica_aroma
211110-13:40:52,32 nipype.workflow CRITICAL:
	 fMRIPrep failed: Workflow did not execute cleanly. Check log for details
211110-13:40:53,133 cli ERROR:
	 Preprocessing did not finish successfully. Errors occurred while processing data from participants: 11 (1). Check the HTML reports for details.
Sentry is attempting to send 2 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit
ubuntu@packer-Ubuntu-18:/data/leo/fmriprep$

Note: Because I found a similar issue on the fMRIprep GitHub I decided to crosspost here: https://github.com/nipreps/fmriprep/issues/2622

Thanks again,

Leo