RuntimeError: ICA-AROMA failed; fmriprep version 1.1.6

fmriprep
ica-aroma

#1

Hey,

I am running fMRIprep with ICA-AROMA and it worked perfectly for all my subjects, apart from one, for which fmriprep crashes with the following error message:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/raven/transport/threaded.py", line 165, in send_sync
    super(ThreadedHTTPTransport, self).send(url, data, headers)
  File "/usr/local/miniconda/lib/python3.6/site-packages/raven/transport/http.py", line 43, in send
    ca_certs=self.ca_certs,
  File "/usr/local/miniconda/lib/python3.6/site-packages/raven/utils/http.py", line 66, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/miniconda/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/usr/local/miniconda/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/usr/local/miniconda/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/local/miniconda/lib/python3.6/site-packages/raven/utils/http.py", line 46, in https_open
    return self.do_open(ValidHTTPSConnection, req)
  File "/usr/local/miniconda/lib/python3.6/urllib/request.py", line 1320, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno -3] Temporary failure in name resolution>
['fMRIPrep running']
180930-09:17:09,261 nipype.workflow INFO:
	 [Node] Setting-up "fmriprep_wf.single_subject_108_wf.anat_preproc_wf.surface_recon_wf.gifti_surface_wf.get_surfaces" in "/tmp/work/fmriprep_wf/single_subject_108_wf/anat_preproc_wf/surface_recon_wf/gifti_surface_wf/get_surfaces".180930-09:17:09,261 nipype.workflow INFO:
	 [Node] Setting-up "fmriprep_wf.single_subject_108_wf.anat_preproc_wf.surface_recon_wf.segs_to_native_aparc_aseg.fs_datasource" in "/tmp/work/fmriprep_wf/single_subject_108_wf/anat_preproc_wf/surface_recon_wf/segs_to_native_aparc_aseg/fs_datasource".

180930-09:17:09,264 nipype.workflow INFO:
	 [Node] Running "fs_datasource" ("nipype.interfaces.io.FreeSurferSource")180930-09:17:09,264 nipype.workflow INFO:
	 [Node] Running "get_surfaces" ("nipype.interfaces.io.FreeSurferSource")

180930-09:17:09,285 nipype.workflow INFO:
	 [Node] Finished "fmriprep_wf.single_subject_108_wf.anat_preproc_wf.surface_recon_wf.segs_to_native_aparc_aseg.fs_datasource".
180930-09:17:09,285 nipype.workflow INFO:
	 [Node] Finished "fmriprep_wf.single_subject_108_wf.anat_preproc_wf.surface_recon_wf.gifti_surface_wf.get_surfaces".
180930-09:17:09,831 nipype.workflow INFO:
	 [Node] Setting-up "fmriprep_wf.single_subject_108_wf.anat_preproc_wf.surface_recon_wf.segs_to_native_aseg.fs_datasource" in "/tmp/work/fmriprep_wf/single_subject_108_wf/anat_preproc_wf/surface_recon_wf/segs_to_native_aseg/fs_datasource".
180930-09:17:09,835 nipype.workflow INFO:
	 [Node] Running "fs_datasource" ("nipype.interfaces.io.FreeSurferSource")
180930-09:17:09,855 nipype.workflow INFO:
	 [Node] Finished "fmriprep_wf.single_subject_108_wf.anat_preproc_wf.surface_recon_wf.segs_to_native_aseg.fs_datasource".
180930-09:17:58,82 nipype.workflow INFO:
	 [Node] Setting-up "fmriprep_wf.single_subject_108_wf.func_preproc_task_visualfixedlength_run_04_wf.ica_aroma_wf.ica_aroma_confound_extraction" in "/tmp/work/fmriprep_wf/single_subject_108_wf/func_preproc_task_visualfixedlength_run_04_wf/ica_aroma_wf/ica_aroma_confound_extraction".
180930-09:17:58,83 nipype.workflow INFO:
	 [Node] Outdated cache found for "fmriprep_wf.single_subject_108_wf.func_preproc_task_visualfixedlength_run_04_wf.ica_aroma_wf.ica_aroma_confound_extraction".
180930-09:17:58,86 nipype.workflow INFO:
	 [Node] Running "ica_aroma_confound_extraction" ("fmriprep.interfaces.confounds.ICAConfounds")
/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/interfaces/confounds.py:224: UserWarning: loadtxt: Empty input file: "/tmp/work/fmriprep_wf/single_subject_108_wf/func_preproc_task_visualfixedlength_run_04_wf/ica_aroma_wf/ica_aroma/out/classified_motion_ICs.txt"
  motion_ic_indices = np.loadtxt(motion_ics, dtype=int, delimiter=',', ndmin=1) - 1
180930-09:17:58,97 nipype.interface WARNING:
	 No noise components were classified
180930-09:17:58,98 nipype.workflow WARNING:
	 [Node] Error on "fmriprep_wf.single_subject_108_wf.func_preproc_task_visualfixedlength_run_04_wf.ica_aroma_wf.ica_aroma_confound_extraction" (/tmp/work/fmriprep_wf/single_subject_108_wf/func_preproc_task_visualfixedlength_run_04_wf/ica_aroma_wf/ica_aroma_confound_extraction)
Traceback (most recent call last):
  File "/usr/local/miniconda/bin/fmriprep", line 11, in <module>
    sys.exit(main())
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/cli/run.py", line 344, in main
    fmriprep_wf.run(**plugin_settings)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/workflows.py", line 595, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/base.py", line 162, in run
    self._clean_queue(jobid, graph, result=result))
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/base.py", line 224, in _clean_queue
    raise RuntimeError("".join(result['traceback']))
RuntimeError: Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 471, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 555, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 521, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/interfaces/confounds.py", line 119, in _run_interface
    raise RuntimeError('ICA-AROMA failed')
RuntimeError: ICA-AROMA failed

Sentry is attempting to send 1 pending error messages

Can anyone tell why it’s failing? I am on an Ubuntu system, using fmriprep v.1.1.6. This is the command I am using:

sudo docker run -it --rm -v /data/data01/BV-RT/sourcedata/:/data:ro -v /data/data01/Reece/BV-RT/fmriprep/:/out -v /data/data01/BV-RT/FS_license.txt:/opt/freesurfer/license.txt poldracklab/fmriprep:latest /data /out/out participant --participant-label 108 --fs-license-file /opt/freesurfer/license.txt --output-space template --use-aroma

Any help would be greatly appreciated,
Kristina


#2

I just tried running it with --ignore-aroma-denoising-errors, but then I got the same error as in this (Ica-aroma error with fmriprep: JoinTSVColumns requires a value for input 'join_file') post:

Traceback (most recent call last):
  File "/usr/local/miniconda/bin/fmriprep", line 11, in <module>
    sys.exit(main())
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/cli/run.py", line 344, in main
    fmriprep_wf.run(**plugin_settings)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/workflows.py", line 595, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/base.py", line 162, in run
    self._clean_queue(jobid, graph, result=result))
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/base.py", line 224, in _clean_queue
    raise RuntimeError("".join(result['traceback']))
RuntimeError: Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 471, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 555, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 480, in run
    self._check_mandatory_inputs()
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 389, in _check_mandatory_inputs
    raise ValueError(msg)
ValueError: JoinTSVColumns requires a value for input 'join_file'. For a list of required inputs, see JoinTSVColumns.help()

This error persisted when I used --low-mem. Any ideas what else I can try?

Thanks,
Kristina


#3

Hi Kristina,

Thanks for posting here. There are a couple things I can think of that could cause this behavior.

  1. The nifti file could have a fairly short number of volumes (does this nifti file look normal?)
  2. Perhaps the classified_noise_ICs.txt file was not written to or was for some reason inaccessible (can you check if you have access to this file: /tmp/work/fmriprep_wf/single_subject_108_wf/func_preproc_task_visualfixedlength_run_04_wf/ica_aroma_wf/ica_aroma/out/classified_motion_ICs.txt)?
  3. The subject is truly noiseless, for a better ground truth, you can run ICA-AROMA.py directly on the data. To do this you can look at the command that will be in this directory (change the prefix /some/directory/ with your working directory):
    /some/directory/fmriprep_wf/single_subject_108_wf/func_preproc_task_visualfixedlength_run_04_wf/ica_aroma_wf/ica_aroma/command.txt
    • additionally, you can check the input nifti file (after the -i) specified in the command.txt file and see if that nifti file also looks “normal” (e.g. same number of volumes, etc.).

For numbers 2 & 3, you will need to modify your docker command to include a working directory to save all the intermediate files -w /some/directory and you may want to mount another directory in your docker command to serve as the working directory in the fmriprep command. for example it may look something like this:

sudo docker run -it --rm \
-v /data/data01/BV-RT/sourcedata/:/data:ro \
-v /data/data01/Reece/BV-RT/fmriprep/:/out \
-v /data/data01/BV-RT/FS_license.txt:/opt/freesurfer/license.txt \
-v /some/working/directory:/work \
poldracklab/fmriprep:latest /data /out/out participant \
--participant-label 108 \
--fs-license-file /opt/freesurfer/license.txt \
--output-space template --use-aroma
-w /work

Let me know if you have any questions I can help with.


#4

Hi @jdkent,

thank you so much for your thorough reply.

I checked the nifti file and it looks ok. The input file to ICA-AROMA also looks fine.

You were right. The classified_noise_ICs.txt file is empty. I can access this file though, and checked permissions and they look fine. In the classification_overview.txt no components are classified as noise (FALSE for every row in Motion/noise), and all rows for Edge fraction and CSF fraction are 0. Do you have any idea why that might be the case, or what else I can check to see what’s going on? Visual inspection of the components definitely suggests that there should be motion noise components in there.

After re-running the subject again, this time a report was generated (so far it always crashed before the report was generated), but again with this error:

Node Name: fmriprep_wf.single_subject_108_wf.func_preproc_task_visualfixedlength_run_04_wf.ica_aroma_wf.ica_aroma_confound_extraction
File: /out/out/fmriprep/sub-108/log/20181001-222836_0b7246f3-9631-457c-b189-eef5f6f19627/crash-20181002-004803-root-ica_aroma_confound_extraction-24887099-550a-4998-8f9e-2f47735bcc74.txt
Working Directory: /work/fmriprep_wf/single_subject_108_wf/func_preproc_task_visualfixedlength_run_04_wf/ica_aroma_wf/ica_aroma_confound_extraction
Inputs:

    ignore_aroma_err: False
    in_directory: /work/fmriprep_wf/single_subject_108_wf/func_preproc_task_visualfixedlength_run_04_wf/ica_aroma_wf/ica_aroma/out

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 471, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 555, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 521, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/interfaces/confounds.py", line 119, in _run_interface
    raise RuntimeError('ICA-AROMA failed')
RuntimeError: ICA-AROMA failed

So is this error due to the fact that none of the components have been classified as noise?

Thanks,
Kristina


#5

To me this indicates there was some failure of the registration to MNI space, but the resulting MNI images look reasonable?


#6

Yeah, the images all look fine. Is there anything else you can think of that I could check?


#7

I don’t have anything off the top of my head currently. Would it be possible to share a minimal BIDS dataset with that subject (structural images defaced) with me?


#8

@jdkent I sent you a private message with the data. Please let me know if you’d prefer me to send it via email. Thank you so much for helping out!


#9

Thanks! got it.

I ran fmriprep and replicated your error, the first thing I found odd is that this file is completely blank:

/work/fmriprep_wf/single_subject_108_wf/func_preproc_task_visualfixedlength_run_04_wf/ica_aroma_wf/ica_aroma/out/melodic_IC_thr_MNI2mm.nii.gz

That would explain why the edge and csf fractions are zero, but I don’t know why that file is empty, the zstat files from melodic that make up that image look normal. I’ll keep you updated on anything I find.


#10

Here’s my current guess for a solution, I cannot tell how the mask is generated for ica_aroma when you just pass in the melodic directory, so I am explicitly passing in a mask, you can see the code change here.

I’m testing this solution on your data now, I’ll update again if this solution works.

EDIT: The fix worked! :tada: I’ll submit a pull request to fmriprep, once it’s merged in, I can give instructions on how to modify your docker command to use a non-release version of fmriprep.


#11

Oh awesome!! Thank you so very very much for your kind help. Looking forward to hearing about the modified docker command. :slight_smile: