RuntimeError: ICA-AROMA failed; fmriprep version 1.1.6

kwiebels · October 1, 2018, 3:47am

Hey,

I am running fMRIprep with ICA-AROMA and it worked perfectly for all my subjects, apart from one, for which fmriprep crashes with the following error message:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/raven/transport/threaded.py", line 165, in send_sync
    super(ThreadedHTTPTransport, self).send(url, data, headers)
  File "/usr/local/miniconda/lib/python3.6/site-packages/raven/transport/http.py", line 43, in send
    ca_certs=self.ca_certs,
  File "/usr/local/miniconda/lib/python3.6/site-packages/raven/utils/http.py", line 66, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/miniconda/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/usr/local/miniconda/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/usr/local/miniconda/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/local/miniconda/lib/python3.6/site-packages/raven/utils/http.py", line 46, in https_open
    return self.do_open(ValidHTTPSConnection, req)
  File "/usr/local/miniconda/lib/python3.6/urllib/request.py", line 1320, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno -3] Temporary failure in name resolution>
['fMRIPrep running']
180930-09:17:09,261 nipype.workflow INFO:
	 [Node] Setting-up "fmriprep_wf.single_subject_108_wf.anat_preproc_wf.surface_recon_wf.gifti_surface_wf.get_surfaces" in "/tmp/work/fmriprep_wf/single_subject_108_wf/anat_preproc_wf/surface_recon_wf/gifti_surface_wf/get_surfaces".180930-09:17:09,261 nipype.workflow INFO:
	 [Node] Setting-up "fmriprep_wf.single_subject_108_wf.anat_preproc_wf.surface_recon_wf.segs_to_native_aparc_aseg.fs_datasource" in "/tmp/work/fmriprep_wf/single_subject_108_wf/anat_preproc_wf/surface_recon_wf/segs_to_native_aparc_aseg/fs_datasource".

180930-09:17:09,264 nipype.workflow INFO:
	 [Node] Running "fs_datasource" ("nipype.interfaces.io.FreeSurferSource")180930-09:17:09,264 nipype.workflow INFO:
	 [Node] Running "get_surfaces" ("nipype.interfaces.io.FreeSurferSource")

180930-09:17:09,285 nipype.workflow INFO:
	 [Node] Finished "fmriprep_wf.single_subject_108_wf.anat_preproc_wf.surface_recon_wf.segs_to_native_aparc_aseg.fs_datasource".
180930-09:17:09,285 nipype.workflow INFO:
	 [Node] Finished "fmriprep_wf.single_subject_108_wf.anat_preproc_wf.surface_recon_wf.gifti_surface_wf.get_surfaces".
180930-09:17:09,831 nipype.workflow INFO:
	 [Node] Setting-up "fmriprep_wf.single_subject_108_wf.anat_preproc_wf.surface_recon_wf.segs_to_native_aseg.fs_datasource" in "/tmp/work/fmriprep_wf/single_subject_108_wf/anat_preproc_wf/surface_recon_wf/segs_to_native_aseg/fs_datasource".
180930-09:17:09,835 nipype.workflow INFO:
	 [Node] Running "fs_datasource" ("nipype.interfaces.io.FreeSurferSource")
180930-09:17:09,855 nipype.workflow INFO:
	 [Node] Finished "fmriprep_wf.single_subject_108_wf.anat_preproc_wf.surface_recon_wf.segs_to_native_aseg.fs_datasource".
180930-09:17:58,82 nipype.workflow INFO:
	 [Node] Setting-up "fmriprep_wf.single_subject_108_wf.func_preproc_task_visualfixedlength_run_04_wf.ica_aroma_wf.ica_aroma_confound_extraction" in "/tmp/work/fmriprep_wf/single_subject_108_wf/func_preproc_task_visualfixedlength_run_04_wf/ica_aroma_wf/ica_aroma_confound_extraction".
180930-09:17:58,83 nipype.workflow INFO:
	 [Node] Outdated cache found for "fmriprep_wf.single_subject_108_wf.func_preproc_task_visualfixedlength_run_04_wf.ica_aroma_wf.ica_aroma_confound_extraction".
180930-09:17:58,86 nipype.workflow INFO:
	 [Node] Running "ica_aroma_confound_extraction" ("fmriprep.interfaces.confounds.ICAConfounds")
/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/interfaces/confounds.py:224: UserWarning: loadtxt: Empty input file: "/tmp/work/fmriprep_wf/single_subject_108_wf/func_preproc_task_visualfixedlength_run_04_wf/ica_aroma_wf/ica_aroma/out/classified_motion_ICs.txt"
  motion_ic_indices = np.loadtxt(motion_ics, dtype=int, delimiter=',', ndmin=1) - 1
180930-09:17:58,97 nipype.interface WARNING:
	 No noise components were classified
180930-09:17:58,98 nipype.workflow WARNING:
	 [Node] Error on "fmriprep_wf.single_subject_108_wf.func_preproc_task_visualfixedlength_run_04_wf.ica_aroma_wf.ica_aroma_confound_extraction" (/tmp/work/fmriprep_wf/single_subject_108_wf/func_preproc_task_visualfixedlength_run_04_wf/ica_aroma_wf/ica_aroma_confound_extraction)
Traceback (most recent call last):
  File "/usr/local/miniconda/bin/fmriprep", line 11, in <module>
    sys.exit(main())
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/cli/run.py", line 344, in main
    fmriprep_wf.run(**plugin_settings)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/workflows.py", line 595, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/base.py", line 162, in run
    self._clean_queue(jobid, graph, result=result))
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/base.py", line 224, in _clean_queue
    raise RuntimeError("".join(result['traceback']))
RuntimeError: Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 471, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 555, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 521, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/interfaces/confounds.py", line 119, in _run_interface
    raise RuntimeError('ICA-AROMA failed')
RuntimeError: ICA-AROMA failed

Sentry is attempting to send 1 pending error messages

Can anyone tell why it’s failing? I am on an Ubuntu system, using fmriprep v.1.1.6. This is the command I am using:

sudo docker run -it --rm -v /data/data01/BV-RT/sourcedata/:/data:ro -v /data/data01/Reece/BV-RT/fmriprep/:/out -v /data/data01/BV-RT/FS_license.txt:/opt/freesurfer/license.txt poldracklab/fmriprep:latest /data /out/out participant --participant-label 108 --fs-license-file /opt/freesurfer/license.txt --output-space template --use-aroma

Any help would be greatly appreciated,
Kristina

kwiebels · October 1, 2018, 8:13am

I just tried running it with --ignore-aroma-denoising-errors, but then I got the same error as in this (Ica-aroma error with fmriprep: JoinTSVColumns requires a value for input 'join_file') post:

Traceback (most recent call last):
  File "/usr/local/miniconda/bin/fmriprep", line 11, in <module>
    sys.exit(main())
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/cli/run.py", line 344, in main
    fmriprep_wf.run(**plugin_settings)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/workflows.py", line 595, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/base.py", line 162, in run
    self._clean_queue(jobid, graph, result=result))
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/base.py", line 224, in _clean_queue
    raise RuntimeError("".join(result['traceback']))
RuntimeError: Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 471, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 555, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 480, in run
    self._check_mandatory_inputs()
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 389, in _check_mandatory_inputs
    raise ValueError(msg)
ValueError: JoinTSVColumns requires a value for input 'join_file'. For a list of required inputs, see JoinTSVColumns.help()

This error persisted when I used --low-mem. Any ideas what else I can try?

Thanks,
Kristina

jdkent · October 1, 2018, 7:09pm

Hi Kristina,

Thanks for posting here. There are a couple things I can think of that could cause this behavior.

The nifti file could have a fairly short number of volumes (does this nifti file look normal?)
Perhaps the classified_noise_ICs.txt file was not written to or was for some reason inaccessible (can you check if you have access to this file: /tmp/work/fmriprep_wf/single_subject_108_wf/func_preproc_task_visualfixedlength_run_04_wf/ica_aroma_wf/ica_aroma/out/classified_motion_ICs.txt)?
The subject is truly noiseless, for a better ground truth, you can run ICA-AROMA.py directly on the data. To do this you can look at the command that will be in this directory (change the prefix /some/directory/ with your working directory):
/some/directory/fmriprep_wf/single_subject_108_wf/func_preproc_task_visualfixedlength_run_04_wf/ica_aroma_wf/ica_aroma/command.txt
- additionally, you can check the input nifti file (after the -i) specified in the command.txt file and see if that nifti file also looks “normal” (e.g. same number of volumes, etc.).

For numbers 2 & 3, you will need to modify your docker command to include a working directory to save all the intermediate files -w /some/directory and you may want to mount another directory in your docker command to serve as the working directory in the fmriprep command. for example it may look something like this:

sudo docker run -it --rm \
-v /data/data01/BV-RT/sourcedata/:/data:ro \
-v /data/data01/Reece/BV-RT/fmriprep/:/out \
-v /data/data01/BV-RT/FS_license.txt:/opt/freesurfer/license.txt \
-v /some/working/directory:/work \
poldracklab/fmriprep:latest /data /out/out participant \
--participant-label 108 \
--fs-license-file /opt/freesurfer/license.txt \
--output-space template --use-aroma
-w /work

Let me know if you have any questions I can help with.

kwiebels · October 2, 2018, 3:13am

Hi @jdkent,

thank you so much for your thorough reply.

I checked the nifti file and it looks ok. The input file to ICA-AROMA also looks fine.

You were right. The classified_noise_ICs.txt file is empty. I can access this file though, and checked permissions and they look fine. In the classification_overview.txt no components are classified as noise (FALSE for every row in Motion/noise), and all rows for Edge fraction and CSF fraction are 0. Do you have any idea why that might be the case, or what else I can check to see what’s going on? Visual inspection of the components definitely suggests that there should be motion noise components in there.

After re-running the subject again, this time a report was generated (so far it always crashed before the report was generated), but again with this error:

Node Name: fmriprep_wf.single_subject_108_wf.func_preproc_task_visualfixedlength_run_04_wf.ica_aroma_wf.ica_aroma_confound_extraction
File: /out/out/fmriprep/sub-108/log/20181001-222836_0b7246f3-9631-457c-b189-eef5f6f19627/crash-20181002-004803-root-ica_aroma_confound_extraction-24887099-550a-4998-8f9e-2f47735bcc74.txt
Working Directory: /work/fmriprep_wf/single_subject_108_wf/func_preproc_task_visualfixedlength_run_04_wf/ica_aroma_wf/ica_aroma_confound_extraction
Inputs:

    ignore_aroma_err: False
    in_directory: /work/fmriprep_wf/single_subject_108_wf/func_preproc_task_visualfixedlength_run_04_wf/ica_aroma_wf/ica_aroma/out

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 471, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 555, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 521, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/interfaces/confounds.py", line 119, in _run_interface
    raise RuntimeError('ICA-AROMA failed')
RuntimeError: ICA-AROMA failed

So is this error due to the fact that none of the components have been classified as noise?

Thanks,
Kristina

jdkent · October 3, 2018, 3:09am

To me this indicates there was some failure of the registration to MNI space, but the resulting MNI images look reasonable?

kwiebels · October 3, 2018, 3:36am

Yeah, the images all look fine. Is there anything else you can think of that I could check?

jdkent · October 3, 2018, 12:51pm

I don’t have anything off the top of my head currently. Would it be possible to share a minimal BIDS dataset with that subject (structural images defaced) with me?

kwiebels · October 14, 2018, 6:01am

@jdkent I sent you a private message with the data. Please let me know if you’d prefer me to send it via email. Thank you so much for helping out!

jdkent · October 17, 2018, 4:08am

Thanks! got it.

I ran fmriprep and replicated your error, the first thing I found odd is that this file is completely blank:

/work/fmriprep_wf/single_subject_108_wf/func_preproc_task_visualfixedlength_run_04_wf/ica_aroma_wf/ica_aroma/out/melodic_IC_thr_MNI2mm.nii.gz

That would explain why the edge and csf fractions are zero, but I don’t know why that file is empty, the zstat files from melodic that make up that image look normal. I’ll keep you updated on anything I find.

jdkent · October 17, 2018, 9:39pm

Here’s my current guess for a solution, I cannot tell how the mask is generated for ica_aroma when you just pass in the melodic directory, so I am explicitly passing in a mask, you can see the code change here.

I’m testing this solution on your data now, I’ll update again if this solution works.

EDIT: The fix worked! I’ll submit a pull request to fmriprep, once it’s merged in, I can give instructions on how to modify your docker command to use a non-release version of fmriprep.

kwiebels · October 21, 2018, 6:02am

Oh awesome!! Thank you so very very much for your kind help. Looking forward to hearing about the modified docker command.

jdkent · October 22, 2018, 1:53am

# go somewhere to download the fmriprep repository
cd /some/dir/
# clone (download) the repository
git clone https://github.com/poldracklab/fmriprep.git

# call your docker command with fmriprep patched
sudo docker run -it --rm \
-v /some/dir/fmriprep/fmriprep:/usr/local/miniconda/lib/python3.6/site-packages/fmriprep:ro \
-v /data/data01/BV-RT/sourcedata/:/data:ro \
-v /data/data01/Reece/BV-RT/fmriprep/:/out \
-v /data/data01/BV-RT/FS_license.txt:/opt/freesurfer/license.txt \
-v /some/working/directory:/work \
poldracklab/fmriprep:latest /data /out/out participant \
--participant-label 108 \
--fs-license-file /opt/freesurfer/license.txt \
--output-space template --use-aroma
-w /work

The command was modified following fmriprep’s documentation
NOTE: this also has the working directory specified which is not necessary.

kwiebels · October 22, 2018, 8:20pm

It worked! Thank you so much!!

TribikramT · March 18, 2019, 2:44am

Hi @jdkent

I am currently using fMRIPrep v1.2.5, and I’m running into a similar issue where I don’t get the .csv, confounds, and melodic.tsv files with one of my subjects (out of 41). The error I get is:

I’ve double checked the nii.gz files, and classified_noise_ICs.txt files to make sure they’re right. No issues here. But what I’ve noticed is that all components are classified as noise. Therefore, I’ve followed the post (https://github.com/poldracklab/fmriprep/pull/1467), and run the --aroma-melodic-dimensionality 1 flag. While this works (i.e, the no. of following dimensionality determines the number of the classified_motion_ICs), I was just curious whether the number following –aroma-melodic-dimensionality should represent the number for ICs identified?

Further, I ran the same subject with fMRIPrep v1.1.1, and received no error with ICA. Here, what I’ve noticed is that the classified_motion_ICs.txt contains 1 to 58 ICs, as opposed to 1 to 55 from fMRIPrep v1.2.5. Any reason why this could be happening?

Hope this makes sense.

Please let me know if you need further information.

Cheers,
Thapa

jdkent · March 18, 2019, 5:07pm

Hi Thapa,

Thank you for your thorough description. I have a followup question.

Is your data multiband or multi-echo (could potentially explain the strange classification)?

I was just curious whether the number following –aroma-melodic-dimensionality should represent the number for ICs identified?

--aroma-melodic-dimensionality does represent the total number of dimensions ICA can split the bold run into, it does not specify how many components should be classified as noise (or signal). The reason I’m using that option in the pull request you referenced is to try to make all the components look like noise. However, limiting the dimensions to 1 did not work for me (see finding 3). Hope that is clearer.

Further, I ran the same subject with fMRIPrep v1.1.1, and received no error with ICA. Here, what I’ve noticed is that the classified_motion_ICs.txt contains 1 to 58 ICs, as opposed to 1 to 55 from fMRIPrep v1.2.5. Any reason why this could be happening?

ICA-AROMA is a bit of a stochastic algorithm (i.e. it include randomness in it’s code), so the different number of components could be due to that, or the fact that in v1.2.5 a mask is explicitly provided to ICA-AROMA, but in v1.1.1 there is not an explicit mask.

A potential solution (perhaps not ideal), is to update your version of fmriprep to at least v1.3.0.post3 where that pull request you referenced is merged into the codebase. You won’t get an error, but all the components may still get classified as noise, which may have something to do with how the data were acquired.

Best,
James

TribikramT · March 19, 2019, 4:00am

Hi @jdkent

Thank you for your reply.

This is a single echo dataset, not a multiband or ME. Thank you for clearing my question around --aroma-melodic-dimensionality, which I understood wrong. I thought the number represented the number of ICs as opposed to the the number of dimensions.

This makes sense as to why I am getting an error with one version but not the other. However, is there a way to determine which approach is correct (or preferred)?

Interesting you mention this issue could arise depending upon how the data was acquired. Could you please clarify this further? I can re-check my raw data and look for issues that you suggest may exist.

Thank you.

Cheers,
Thapa

jdkent · March 20, 2019, 4:08am

Sorry, I must be confusing you, the option --aroma-melodic-dimensionality does represent the number of ICs you receive, when I reduced the number to just 1 IC (a 3D image), ICA-AROMA barfs since it expects at least 2 ICs (a 4D image), if I passed 3 or 4 or whatever number above 2 into --aroma-melodic-dimensionality, the image would still be 4D and ICA-AROMA would be happy.

Could you share the classification_overview.txt file with me? That will at least give a sense of why the components are being classified as noise (e.g. do they all strongly correlate with motion (in this case the data may too noisy for ICA-AROMA), do they all have large edge-fractions (in this case the registration may be off)).

If we can’t solve the issue using classification_overview.txt, I may ask if you could share a minimal BIDS dataset with that problematic individual if you are able.

TribikramT · March 21, 2019, 4:30am

Ah! Thanks for clearing my misunderstanding

I’ve attached images of the subject with the issue. Please take note of the different versions of fMRI prep used, and the different number of ICs obtained.

Here are the classified_moion_ICs.txt

Yes, I’d be happy to share the dataset with you.

Thank you for your help.

Cheers,
Thapa

jdkent · March 22, 2019, 2:57am

Thank you for showing those files. No pattern really jumps out at me. Looks like noisy data. It is possible that something went wrong with the registration leading to inflated values for the edge-fraction, but I may as well ask for the data.

Before I do that, however, I’m curious if you are using a separate working directory (specified with the -w option) for your fmriprep v1.3.0_post2 run? If you did, then the next step is to make sure your anatomical images are defaced (you can use pydeface) and you can send me a minimal bids dataset containing sub-22 and I will be in a better place to reproduce and debug your error.

I will be away at a conference until next Tuesday so I will have limited time to work on this until then, but I will see what I can do.

ChrisGorgolewski · March 22, 2019, 4:09am

Sharing the HTML reports (with figures) might be also helpful. They are generated even if there were errors.

As a side note - never encode personal information (such as names or initials) in participant labels. It makes an accidental release of participant data much more likely.