FMRIPREP errors during freesurfer recon-all or during ICA-aroma

1

Hi there,

I’ve started using fmriprep very recently to preprocess my data on our lab’s Linux cluster. I am encountering several issues and I am trying to determine if they are due to bugs in the pipeline or to computing issues related to the clusters.
So far, I’ve mainly tried to only run one subject at a time (or max 1 subject per node).

I am using FMRIPREP version: 1.0.6-2, from a singularity container, and my fmriprep command is as follows:

singularity run /home/ccharpen/fmriprep/poldracklab_fmriprep_latest-2018-01-30-2863112d2d8d.img -w /home/ccharpen/ObsLearn/prepdata/work /home/ccharpen/ObsLearn/rawdata/rawBIDS /home/ccharpen/ObsLearn/prepdata/fmriprep participant --participant-label 007 --ignore slicetiming --use-aroma --ignore-aroma-denoising-errors --fs-license-file /home/ccharpen/fmriprep/license.txt

So far, out of 7 subjects, one has succeeded on the first try. For all the others, it’s either crashing during freesurfer surface reconstruction or during ICA-AROMA for one or more functional runs (see error messages below).

At first my guess was that it was maybe a memory error since those processes seem to be the heaviest in terms of memory usage. Also our HPcluster is set up to kill jobs when they go over the requested memory for that job (which I had specified at 6GB); but when trying to re-run them with either 24GB or 32GB, some subjects ran but some others still crashed - which makes me think there is more to it than memory…

Any thoughts?? Let me know if you need more information!

Thanks,
Caroline

Freesurfer recon-all error:

Applying LTAtransformInterp (resample_type 1)
writing to /home/ccharpen/ObsLearn/prepdata/fmriprep/freesurfer/sub-004/mri/T2.prenorm.mgz…

mri_normalize -sigma 0.5 -nonmax_suppress 0 -min_dist 1 -aseg /home/ccharpen/ObsLearn/prepdata/fmriprep/freesurfer/sub-004/mri/aseg.presurf.mgz -surface /home/ccharpen/ObsLearn/prepdata/fmriprep/freesurfer/sub-004/surf/rh.white identity.nofile -surface /home/ccharpen/ObsLearn/prepdata/fmriprep/freesurfer/sub-004/surf/lh.white identity.nofile /home/ccharpen/ObsLearn/prepdata/fmriprep/freesurfer/sub-004/mri/T2.prenorm.mgz /home/ccharpen/ObsLearn/prepdata/fmriprep/freesurfer/sub-004/mri/T2.norm.mgz

mghRead(/home/ccharpen/ObsLearn/prepdata/fmriprep/freesurfer/sub-004/mri/T2.prenorm.mgz): could not read 262144 bytes at slice 150
using Gaussian smoothing of bias field, sigma=0.500
disabling nonmaximum suppression
retaining points that are at least 1.000mm from the boundary
using segmentation for initial intensity normalization
reading from /home/ccharpen/ObsLearn/prepdata/fmriprep/freesurfer/sub-004/mri/T2.prenorm.mgz…
No such file or directory
mri_normalize: could not open source file /home/ccharpen/ObsLearn/prepdata/fmriprep/freesurfer/sub-004/mri/T2.prenorm.mgz
No such file or directory
Linux master 3.13.0-109-generic #156-Ubuntu SMP Wed Feb 8 16:09:17 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

recon-all -s sub-004 exited with ERRORS at Thu Mar 22 10:21:51 UTC 2018

ICA-AROMA error:

Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/multiproc.py”, line 68, in run_node
result[‘result’] = node.run(updatehash=updatehash)
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py”, line 487, in run
result = self._run_interface(execute=True)
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py”, line 571, in _run_interface
return self._run_command(execute)
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py”, line 650, in _run_command
result = self._interface.run(cwd=outdir)
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base/core.py”, line 516, in run
runtime = self._run_interface(runtime)
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/interfaces/report_base.py”, line 54, in _run_interface
ReportCapableInterface, self)._run_interface(runtime)
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base/core.py”, line 1023, in _run_interface
self.raise_exception(runtime)
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base/core.py”, line 960, in raise_exception
).format(**runtime.dictcopy()))
RuntimeError: Command:
ICA_AROMA.py -den nonaggr -i /home/ccharpen/ObsLearn/prepdata/work/fmriprep_wf/single_subject_007_wf/func_preproc_task_ObsLearn_run_04_wf/bold_confounds_wf/ica_aroma_wf/smooth/vol0000_xform-00000_merged_smooth.nii.gz -meldir /home/ccharpen/ObsLearn/prepdata/work/fmriprep_wf/single_subject_007_wf/func_preproc_task_ObsLearn_run_04_wf/bold_confounds_wf/ica_aroma_wf/melodic -mc /home/ccharpen/ObsLearn/prepdata/work/fmriprep_wf/single_subject_007_wf/func_preproc_task_ObsLearn_run_04_wf/bold_hmc_wf/normalize_motion/motion_params.txt -o /home/ccharpen/ObsLearn/prepdata/work/fmriprep_wf/single_subject_007_wf/func_preproc_task_ObsLearn_run_04_wf/bold_confounds_wf/ica_aroma_wf/ica_aroma
Standard output:

------------------------------- RUNNING ICA-AROMA -------------------------------
--------------- ‘ICA-based Automatic Removal Of Motion Artifacts’ ---------------

Warning! Please check whether the determined TR (of 1.0s) is correct!

Step 1) MELODIC

  • The existing/specified MELODIC directory will be used.
  • The MELODIC directory does not contain the required ‘stats’ folder. Mixture modeling on the Z-statistical maps will be run.
    Step 2) Automatic classification of the components
  • registering the spatial maps to MNI
  • extracting the CSF & Edge fraction features
    • The spatial map of component 1 is empty. Please check!
    • The spatial map of component 2 is empty. Please check!
    • The spatial map of component 3 is empty. Please check!
    • The spatial map of component 4 is empty. Please check!
    • The spatial map of component 5 is empty. Please check!
    • The spatial map of component 6 is empty. Please check!
    • The spatial map of component 7 is empty. Please check!
    • The spatial map of component 8 is empty. Please check!
    • The spatial map of component 9 is empty. Please check!
    • The spatial map of component 10 is empty. Please check!
    • The spatial map of component 11 is empty. Please check!
    • The spatial map of component 12 is empty. Please check!
    • The spatial map of component 13 is empty. Please check!
    • The spatial map of component 14 is empty. Please check!
    • The spatial map of component 15 is empty. Please check!
    • The spatial map of component 16 is empty. Please check!
    • The spatial map of component 17 is empty. Please check!
    • The spatial map of component 18 is empty. Please check!
    • The spatial map of component 19 is empty. Please check!
    • The spatial map of component 20 is empty. Please check!
    • The spatial map of component 21 is empty. Please check!
    • The spatial map of component 22 is empty. Please check!
    • The spatial map of component 23 is empty. Please check!
    • The spatial map of component 24 is empty. Please check!
    • The spatial map of component 25 is empty. Please check!
    • The spatial map of component 26 is empty. Please check!
    • The spatial map of component 27 is empty. Please check!
    • The spatial map of component 28 is empty. Please check!
    • The spatial map of component 29 is empty. Please check!
    • The spatial map of component 30 is empty. Please check!
    • The spatial map of component 31 is empty. Please check!
    • The spatial map of component 32 is empty. Please check!
    • The spatial map of component 33 is empty. Please check!
    • The spatial map of component 34 is empty. Please check!
    • The spatial map of component 35 is empty. Please check!
    • The spatial map of component 36 is empty. Please check!
    • The spatial map of component 37 is empty. Please check!
    • The spatial map of component 38 is empty. Please check!
    • The spatial map of component 39 is empty. Please check!
    • The spatial map of component 40 is empty. Please check!
    • The spatial map of component 41 is empty. Please check!
    • The spatial map of component 42 is empty. Please check!
    • The spatial map of component 43 is empty. Please check!
    • The spatial map of component 44 is empty. Please check!
    • The spatial map of component 45 is empty. Please check!
    • The spatial map of component 46 is empty. Please check!
    • The spatial map of component 47 is empty. Please check!
    • The spatial map of component 48 is empty. Please check!
    • The spatial map of component 49 is empty. Please check!
    • The spatial map of component 50 is empty. Please check!
    • The spatial map of component 51 is empty. Please check!
    • The spatial map of component 52 is empty. Please check!
    • The spatial map of component 53 is empty. Please check!
    • The spatial map of component 54 is empty. Please check!
    • The spatial map of component 55 is empty. Please check!
    • The spatial map of component 56 is empty. Please check!
    • The spatial map of component 57 is empty. Please check!
    • The spatial map of component 58 is empty. Please check!
    • The spatial map of component 59 is empty. Please check!
    • The spatial map of component 60 is empty. Please check!
    • The spatial map of component 61 is empty. Please check!
    • The spatial map of component 62 is empty. Please check!
    • The spatial map of component 63 is empty. Please check!
    • The spatial map of component 64 is empty. Please check!
    • The spatial map of component 65 is empty. Please check!
    • The spatial map of component 66 is empty. Please check!
    • The spatial map of component 67 is empty. Please check!
    • The spatial map of component 68 is empty. Please check!
    • The spatial map of component 69 is empty. Please check!
    • The spatial map of component 70 is empty. Please check!
    • The spatial map of component 71 is empty. Please check!
    • The spatial map of component 72 is empty. Please check!
    • The spatial map of component 73 is empty. Please check!
    • The spatial map of component 74 is empty. Please check!
    • The spatial map of component 75 is empty. Please check!
    • The spatial map of component 76 is empty. Please check!
    • The spatial map of component 77 is empty. Please check!
    • The spatial map of component 78 is empty. Please check!
    • The spatial map of component 79 is empty. Please check!
    • The spatial map of component 80 is empty. Please check!
    • The spatial map of component 81 is empty. Please check!
    • The spatial map of component 82 is empty. Please check!
    • The spatial map of component 83 is empty. Please check!
    • The spatial map of component 84 is empty. Please check!
    • The spatial map of component 85 is empty. Please check!
    • The spatial map of component 86 is empty. Please check!
    • The spatial map of component 87 is empty. Please check!
    • The spatial map of component 88 is empty. Please check!
    • The spatial map of component 89 is empty. Please check!
    • The spatial map of component 90 is empty. Please check!
    • The spatial map of component 91 is empty. Please check!
    • The spatial map of component 92 is empty. Please check!
    • The spatial map of component 93 is empty. Please check!
    • The spatial map of component 94 is empty. Please check!
    • The spatial map of component 95 is empty. Please check!
    • The spatial map of component 96 is empty. Please check!
    • The spatial map of component 97 is empty. Please check!
    • The spatial map of component 98 is empty. Please check!
    • The spatial map of component 99 is empty. Please check!
    • The spatial map of component 100 is empty. Please check!
    • The spatial map of component 101 is empty. Please check!
    • The spatial map of component 102 is empty. Please check!
    • The spatial map of component 103 is empty. Please check!
    • The spatial map of component 104 is empty. Please check!
  • extracting the Maximum RP correlation feature
  • extracting the High-frequency content feature
  • classification
    Standard error:
    Traceback (most recent call last):
    File “/opt/ICA-AROMA/ICA_AROMA.py”, line 211, in
    motionICs = aromafunc.classification(outDir, maxRPcorr, edgeFract, HFC, csfFract)
    File “/opt/ICA-AROMA/ICA_AROMA_functions.py”, line 487, in classification
    if len(motionICs) != 0:
    TypeError: len() of unsized object
2

The first error seems like a filesystem issue. Is it reproducible (as in if you run the same subject again does it break in the same way)?

The ICA AROMA bug is a known issue - you can track our progress on it here: https://github.com/poldracklab/fmriprep/issues/981

BTW - did you run bids-validator on the input data? Your TR is exactly 1.0s which is unusual…

3

Thanks for your reply!
I did run bids-validator on the input data, and there were no error - I double checked my protocol anyway and my TR is indeed exactly 1.0s (I am using multiband).

Re- the first error, I re-ran the subject and it ran through without error, which probably means it was indeed a filesystem issue. However, I am now getting a mask problem for one of the functional runs, whereby the brainmask file for that run has a big hole in it, even though both the preproc BOLD and the AROMA-denoised timeseries look just fine!

This is the fmriprep output showing the hole in the mask:

sub-004_run-08_brainmask_error
sub-004_run-08_brainmask_error.jpg1906×898 249 KB

Any idea why this could be happening? At what point is the functional mask being created? I am guessing this could be a problem for the confounds estimation and ICA, since those will only be estimated within the mask? What would be the best way to re-run that part?

As for the ICA-AROMA bug, good to know that this was known - I will definitely track the process.
I also noticed another type of error at the time of the ICA, but not sure yet if this is reproducible or not, so I will try to run that subject again:

Node: fmriprep_wf.single_subject_002_wf.func_preproc_task_ObsLearn_run_06_wf.bold_confounds_wf.ica_aroma_wf.ica_aroma
Working directory: /home/ccharpen/ObsLearn/prepdata/work/fmriprep_wf/single_subject_002_wf/func_preproc_task_ObsLearn_run_06_wf/bold_confounds_wf/ica_aroma_wf/ica_aroma

Node inputs:

TR =
args =
compress_report = auto
denoise_type = nonaggr
dim =
environ = {}
feat_dir =
fnirt_warp_file =
generate_report = True
ignore_exception = False
in_file = /home/ccharpen/ObsLearn/prepdata/work/fmriprep_wf/single_subject_002_wf/func_preproc_task_ObsLearn_run_06_wf/bold_confounds_wf/ica_aroma_wf/smooth/vol0000_xform-00000_merged_smooth.nii.gz
mask =
mat_file =
melodic_dir = /home/ccharpen/ObsLearn/prepdata/work/fmriprep_wf/single_subject_002_wf/func_preproc_task_ObsLearn_run_06_wf/bold_confounds_wf/ica_aroma_wf/melodic
motion_parameters = /home/ccharpen/ObsLearn/prepdata/work/fmriprep_wf/single_subject_002_wf/func_preproc_task_ObsLearn_run_06_wf/bold_hmc_wf/normalize_motion/motion_params.txt
out_dir =
out_report = ica_aroma_reportlet.svg
report_mask = /home/ccharpen/ObsLearn/prepdata/work/fmriprep_wf/single_subject_002_wf/func_preproc_task_ObsLearn_run_06_wf/bold_mni_trans_wf/mask_mni_tfm/ref_image_corrected_trans_corrected_brain_mask_maths_trans.nii.gz
terminal_output =

Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/multiproc.py”, line 68, in run_node
result[‘result’] = node.run(updatehash=updatehash)
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py”, line 487, in run
result = self._run_interface(execute=True)
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py”, line 571, in _run_interface
return self._run_command(execute)
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py”, line 650, in _run_command
result = self._interface.run(cwd=outdir)
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base/core.py”, line 516, in run
runtime = self._run_interface(runtime)
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/interfaces/report_base.py”, line 74, in _run_interface
self._generate_report()
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/interfaces/segmentation.py”, line 152, in _generate_report
noise_components_file=self._noise_components_file
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/viz/utils.py”, line 543, in plot_melodic_components
noise_components = [int© for c in cf.read().split(",")]
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/viz/utils.py”, line 543, in
noise_components = [int© for c in cf.read().split(",")]
ValueError: invalid literal for int() with base 10: ‘’

4

The mask/report issue was resolved in the latest version of FMRIPREP - please try 1.0.8

5

Hi @ccharpen, could you try the poldracklab/fmriprep:1.0.12-dev0 docker image and check whether the ICA-AROMA errors are fixed for you?

Thanks very much!

6

Hi! So I just tried running 1.0.12-dev0 on one subject for which I had the two ICA-AROMA errors (i.e. “TypeError: len() of unsized object” for one of the runs and “ValueError: invalid literal for int() with base 10: ‘’” for 2 of the runs).

I now have only one error left (“ValueError: invalid literal for int() with base 10: ‘’” for one of the runs). So I guess this is definitely a move in the right direction, as the first error is gone; however the second one still remains for one of the runs. Here is what I get in the log:

Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/multiproc.py”, line 68, in run_node
result[‘result’] = node.run(updatehash=updatehash)
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py”, line 479, in run
result = self._run_interface(execute=True)
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py”, line 563, in _run_interface
return self._run_command(execute)
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py”, line 643, in _run_command
result = self._interface.run(cwd=outdir)
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base/core.py”, line 516, in run
runtime = self._run_interface(runtime)
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/interfaces/report_base.py”, line 69, in _run_interface
self._generate_report()
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/interfaces/segmentation.py”, line 153, in _generate_report
noise_components_file=self._noise_components_file
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/viz/utils.py”, line 543, in plot_melodic_components
noise_components = [int© for c in cf.read().split(",")]
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/viz/utils.py”, line 543, in
noise_components = [int© for c in cf.read().split(",")]
ValueError: invalid literal for int() with base 10: ‘’

I will try to re-run that same subject to see if it reproduces, and also another subject or two who also showed errors.

Thanks!

7

Okay, we released version 1.0.12 and that one should fix for this problem. Please let us know if it was indeed solved.

8

Ok, good to know - I will try the new version.

In the meantime, I also noticed that the classification definitely fails for some subjects/runs - as in many noise components classified as signals OR (more problematic) signal classified as noise.

See below for an example of each - this is from the same subject, two different runs:

I am wondering if something is wrong with the classifier… Any chance that this might be fixed in the new version as well?

Thanks!

9

This particular answer (and probably the whole thread) will probably be of your interest.

EDIT: Let’s keep this conversation there, if you don’t mind :slight_smile: