ICA-AROMA error on fmriprep 20.2.0 LTS

dlevitas · October 6, 2020, 2:33pm

I was recently re-processing my data (BIDS validated) using this command

unset PYTHONPATH; singularity run -B /N/slate/$(whoami):/N/slate/$(whoami) /N/dcwan/projects/irf/containers/fmriprep-20.2.0.simg \
		$bids_root_dir $bids_root_dir/derivatives						\
		participant														\
		--skip-bids-validation 											\
		--md-only-boilerplate											\
		--participant-label $s 											\
		--fs-license-file $FREESURFER_HOME/license.txt 					\
		--ignore slicetiming 											\
		--use-aroma 													\
		--aroma-melodic-dimensionality 100 								\
		--output-spaces T1w fsnative fsaverage:den-10k MNI152NLin6Asym:res-2	\
		--nthreads $nthreads											\
		--fd-spike-threshold 1.0										\
		--dvars-spike-threshold 5 										\
		--stop-on-first-crash 											\
		--resource-monitor 												\
		--low-mem 														\
		--mem_mb $mem_mb 												\
		--use-plugin $bids_root_dir/fmriprep_plugin_${s}.yml 			\
		--verbose 														\
		-w /N/slate/$(whoami)

and encountered this error:

Node ica_aroma failed to run on host c35.
201001-23:45:59,398 nipype.workflow ERROR:
	 Saving crash info to /N/slate/dlevitas/Dan_STD/derivatives/fmriprep/sub-1000/log/20201001-230303_b254c475-892d-420e-9625-28302e07d2de/crash-20201001-234559-dlevitas-ica_aroma-47e99a56-0a43-4119-a1cf-4a0826563f95.txt
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/legacymultiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 814, in _run_interface
    self.raise_exception(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 745, in raise_exception
    ).format(**runtime.dictcopy())
RuntimeError: Command:
ICA_AROMA.py -tr 0.420 -np -den nonaggr -i /N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/ica_aroma_wf/smooth/vol0000_xform-00000_merged_smooth.nii.gz -m /N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/bold_std_trans_wf/_std_target_MNI152NLin6Asym.res2/mask_std_tfm/ref_bold_corrected_brain_mask_maths_trans.nii.gz -meldir /N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/ica_aroma_wf/melodic -mc /N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/bold_hmc_wf/normalize_motion/motion_params.txt -o /N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/ica_aroma_wf/ica_aroma/out
Standard output:

------------------------------- RUNNING ICA-AROMA ------------------------------- 
--------------- 'ICA-based Automatic Removal Of Motion Artifacts' --------------- 

Step 1) MELODIC
  - The existing/specified MELODIC directory will be used.
Standard error:
Traceback (most recent call last):
  File "/opt/ICA-AROMA/ICA_AROMA.py", line 204, in <module>
    aromafunc.runICA(fslDir, inFile, outDir, melDir, mask, dim, TR)
  File "/opt/ICA-AROMA/ICA_AROMA_functions.py", line 103, in runICA
    lenIC = int(float(subprocess.getoutput(cmd)))
ValueError: could not convert string to float: "** ERROR (nifti_image_read): short header read '/N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/ica_aroma_wf/ica_aroma/out/melodic.ica/stats/thresh_zstat76.nii.gz'\n** ERROR: nifti_image_open(/N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/ica_aroma_wf/ica_aroma/out/melodic.ica/stats/thresh_zstat76): bad header info\nERROR: failed to open file /N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/ica_aroma_wf/ica_aroma/out/melodic.ica/stats/thresh_zstat76\nERROR: Could not open file"
Return code: 1

201001-23:45:59,424 nipype.workflow CRITICAL:
	 fMRIPrep failed: Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/legacymultiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 814, in _run_interface
    self.raise_exception(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 745, in raise_exception
    ).format(**runtime.dictcopy())
RuntimeError: Command:
ICA_AROMA.py -tr 0.420 -np -den nonaggr -i /N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/ica_aroma_wf/smooth/vol0000_xform-00000_merged_smooth.nii.gz -m /N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/bold_std_trans_wf/_std_target_MNI152NLin6Asym.res2/mask_std_tfm/ref_bold_corrected_brain_mask_maths_trans.nii.gz -meldir /N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/ica_aroma_wf/melodic -mc /N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/bold_hmc_wf/normalize_motion/motion_params.txt -o /N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/ica_aroma_wf/ica_aroma/out
Standard output:

------------------------------- RUNNING ICA-AROMA ------------------------------- 
--------------- 'ICA-based Automatic Removal Of Motion Artifacts' --------------- 

Step 1) MELODIC
  - The existing/specified MELODIC directory will be used.
Standard error:
Traceback (most recent call last):
  File "/opt/ICA-AROMA/ICA_AROMA.py", line 204, in <module>
    aromafunc.runICA(fslDir, inFile, outDir, melDir, mask, dim, TR)
  File "/opt/ICA-AROMA/ICA_AROMA_functions.py", line 103, in runICA
    lenIC = int(float(subprocess.getoutput(cmd)))
ValueError: could not convert string to float: "** ERROR (nifti_image_read): short header read '/N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/ica_aroma_wf/ica_aroma/out/melodic.ica/stats/thresh_zstat76.nii.gz'\n** ERROR: nifti_image_open(/N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/ica_aroma_wf/ica_aroma/out/melodic.ica/stats/thresh_zstat76): bad header info\nERROR: failed to open file /N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/ica_aroma_wf/ica_aroma/out/melodic.ica/stats/thresh_zstat76\nERROR: Could not open file"
Return code: 1

Our HPC recently switched data storage (slate), so it may well be something to do with that; however, I didn’t any binding errors in the logs. The file permissions are -rw-r--r-- 1 and I’m the owner of them. I haven’t encountered this error in the past; however, with the new fmriprep version and data storage system on our HPC, I’m unsure which of these (or both) is potentially causing the issue.

mgxd · October 7, 2020, 3:18pm

Yes, short header errors are typically encountered when either:

your working directory is out of space
you’re on a network filesystem and the synchronization took too long.

My guess is for your case, it’s the latter.

m.pereira · January 22, 2021, 3:41pm

Hi @dlevitas, I have the same issue. Could you please share whether and how you fixed it? Thanks in advance!

dlevitas · January 22, 2021, 4:15pm

Hi @m.pereira

If memory serves me well, this was a storage issue. My institution’s HPC had recently changed its allocated storage, and I had less storage space that before. I had to request additional space from an HPC sys admin, and I believe that resolved this specific issue of mine.

Dan