ICA-AROMA error on fmriprep 20.2.0 LTS

I was recently re-processing my data (BIDS validated) using this command

unset PYTHONPATH; singularity run -B /N/slate/$(whoami):/N/slate/$(whoami) /N/dcwan/projects/irf/containers/fmriprep-20.2.0.simg \
		$bids_root_dir $bids_root_dir/derivatives						\
		participant														\
		--skip-bids-validation 											\
		--md-only-boilerplate											\
		--participant-label $s 											\
		--fs-license-file $FREESURFER_HOME/license.txt 					\
		--ignore slicetiming 											\
		--use-aroma 													\
		--aroma-melodic-dimensionality 100 								\
		--output-spaces T1w fsnative fsaverage:den-10k MNI152NLin6Asym:res-2	\
		--nthreads $nthreads											\
		--fd-spike-threshold 1.0										\
		--dvars-spike-threshold 5 										\
		--stop-on-first-crash 											\
		--resource-monitor 												\
		--low-mem 														\
		--mem_mb $mem_mb 												\
		--use-plugin $bids_root_dir/fmriprep_plugin_${s}.yml 			\
		--verbose 														\
		-w /N/slate/$(whoami)

and encountered this error:

Node ica_aroma failed to run on host c35.
201001-23:45:59,398 nipype.workflow ERROR:
	 Saving crash info to /N/slate/dlevitas/Dan_STD/derivatives/fmriprep/sub-1000/log/20201001-230303_b254c475-892d-420e-9625-28302e07d2de/crash-20201001-234559-dlevitas-ica_aroma-47e99a56-0a43-4119-a1cf-4a0826563f95.txt
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/legacymultiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 814, in _run_interface
    self.raise_exception(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 745, in raise_exception
    ).format(**runtime.dictcopy())
RuntimeError: Command:
ICA_AROMA.py -tr 0.420 -np -den nonaggr -i /N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/ica_aroma_wf/smooth/vol0000_xform-00000_merged_smooth.nii.gz -m /N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/bold_std_trans_wf/_std_target_MNI152NLin6Asym.res2/mask_std_tfm/ref_bold_corrected_brain_mask_maths_trans.nii.gz -meldir /N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/ica_aroma_wf/melodic -mc /N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/bold_hmc_wf/normalize_motion/motion_params.txt -o /N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/ica_aroma_wf/ica_aroma/out
Standard output:

------------------------------- RUNNING ICA-AROMA ------------------------------- 
--------------- 'ICA-based Automatic Removal Of Motion Artifacts' --------------- 

Step 1) MELODIC
  - The existing/specified MELODIC directory will be used.
Standard error:
Traceback (most recent call last):
  File "/opt/ICA-AROMA/ICA_AROMA.py", line 204, in <module>
    aromafunc.runICA(fslDir, inFile, outDir, melDir, mask, dim, TR)
  File "/opt/ICA-AROMA/ICA_AROMA_functions.py", line 103, in runICA
    lenIC = int(float(subprocess.getoutput(cmd)))
ValueError: could not convert string to float: "** ERROR (nifti_image_read): short header read '/N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/ica_aroma_wf/ica_aroma/out/melodic.ica/stats/thresh_zstat76.nii.gz'\n** ERROR: nifti_image_open(/N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/ica_aroma_wf/ica_aroma/out/melodic.ica/stats/thresh_zstat76): bad header info\nERROR: failed to open file /N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/ica_aroma_wf/ica_aroma/out/melodic.ica/stats/thresh_zstat76\nERROR: Could not open file"
Return code: 1

201001-23:45:59,424 nipype.workflow CRITICAL:
	 fMRIPrep failed: Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/legacymultiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 814, in _run_interface
    self.raise_exception(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 745, in raise_exception
    ).format(**runtime.dictcopy())
RuntimeError: Command:
ICA_AROMA.py -tr 0.420 -np -den nonaggr -i /N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/ica_aroma_wf/smooth/vol0000_xform-00000_merged_smooth.nii.gz -m /N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/bold_std_trans_wf/_std_target_MNI152NLin6Asym.res2/mask_std_tfm/ref_bold_corrected_brain_mask_maths_trans.nii.gz -meldir /N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/ica_aroma_wf/melodic -mc /N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/bold_hmc_wf/normalize_motion/motion_params.txt -o /N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/ica_aroma_wf/ica_aroma/out
Standard output:

------------------------------- RUNNING ICA-AROMA ------------------------------- 
--------------- 'ICA-based Automatic Removal Of Motion Artifacts' --------------- 

Step 1) MELODIC
  - The existing/specified MELODIC directory will be used.
Standard error:
Traceback (most recent call last):
  File "/opt/ICA-AROMA/ICA_AROMA.py", line 204, in <module>
    aromafunc.runICA(fslDir, inFile, outDir, melDir, mask, dim, TR)
  File "/opt/ICA-AROMA/ICA_AROMA_functions.py", line 103, in runICA
    lenIC = int(float(subprocess.getoutput(cmd)))
ValueError: could not convert string to float: "** ERROR (nifti_image_read): short header read '/N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/ica_aroma_wf/ica_aroma/out/melodic.ica/stats/thresh_zstat76.nii.gz'\n** ERROR: nifti_image_open(/N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/ica_aroma_wf/ica_aroma/out/melodic.ica/stats/thresh_zstat76): bad header info\nERROR: failed to open file /N/slate/dlevitas/fmriprep_wf/single_subject_1000_wf/func_preproc_task_std_run_03_wf/ica_aroma_wf/ica_aroma/out/melodic.ica/stats/thresh_zstat76\nERROR: Could not open file"
Return code: 1

Our HPC recently switched data storage (slate), so it may well be something to do with that; however, I didn’t any binding errors in the logs. The file permissions are -rw-r--r-- 1 and I’m the owner of them. I haven’t encountered this error in the past; however, with the new fmriprep version and data storage system on our HPC, I’m unsure which of these (or both) is potentially causing the issue.

Yes, short header errors are typically encountered when either:

  1. your working directory is out of space
  2. you’re on a network filesystem and the synchronization took too long.

My guess is for your case, it’s the latter.

Hi @dlevitas, I have the same issue. Could you please share whether and how you fixed it? Thanks in advance!

Hi @m.pereira

If memory serves me well, this was a storage issue. My institution’s HPC had recently changed its allocated storage, and I had less storage space that before. I had to request additional space from an HPC sys admin, and I believe that resolved this specific issue of mine.

Dan

1 Like

MANAGED BY INCF