indexed_gzip.indexed_gzip.ZranError: zran_read returned error: ZRAN_READ_FAIL

ejcorn · June 3, 2024, 5:50pm

Summary of what happened:

fMRIprep error – thank you in advance for your help and apologies for the complexity of this one! I think it may be related to memory or using an external hard drive.

Command used (and if a helper script was used, a link to the helper script or the command generated):

#!/bin/bash
set -euxo pipefail

export FREESURFER_HOME=/Applications/freesurfer/7.2.0
export HOME_DIR=/Volumes/Seagate/
#export HOME_DIR=/Users/Eli/Documents/AEON/
mkdir -p ${HOME_DIR}work
fmriprep-docker ${HOME_DIR}BIDS/ ${HOME_DIR}BIDS/derivatives/ participant -w ${HOME_DIR}work --verbose \
	--mem-mb 15500 --low-mem --fs-license-file $FREESURFER_HOME/license.txt --participant_label AEON108

Version:

23.2.0

Environment (Docker, Singularity / Apptainer, custom installation):

Docker

Data formatted according to a validatable standard? Please provide the output of the validator:

	 Making sure the input data is BIDS compliant (warnings can be ignored in most cases).
bids-validator@1.14.0
(node:9) Warning: Closing directory handle on garbage collection
(Use `node --trace-warnings ...` to show where the warning was created)
	1: [WARN] The recommended file /README is very small. Please consider expanding it with additional information about the dataset. (code: 213 - README_FILE_SMALL)
		./README

	Please visit https://neurostars.org/search?q=README_FILE_SMALL for existing conversations about this issue.

        Summary:                   Available Tasks:        Available Modalities: 
        1618 Files, 76.92GB                                MRI                   
        11 - Subjects                                                            
        10 - Sessions                                                            


	If you have any questions, please post on https://neurostars.org/tags/bids.

Relevant log outputs (up to 20 lines):

Traceback:
	Traceback (most recent call last):
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 397, in run
	    runtime = self._run_interface(runtime)
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/niworkflows/interfaces/bids.py", line 737, in _run_interface
	    new_data = orig_img.dataobj.get_unscaled()
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nibabel/arrayproxy.py", line 416, in get_unscaled
	    return self._get_unscaled(slicer=())
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nibabel/arrayproxy.py", line 376, in _get_unscaled
	    return array_from_file(
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nibabel/volumeutils.py", line 465, in array_from_file
	    n_read = infile.readinto(data_bytes)
	  File "indexed_gzip/indexed_gzip.pyx", line 804, in indexed_gzip.indexed_gzip._IndexedGzipFile.readinto
	indexed_gzip.indexed_gzip.ZranError: zran_read returned error: ZRAN_READ_FAIL (file: /scratch/fmriprep_23_2_wf/sub_AEON108_wf/bold_ses_003_task_rest_run_01_wf/bold_std_wf/_in_tuple_MNI152NLin2009cAsym.resnative/resample/sub-AEON108_ses-003_task-rest_run-01_bold_tshift_xformresampled.nii.gz)

Screenshots / relevant information:

When I run

ls work/fmriprep_23_2_wf/sub_AEON108_wf/bold_ses_003_task_rest_run_01_wf/bold_std_wf/_in_tuple_MNI152NLin2009cAsym.resnative/resample/sub-AEON108_ses-003_task-rest_run-01_bold_tshift_xformresampled.nii.gz

Then it shows that this file actually exists.

I have run this multiple times and it will complete on some of my subjects. On other subjects it will hit this error but complete processing on all but 1 run (I have 10 sessions with 4 runs each). If I run it again, it will hit this error on a different run than it did on the first time. My BIDS directory is on ExFAT formatted hard drive. I have tried using the low_mem flag or not, and I have tried running it on just one subject at a time. It always seems to get 99% complete (i.e. processes everything except for 1 or 2 runs out of 40 per subject) but I am struggling to completely process all of the data.

I’m also dealing with two other errors that seem surmountable but are a hassle, and may be helpful to know about:

Sometimes when I run the command above I get this error and fMRIprep won’t even start running. If I just run the command again then it works:

bids-validator@1.14.0
Unhandled rejection (
  reason: Error: ENOSYS: function not implemented, fstat
).

Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/bin/fmriprep", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/fmriprep/cli/run.py", line 40, in main
    parse_args()
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/fmriprep/cli/parser.py", line 828, in parse_args
    validate_input_dir(config.environment.exec_env, opts.bids_dir, opts.participant_label)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/fmriprep/utils/bids.py", line 237, in validate_input_dir
    subprocess.check_call(['bids-validator', str(bids_dir), '-c', temp.name])
  File "/opt/conda/envs/fmriprep/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['bids-validator', '/data', '-c', '/tmp/tmpgz9xaqsd.json']' returned non-zero exit status 3.
fMRIPrep: Please report errors to https://github.com/nipreps/fmriprep/issues

Sometimes when I run it again second time (under the assumption that since the error seems stochastic and possibly memory related) then I hit this error. If I delete that participant’s freesurfer directory and run it again then it works but hits the main error above. However, I think that freesurfer directory is complete because most of the runs are able to register to the normalized template and complete preprocessing.

	 fMRIPrep failed: Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/utils.py", line 94, in nodelist_runner
    result = node.run(updatehash=updatehash)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 543, in run
    write_node_report(self, result=result, is_mapnode=isinstance(self, MapNode))
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/utils.py", line 208, in write_node_report
    report_file.write_text("\n".join(lines), encoding='utf-8')
  File "/opt/conda/envs/fmriprep/lib/python3.10/pathlib.py", line 1154, in write_text
    with self.open(mode='w', encoding=encoding, errors=errors, newline=newline) as f:
  File "/opt/conda/envs/fmriprep/lib/python3.10/pathlib.py", line 1119, in open
    return self._accessor.open(self, mode, buffering, encoding, errors,
FileNotFoundError: [Errno 2] No such file or directory: '/scratch/fmriprep_23_2_wf/sub_AEON108_wf/anat_fit_wf/surface_recon_wf/autorecon_resume_wf/autorecon_surfs/mapflow/_autorecon_surfs1/_report/report.rst'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 1380, in _run_interface
    result = self._collate_results(
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 1249, in _collate_results
    for i, nresult, err in nodes:
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/utils.py", line 99, in nodelist_runner
    result = node.result
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 223, in result
    return _load_resultfile(
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/utils.py", line 293, in load_resultfile
    result = loadpkl(results_file)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/utils/filemanip.py", line 621, in loadpkl
    with pklopen(str(infile), "rb") as pkl_file:
  File "/opt/conda/envs/fmriprep/lib/python3.10/gzip.py", line 58, in open
    binary_file = GzipFile(filename, gz_mode, compresslevel)
  File "/opt/conda/envs/fmriprep/lib/python3.10/gzip.py", line 174, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
FileNotFoundError: [Errno 2] No such file or directory: '/scratch/fmriprep_23_2_wf/sub_AEON108_wf/anat_fit_wf/surface_recon_wf/autorecon_resume_wf/autorecon_surfs/mapflow/_autorecon_surfs1/result__autorecon_surfs1.pklz'

paulmccarthy · June 4, 2024, 11:29am

Hi @ejcorn, regarding the ZRAN_READ_FAIL error, can you double check that the affected file(s) is a valid compressed NIFTI image? For example, you can check that it is a compressed file with the file command - a .gz file will be reported as being of type gzip compressed data, e.g.:

$ file $FSLDIR/data/standard/MNI152_T1_2mm.nii.gz
MNI152_T1_2mm.nii.gz: gzip compressed data

whereas an uncompressed NIfTI image will be reported as being of type data. If the file is actually uncompressed, but incorrectly has a .gz file extension, this can cause many software applications to break.

You can check that it is a valid NIfTI image by loading into your favourite image viewer (e.g. FSLeyes).

ejcorn · June 4, 2024, 11:38am

Thanks for your reply!

file work/fmriprep_23_2_wf/sub_AEON108_wf/bold_ses_003_task_rest_run_01_wf/bold_std_wf/_in_tuple_MNI152NLin2009cAsym.resnative/resample/sub-AEON108_ses-003_task-rest_run-01_bold_tshift_xformresampled.nii.gz

work/fmriprep_23_2_wf/sub_AEON108_wf/bold_ses_003_task_rest_run_01_wf/bold_std_wf/_in_tuple_MNI152NLin2009cAsym.resnative/resample/sub-AEON108_ses-003_task-rest_run-01_bold_tshift_xformresampled.nii.gz: gzip compressed data, max speed, original size modulo 2^32 742560352

The images look as expected on ITK SNAP.

I just reran fMRIprep with a bids filter file to only run that session and run, and that completed. However, in the past, when I have rerun it, then it errors out on a different session and run than it did before. I have more subjects to run and expect that this will keep happening.

paulmccarthy · June 4, 2024, 12:19pm

Hmm, if the result is non-deterministic (i.e. sometimes passing, sometimes failing, on the same data) then it sounds like this could be an issue with your external hard drive. Are you processing many subjects in parallel?

I’m familiar with indexed_gzip (I’m the developer), but am not particularly familiar with fMRIprep, so perhaps others will be able to chime in.

ejcorn · June 4, 2024, 9:03pm

I have tried both one subject at a time and multiple in parallel. It gets further with one subject at a time and will error out sooner with multiple in parallel.

ejcorn · June 12, 2024, 11:54am

Hi - just following up on this issue as I am still running into the above error. Do you think reformatting the hard drive from ExFAT to APFS would help?

paulmccarthy · June 12, 2024, 12:54pm

Sorry, I don’t really know. Although I would personally avoid using a Windows file system format when working on macOS or Docker/Linux. Many Linux tools expect a case-sensitive file system, for instance.