fMRI prep: ERROR: EDIT_full_copy: can't malloc 3115008 bytes for new sub-brick 72

Software Support
1

Summary of what happened:

Command used (and if a helper script was used, a link to the helper script or the command generated):

bids_dir=/home/fs0/scratch/fmriprep/pRFmapping-BIDS
out_dir=/home/fs0/scratch/fmriprep/pRFmapping-deriv
work_dir=/home/fs0/scratch/fmriprep/work
fs_dir=$work_dir/license.txt
programme_DIR=/home/fs0/scratch/pRF/software
SINGULARITYENV_TEMPLATEFLOW_HOME=/home/fs0/.cache/templateflow
export SINGULARITYENV_TEMPLATEFLOW_HOME

singularity run -e --containall -B ${bids_dir},${out_dir},${work_dir},${SINGULARITYENV_TEMPLATEFLOW_HOME},${fs_dir} \
/$programme_DIR/fmriprep-23.2.0.simg \
-w $work_dir \
--fs-license-file $fs_dir \
/$bids_dir $out_dir/ participant \
--participant-label sub-01 --output-spaces fsnative --nthreads 15 --omp-nthreads 15 --use-syn-sdc

Version:

23.2.0

Environment (Docker, Singularity / Apptainer, custom installation):

Singularity

Data formatted according to a validatable standard? Please provide the output of the validator:

Yes - BIDS

bids-validator@1.14.0
(node:18) Warning: Closing directory handle on garbage collection
(Use `node --trace-warnings ...` to show where the warning was created)
	1: [WARN] The recommended file /README is missing. See Section 03 (Modality agnostic files) of the BIDS specification. (code: 101 - README_FILE_MISSING)

	Please visit https://neurostars.org/search?q=README_FILE_MISSING for existing conversations about this issue.

        Summary:                Available Tasks:        Available Modalities: 
        45 Files, 5.45GB        motion                  MRI                   
        2 - Subjects            retino                                        
        1 - Session

Relevant log outputs (up to 20 lines):

240304-14:42:49,445 nipype.workflow INFO:
	 [Node] Executing "mcflirt" <nipype.interfaces.fsl.preprocess.MCFLIRT>
240304-14:42:49,622 nipype.workflow INFO:
	 [Node] Finished "gen_avg", elapsed time 19.252861s.
240304-14:42:49,720 nipype.workflow INFO:
	 [Node] Finished "slice_timing_correction", elapsed time 4.8994s.
240304-14:42:49,720 nipype.workflow WARNING:
	 Storing result file without outputs
240304-14:42:49,724 nipype.workflow WARNING:
	 [Node] Error on "fmriprep_23_2_wf.sub_01_wf.bold_ses_1_task_retino_run_1_wf.bold_native_wf.bold_stc_wf.slice_timing_correction" (/home/fs0/hwillis/scratch/Vision_Lab/pRF/fmriprep/work/fmriprep_23_2_wf/sub_01_wf/bold_ses_1_task_retino_run_1_wf/bold_native_wf/bold_stc_wf/slice_timing_correction)
240304-14:42:49,733 nipype.workflow INFO:
	 [Node] Finished "slice_timing_correction", elapsed time 5.1988900000000005s.
240304-14:42:49,733 nipype.workflow WARNING:
	 Storing result file without outputs
240304-14:42:49,736 nipype.workflow WARNING:
	 [Node] Error on "fmriprep_23_2_wf.sub_01_wf.bold_ses_1_task_motion_run_1_wf.bold_native_wf.bold_stc_wf.slice_timing_correction" (/home/fs0/hwillis/scratch/Vision_Lab/pRF/fmriprep/work/fmriprep_23_2_wf/sub_01_wf/bold_ses_1_task_motion_run_1_wf/bold_native_wf/bold_stc_wf/slice_timing_correction)
240304-14:42:49,785 nipype.workflow INFO:
	 [Node] Finished "slice_timing_correction", elapsed time 6.682803s.
240304-14:42:49,785 nipype.workflow WARNING:
	 Storing result file without outputs
240304-14:42:49,788 nipype.workflow WARNING:
	 [Node] Error on "fmriprep_23_2_wf.sub_01_wf.bold_ses_1_task_motion_run_3_wf.bold_native_wf.bold_stc_wf.slice_timing_correction" (/home/fs0/hwillis/scratch/Vision_Lab/pRF/fmriprep/work/fmriprep_23_2_wf/sub_01_wf/bold_ses_1_task_motion_run_3_wf/bold_native_wf/bold_stc_wf/slice_timing_correction)
240304-14:42:49,946 nipype.workflow INFO:
	 [Node] Finished "gen_avg", elapsed time 16.587865999999998s.
240304-14:42:50,569 nipype.workflow ERROR:
	 Node slice_timing_correction failed to run on host jalapeno.cluster.fmrib.ox.ac.uk.
240304-14:42:50,576 nipype.workflow ERROR:
	 Saving crash info to /home/fs0/hwillis/scratch/Vision_Lab/pRF/fmriprep/pRFmapping-deriv/sub-01/log/20240304-144047_16b66ea5-a25e-4ec6-a3d8-4cd4d105be17/crash-20240304-144250-hwillis-slice_timing_correction-c9089df5-b178-430c-a92d-0eef207c4db0.txt
Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node slice_timing_correction.

Cmdline:
	3dTshift -ignore 2 -prefix sub-01_ses-1_task-motion_run-1_bold_tshift.nii.gz -tpattern @slice_timing.1D -TR 1.05s -tzero 0.472 /home/fs0/hwillis/scratch/Vision_Lab/pRF/fmriprep/work/fmriprep_23_2_wf/sub_01_wf/bold_ses_1_task_motion_run_1_wf/bold_native_wf/bold_stc_wf/slice_timing_correction/sub-01_ses-1_task-motion_run-1_bold.nii.gz
Stdout:

Stderr:
	++ 3dTshift: AFNI version=AFNI_24.0.00 (Jan  7 2024) [64-bit]
	** AFNI converts NIFTI_datatype=512 (UINT16) in file /home/fs0/hwillis/scratch/Vision_Lab/pRF/fmriprep/work/fmriprep_23_2_wf/sub_01_wf/bold_ses_1_task_motion_run_1_wf/bold_native_wf/bold_stc_wf/slice_timing_correction/sub-01_ses-1_task-motion_run-1_bold.nii.gz to FLOAT32
	     Warnings of this type will be muted for this session.
	     Set AFNI_NIFTI_TYPE_WARN to YES to see them all, NO to see none.
	*+ WARNING:   If you are performing spatial transformations on an oblique dset,
	  such as /home/fs0/hwillis/scratch/Vision_Lab/pRF/fmriprep/work/fmriprep_23_2_wf/sub_01_wf/bold_ses_1_task_motion_run_1_wf/bold_native_wf/bold_stc_wf/slice_timing_correction/sub-01_ses-1_task-motion_run-1_bold.nii.gz,
	  or viewing/combining it with volumes of differing obliquity,
	  you should consider running: 
	     3dWarp -deoblique 
	  on this and  other oblique datasets in the same session.
	 See 3dWarp -help for details.
	++ Oblique dataset:/home/fs0/hwillis/scratch/Vision_Lab/pRF/fmriprep/work/fmriprep_23_2_wf/sub_01_wf/bold_ses_1_task_motion_run_1_wf/bold_native_wf/bold_stc_wf/slice_timing_correction/sub-01_ses-1_task-motion_run-1_bold.nii.gz is 5.300013 degrees from plumb.
	** ERROR: EDIT_full_copy: can't malloc 3115008 bytes for new sub-brick 59
	 +    Dataset /home/fs0/hwillis/scratch/Vision_Lab/pRF/fmriprep/work/fmriprep_23_2_wf/sub_01_wf/bold_ses_1_task_motion_run_1_wf/bold_native_wf/bold_stc_wf/slice_timing_correction/sub-01_ses-1_task-motion_run-1_bold.nii.gz
	** FATAL ERROR: Can't copy input dataset '/home/fs0/hwillis/scratch/Vision_Lab/pRF/fmriprep/work/fmriprep_23_2_wf/sub_01_wf/bold_ses_1_task_motion_run_1_wf/bold_native_wf/bold_stc_wf/slice_timing_correction/sub-01_ses-1_task-motion_run-1_bold.nii.gz'
	** Program compile date = Jan  7 2024
Traceback:
	Traceback (most recent call last):
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 453, in aggregate_outputs
	    setattr(outputs, key, val)
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
	    value = super(File, self).validate(objekt, name, value, return_pathlike=True)
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
	    self.error(objekt, name, str(value))
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/traits/base_trait_handler.py", line 74, in error
	    raise TraitError(
	traits.trait_errors.TraitError: The 'out_file' trait of a TShiftOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/home/fs0/hwillis/scratch/Vision_Lab/pRF/fmriprep/work/fmriprep_23_2_wf/sub_01_wf/bold_ses_1_task_motion_run_1_wf/bold_native_wf/bold_stc_wf/slice_timing_correction/sub-01_ses-1_task-motion_run-1_bold_tshift.nii.gz' <class 'str'> was specified.

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 400, in run
	    outputs = self.aggregate_outputs(runtime)
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 460, in aggregate_outputs
	    raise FileNotFoundError(msg)
	FileNotFoundError: No such file or directory '/home/fs0/hwillis/scratch/Vision_Lab/pRF/fmriprep/work/fmriprep_23_2_wf/sub_01_wf/bold_ses_1_task_motion_run_1_wf/bold_native_wf/bold_stc_wf/slice_timing_correction/sub-01_ses-1_task-motion_run-1_bold_tshift.nii.gz' for output 'out_file' of a TShift interface

Screenshots / relevant information:

There have been multiple errors at different stages. Firstly with mounting the folders (which is why they are mounted separately), and now with completing all the processing. Analysis is being run on a HPC.

2

Hi @hannawillis and welcome to neurostars!

How much memory are you devoting to the job? And have you confirmed you have a lot of storage in your work_dir? Is this error (or something similar) present across all subjects? Are subjects being processed in serial or are there multiple subjects running on the same working directory at one time?

Best,
Steven

3

Hello Steven,

Thanks for your help!
How would I check the memory devoted to the job? - this isn’t something I’ve specifically set.
Yes, there is lots of space in the work directory (101.6 GiB available in my scratch - where this is being run)
Yes, it happens across subjects.
The subjects are being processed serially.

Hanna

4

Hi @hannawillis,

How are you submitting job (e.g., an sbatch job array on the SLURM job scheduler, which many HPCs user)? For SLURM, for example, memory would be set in the SBATCH header at the top of your script.

Best,
Steven

5

Hello Steven,

I’ve just submitted it on the HPC but not to a queue, so I wouldn’t normally set the memory allocation.

Hanna

6

Hi @hannawillis,

If you are just using a head node / login node memory, that is likely very limited (and can also cause troubles for other users on the cluster who login on the same node!). You should submit to a job scheduler where you can allocate specific memory (I would recommend starting with something big like 32GB, then seeing how much one job took after the fact, and using that to find a good memory setting to use for the rest of your jobs). In SLURM this is done with seff <jobID>.

Best,
Steven