Summary of what happened:
While running fMRIprep in Singularity, I get an error on two subjects related to the inu_n4 process. I am using fMRIprep v23.2.1 and Freesurfer v7.3.2.
Command used (and if a helper script was used, a link to the helper script or the command generated):
singularity run \
--cleanenv \
--containall \
-B ${TEMPLATEFLOW_DIR}:/opt/templateflow \
-B ${BIDS_ROOT_DIR}:/bids \
-B ${DERIVATIVES_DIR}:/out \
-B ${WORK_DIR}:/work \
-B ${FS_LICENSE}:/license.txt \
$fmriprep_path \
/bids \
/out \
participant \
--participant-label $subj \
--skip-bids-validation \
--fs-license-file $BIDS_ROOT_DIR/derivatives/freesurfer_license.txt \
--work-dir $WORK_DIR \
--mem_mb 80000 \
--nprocs 16 \
--omp-nthreads 8 \
--longitudinal \
--bids-filter-file $BIDS_ROOT_DIR/code/bids_filter.json \
--output-spaces MNI152NLin2009cAsym:res-2 \
--stop-on-first-crash \
--verbose \
Relevant log outputs (up to 20 lines):
Traceback (most recent call last):
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
result["result"] = node.run(updatehash=updatehash)
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
result = self._run_interface(execute=True)
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 1380, in _run_interface
result = self._collate_results(
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 1249, in _collate_results
for i, nresult, err in nodes:
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/utils.py", line 94, in nodelist_runner
result = node.run(updatehash=updatehash)
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
result = self._run_interface(execute=True)
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
return self._run_command(execute)
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node _inu_n40.
Cmdline:
N4BiasFieldCorrection --bspline-fitting [ 200 ] -d 3 --input-image /scratch/groups/kpohl/data/work/fmriprep_23_2_wf/sub_102447_wf/anat_fit_wf/brain_extraction_wf/truncate_images/mapflow/_truncate_images0/sub-102447_acq-3d_T1w_noise_corrected_ras_valid_maths.nii.gz --convergence [ 50x50x50x50, 1e-07 ] --output sub-102447_acq-3d_T1w_noise_corrected_ras_valid_maths_corrected.nii.gz --shrink-factor 4
Stdout:
Stderr:
Traceback:
Traceback (most recent call last):
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 453, in aggregate_outputs
setattr(outputs, key, val)
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
value = super(File, self).validate(objekt, name, value, return_pathlike=True)
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
self.error(objekt, name, str(value))
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/traits/base_trait_handler.py", line 74, in error
raise TraitError(
traits.trait_errors.TraitError: The 'output_image' trait of a N4BiasFieldCorrectionOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/scratch/groups/kpohl/data/work/fmriprep_23_2_wf/sub_102447_wf/anat_fit_wf/brain_extraction_wf/inu_n4/mapflow/_inu_n40/sub-102447_acq-3d_T1w_noise_corrected_ras_valid_maths_corrected.nii.gz' <class 'str'> was specified.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 398, in run
runtime = self._post_run_hook(runtime)
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/mixins/fixheader.py", line 127, in _post_run_hook
outputs = self.aggregate_outputs(runtime=runtime).get_traitsfree()
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 460, in aggregate_outputs
raise FileNotFoundError(msg)
FileNotFoundError: No such file or directory '/scratch/groups/redacted/data/work/fmriprep_23_2_wf/sub_102447_wf/anat_fit_wf/brain_extraction_wf/inu_n4/mapflow/_inu_n40/sub-102447_acq-3d_T1w_noise_corrected_ras_valid_maths_corrected.nii.gz' for output 'output_image' of a N4BiasFieldCorrection interface
More information:
I have verified that the subjects’ folders (BIDS validated) have 3D T1w images under anat/ and BOLD images under func/. I also deleted any derivatives and temporary work files created and re-ran the script but they failed again. Lastly, I read previous posts and increased mem_mb; in my slurm job, I assign --mem=70GB and --cpus-per-task=32, and that also hasn’t made a difference.