Fmriprep crash wf.initial_boldref_wf.enhance_and_skullstrip_bold_wf.n4_correct

Software Support
1

Summary of what happened:

Hi all,

I’m getting a crash while using fmriprep. I successfully completed preprocessing using fmriprep with the exact same command for all other subjects but two that keep crashing.

Command used (and if a helper script was used, a link to the helper script or the command generated):

docker run -it --rm ^
    -v C:\Users\YossiY100\Desktop\2020_analysis\Nifti:/data:ro ^
    -v C:\Users\YossiY100\Desktop\2020_analysis\derivatives:/out ^
    -v C:\Users\YossiY100\Desktop\license.txt:/opt/freesurfer/license.txt:ro ^
    nipreps/fmriprep:23.0.2 ^
    /data /out/fmriprep-23.0.2 ^
    participant --participant-label sub-1737202007130846 ^
    --fs-license-file /opt/freesurfer/license.txt ^
    --ignore slicetiming ^
    --low-mem ^
    --mem-mb 11000 ^
    --skip_bids_validation

Version:

fmriprep 23.0.2

Environment (Docker, Singularity, custom installation):

docker

Data formatted according to a validatable standard? Please provide the output of the validator:

File Path: Not all subjects contain the same files. Each subject should contain the same number of files with the same naming unless some files are known to be missing.

	Type:		Warning
	File:		sub-1738202007131018_ses-1_FLAIR.json
	Location:		/sub-1738202007131018/ses-1/anat/sub-1738202007131018_ses-1_FLAIR.json
	Reason:		This file is missing for subject sub-1738202007131018, but is present for at least one other subject.
	Evidence:	Subject: sub-1738202007131018; Missing file: sub-1738202007131018_ses-1_FLAIR.json

	Type:		Warning
	File:		sub-1738202007131018_ses-1_FLAIR.nii
	Location:		/sub-1738202007131018/ses-1/anat/sub-1738202007131018_ses-1_FLAIR.nii
	Reason:		This file is missing for subject sub-1738202007131018, but is present for at least one other subject.
	Evidence:	Subject: sub-1738202007131018; Missing file: sub-1738202007131018_ses-1_FLAIR.nii


======================================================


File Path: Not all subjects/sessions/runs have the same scanning parameters.

	Type:		Warning
	File:		sub-1642202006170835_ses-1_task-IndividualSounds2020_acq-TR1000_bold.nii
	Location:		Nifti/sub-1642202006170835/ses-1/func/sub-1642202006170835_ses-1_task-IndividualSounds2020_acq-TR1000_bold.nii
	Reason:		 The most common set of dimensions is: 102,102,72,417 (voxels), This file has the dimensions: 98,98,72,387 (voxels).
	Type:		Warning
	File:		sub-1642202006170835_ses-1_task-IndividualSounds2020_acq-TR1000_sbref.nii
	Location:		Nifti/sub-1642202006170835/ses-1/func/sub-1642202006170835_ses-1_task-IndividualSounds2020_acq-TR1000_sbref.nii
	Reason:		 The most common set of dimensions is: 102,102,72 (voxels), This file has the dimensions: 98,98,72 (voxels).

======================================================


File Path: The onset of the last event is after the total duration of the corresponding scan. This design is suspiciously long. 

	Type:		Warning
	File:		task-IndividualSounds2020_events.tsv
	Location:		Nifti/task-IndividualSounds2020_events.tsv
	Reason:		The onset of the last event is after the total duration of the corresponding scan. This design is suspiciously long. 
	Type:		Warning
	File:		task-IndividualSounds2020_events.tsv
	Location:		Nifti/task-IndividualSounds2020_events.tsv
	Reason:		The onset of the last event is after the total duration of the corresponding scan. This design is suspiciously long. 
	Type:		Warning
	File:		task-IndividualSounds2020_events.tsv
	Location:		Nifti/task-IndividualSounds2020_events.tsv
	Reason:		The onset of the last event is after the total duration of the corresponding scan. This design is suspiciously long. 
	Type:		Warning
	File:		task-IndividualSounds2020_events.tsv
	Location:		Nifti/task-IndividualSounds2020_events.tsv
	Reason:		The onset of the last event is after the total duration of the corresponding scan. This design is suspiciously long. 
	Type:		Warning
	File:		task-IndividualSounds2020_events.tsv
	Location:		Nifti/task-IndividualSounds2020_events.tsv
	Reason:		The onset of the last event is after the total duration of the corresponding scan. This design is suspiciously long. 
	Type:		Warning
	File:		task-IndividualSounds2020_events.tsv
	Location:		Nifti/task-IndividualSounds2020_events.tsv
	Reason:		The onset of the last event is after the total duration of the corresponding scan. This design is suspiciously long. 
	Type:		Warning
	File:		task-IndividualSounds2020_events.tsv
	Location:		Nifti/task-IndividualSounds2020_events.tsv
	Reason:		The onset of the last event is after the total duration of the corresponding scan. This design is suspiciously long. 
	Type:		Warning
	File:		task-IndividualSounds2020_events.tsv
	Location:		Nifti/task-IndividualSounds2020_events.tsv
	Reason:		The onset of the last event is after the total duration of the corresponding scan. This design is suspiciously long. 
	Type:		Warning
	File:		task-IndividualSounds2020_events.tsv
	Location:		Nifti/task-IndividualSounds2020_events.tsv
	Reason:		The onset of the last event is after the total duration of the corresponding scan. This design is suspiciously long. 
	Type:		Warning
	File:		task-IndividualSounds2020_events.tsv
	Location:		Nifti/task-IndividualSounds2020_events.tsv
	Reason:		The onset of the last event is after the total duration of the corresponding scan. This design is suspiciously long. 
	Type:		Warning
	File:		task-IndividualSounds2020_events.tsv
	Location:		Nifti/task-IndividualSounds2020_events.tsv
	Reason:		The onset of the last event is after the total duration of the corresponding scan. This design is suspiciously long. 
	Type:		Warning
	File:		task-IndividualSounds2020_events.tsv
	Location:		Nifti/task-IndividualSounds2020_events.tsv
	Reason:		The onset of the last event is after the total duration of the corresponding scan. This design is suspiciously long. 
	Type:		Warning
	File:		task-IndividualSounds2020_events.tsv
	Location:		Nifti/task-IndividualSounds2020_events.tsv
	Reason:		The onset of the last event is after the total duration of the corresponding scan. This design is suspiciously long. 
	Type:		Warning
	File:		task-IndividualSounds2020_events.tsv
	Location:		Nifti/task-IndividualSounds2020_events.tsv
	Reason:		The onset of the last event is after the total duration of the corresponding scan. This design is suspiciously long. 
	Type:		Warning
	File:		task-IndividualSounds2020_events.tsv
	Location:		Nifti/task-IndividualSounds2020_events.tsv
	Reason:		The onset of the last event is after the total duration of the corresponding scan. This design is suspiciously long. 
	Type:		Warning
	File:		task-IndividualSounds2020_events.tsv
	Location:		Nifti/task-IndividualSounds2020_events.tsv
	Reason:		The onset of the last event is after the total duration of the corresponding scan. This design is suspiciously long. 
	Type:		Warning
	File:		task-IndividualSounds2020_events.tsv
	Location:		Nifti/task-IndividualSounds2020_events.tsv
	Reason:		The onset of the last event is after the total duration of the corresponding scan. This design is suspiciously long. 

======================================================

Relevant log outputs (up to 20 lines):

Here is the log from the crash .txt:

Node: fmriprep_23_0_wf.single_subject_1737202007130846_wf.func_preproc_ses_1_task_IndividualSounds2020_acq_TR1000_wf.initial_boldref_wf.enhance_and_skullstrip_bold_wf.n4_correct
Working directory: /tmp/work/fmriprep_23_0_wf/single_subject_1737202007130846_wf/func_preproc_ses_1_task_IndividualSounds2020_acq_TR1000_wf/initial_boldref_wf/enhance_and_skullstrip_bold_wf/n4_correct

Node inputs:

args = <undefined>
bias_image = <undefined>
bspline_fitting_distance = 200.0
bspline_order = <undefined>
convergence_threshold = <undefined>
copy_header = True
dimension = 3
environ = {'NSLOTS': '1'}
histogram_sharpening = <undefined>
input_image = <undefined>
mask_image = <undefined>
n_iterations = <undefined>
num_threads = 1
output_image = <undefined>
rescale_intensities = True
save_bias = False
shrink_factor = <undefined>
weight_image = <undefined>

Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node n4_correct.

Cmdline:
	N4BiasFieldCorrection --bspline-fitting [ 200 ] -d 3 --input-image /tmp/work/fmriprep_23_0_wf/single_subject_1737202007130846_wf/func_preproc_ses_1_task_IndividualSounds2020_acq_TR1000_wf/initial_boldref_wf/gen_avg/sub-1737202007130846_ses-1_task-IndividualSounds2020_acq-TR1000_sbref_merged_squeezed.nii --output sub-1737202007130846_ses-1_task-IndividualSounds2020_acq-TR1000_sbref_merged_squeezed_corrected.nii -r --weight-image /tmp/work/fmriprep_23_0_wf/single_subject_1737202007130846_wf/func_preproc_ses_1_task_IndividualSounds2020_acq_TR1000_wf/initial_boldref_wf/enhance_and_skullstrip_bold_wf/check_hdr/tpl-MNI152NLin2009cAsym_res-01_label-brain_probseg_trans_hdr.nii.gz
Stdout:

Stderr:

Traceback:
	Traceback (most recent call last):
	  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 453, in aggregate_outputs
	    setattr(outputs, key, val)
	  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
	    value = super(File, self).validate(objekt, name, value, return_pathlike=True)
	  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
	    self.error(objekt, name, str(value))
	  File "/opt/conda/lib/python3.9/site-packages/traits/base_trait_handler.py", line 74, in error
	    raise TraitError(
	traits.trait_errors.TraitError: The 'output_image' trait of a _FixN4BiasFieldCorrectionOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/tmp/work/fmriprep_23_0_wf/single_subject_1737202007130846_wf/func_preproc_ses_1_task_IndividualSounds2020_acq_TR1000_wf/initial_boldref_wf/enhance_and_skullstrip_bold_wf/n4_correct/sub-1737202007130846_ses-1_task-IndividualSounds2020_acq-TR1000_sbref_merged_squeezed_corrected.nii' <class 'str'> was specified.

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 398, in run
	    runtime = self._post_run_hook(runtime)
	  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/mixins/fixheader.py", line 127, in _post_run_hook
	    outputs = self.aggregate_outputs(runtime=runtime).get_traitsfree()
	  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 460, in aggregate_outputs
	    raise FileNotFoundError(msg)
	FileNotFoundError: No such file or directory '/tmp/work/fmriprep_23_0_wf/single_subject_1737202007130846_wf/func_preproc_ses_1_task_IndividualSounds2020_acq_TR1000_wf/initial_boldref_wf/enhance_and_skullstrip_bold_wf/n4_correct/sub-1737202007130846_ses-1_task-IndividualSounds2020_acq-TR1000_sbref_merged_squeezed_corrected.nii' for output 'output_image' of a FixN4BiasFieldCorrection interface

Screenshots / relevant information:

Would really appreciate your help with this! Thank you in advance

2

Hi @Uri_Shinitsky,

I have relabeled your post as Software Support, added the fmriprep tag, and added the corresponding software support template. In the future, please use this category for software related question and tag your posts accordingly so it can get the proper help.

Please fill in the missing information of the template by editing your post.

Best,
Steven

1 Like
3

Hi @Uri_Shinitsky,

Thank you for updating your post, please try specifying a work directory in your fmriprep command with the -w flag.

Best,
Steven

1 Like
4

Thanks for the reply. Tried adding -w flag, unfortunately exact same error occurs.

Command used (and if a helper script was used, a link to the helper script or the command generated):

docker run -it --rm ^
    -v C:\Users\YossiY100\Desktop\2020_analysis\Nifti:/data:ro ^
    -v C:\Users\YossiY100\Desktop\2020_analysis\derivatives:/out ^
    -v C:\Users\YossiY100\Desktop\license.txt:/opt/freesurfer/license.txt:ro ^
    -v C:\Users\YossiY100\Desktop\tmp:/mnt/c/Users/YossiY100/Desktop/tmp ^
    nipreps/fmriprep:23.0.2 ^
    /data /out/fmriprep-23.0.2 ^
    participant --participant-label sub-1737202007130846 ^
    --fs-license-file /opt/freesurfer/license.txt ^
    --ignore slicetiming ^
    --low-mem ^
    --mem-mb 11000 ^
    --work-dir /mnt/c/Users/YossiY100/Desktop/tmp ^
    --skip_bids_validation
5

Can you try without the low-mem flag?

6

Tried just now - same error

7

The exact same? Is it still looking in /tmp/work/fmriprep_23_0_wf/....? Because that would suggest the working directory argument is not being recognized.

8

Seems like it does recognize the working directory argument

Node: fmriprep_23_0_wf.single_subject_1737202007130846_wf.func_preproc_ses_1_task_IndividualSounds2020_acq_TR1000_wf.initial_boldref_wf.enhance_and_skullstrip_bold_wf.n4_correct
Working directory: /mnt/c/Users/YossiY100/Desktop/tmp/fmriprep_23_0_wf/single_subject_1737202007130846_wf/func_preproc_ses_1_task_IndividualSounds2020_acq_TR1000_wf/initial_boldref_wf/enhance_and_skullstrip_bold_wf/n4_correct

Node inputs:

args = <undefined>
bias_image = <undefined>
bspline_fitting_distance = 200.0
bspline_order = <undefined>
convergence_threshold = <undefined>
copy_header = True
dimension = 3
environ = {'NSLOTS': '1'}
histogram_sharpening = <undefined>
input_image = /mnt/c/Users/YossiY100/Desktop/tmp/fmriprep_23_0_wf/single_subject_1737202007130846_wf/func_preproc_ses_1_task_IndividualSounds2020_acq_TR1000_wf/initial_boldref_wf/gen_avg/sub-1737202007130846_ses-1_task-IndividualSounds2020_acq-TR1000_sbref_merged_squeezed.nii
mask_image = <undefined>
n_iterations = <undefined>
num_threads = 1
output_image = <undefined>
rescale_intensities = True
save_bias = False
shrink_factor = <undefined>
weight_image = /mnt/c/Users/YossiY100/Desktop/tmp/fmriprep_23_0_wf/single_subject_1737202007130846_wf/func_preproc_ses_1_task_IndividualSounds2020_acq_TR1000_wf/initial_boldref_wf/enhance_and_skullstrip_bold_wf/check_hdr/tpl-MNI152NLin2009cAsym_res-01_label-brain_probseg_trans_hdr.nii.gz

Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node n4_correct.

Cmdline:
	N4BiasFieldCorrection --bspline-fitting [ 200 ] -d 3 --input-image /mnt/c/Users/YossiY100/Desktop/tmp/fmriprep_23_0_wf/single_subject_1737202007130846_wf/func_preproc_ses_1_task_IndividualSounds2020_acq_TR1000_wf/initial_boldref_wf/gen_avg/sub-1737202007130846_ses-1_task-IndividualSounds2020_acq-TR1000_sbref_merged_squeezed.nii --output sub-1737202007130846_ses-1_task-IndividualSounds2020_acq-TR1000_sbref_merged_squeezed_corrected.nii -r --weight-image /mnt/c/Users/YossiY100/Desktop/tmp/fmriprep_23_0_wf/single_subject_1737202007130846_wf/func_preproc_ses_1_task_IndividualSounds2020_acq_TR1000_wf/initial_boldref_wf/enhance_and_skullstrip_bold_wf/check_hdr/tpl-MNI152NLin2009cAsym_res-01_label-brain_probseg_trans_hdr.nii.gz
Stdout:

Stderr:

Traceback:
	Traceback (most recent call last):
	  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 453, in aggregate_outputs
	    setattr(outputs, key, val)
	  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
	    value = super(File, self).validate(objekt, name, value, return_pathlike=True)
	  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
	    self.error(objekt, name, str(value))
	  File "/opt/conda/lib/python3.9/site-packages/traits/base_trait_handler.py", line 74, in error
	    raise TraitError(
	traits.trait_errors.TraitError: The 'output_image' trait of a _FixN4BiasFieldCorrectionOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/mnt/c/Users/YossiY100/Desktop/tmp/fmriprep_23_0_wf/single_subject_1737202007130846_wf/func_preproc_ses_1_task_IndividualSounds2020_acq_TR1000_wf/initial_boldref_wf/enhance_and_skullstrip_bold_wf/n4_correct/sub-1737202007130846_ses-1_task-IndividualSounds2020_acq-TR1000_sbref_merged_squeezed_corrected.nii' <class 'str'> was specified.

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 398, in run
	    runtime = self._post_run_hook(runtime)
	  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/mixins/fixheader.py", line 127, in _post_run_hook
	    outputs = self.aggregate_outputs(runtime=runtime).get_traitsfree()
	  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 460, in aggregate_outputs
	    raise FileNotFoundError(msg)
	FileNotFoundError: No such file or directory '/mnt/c/Users/YossiY100/Desktop/tmp/fmriprep_23_0_wf/single_subject_1737202007130846_wf/func_preproc_ses_1_task_IndividualSounds2020_acq_TR1000_wf/initial_boldref_wf/enhance_and_skullstrip_bold_wf/n4_correct/sub-1737202007130846_ses-1_task-IndividualSounds2020_acq-TR1000_sbref_merged_squeezed_corrected.nii' for output 'output_image' of a FixN4BiasFieldCorrection interface
9

Has this been resolved? I have a similar issue. Running fmriprep using Singularity.

  • fMRIPrep version: 23.0.2
  • fMRIPrep command: /opt/conda/bin/fmriprep /data /out/fmriprep participant --participant-label sub-sid002311 -w /out/work/ --nthreads 8 --omp-nthreads 8 --fs-license-file /out/code/license.txt --fs-no-reconall --notrack --skip-bids-validation --ignore slicetiming --output-spaces MNI152NLin2009cAsym:res-2

Same singularity command was used for each participant individually. most of them worked fine but a few (5) of a large dataset (>100) returned the same error and only for some of their runs (Traceback below).

The runs that have errors do have a final -preproc_bold.nii.gz file though. Should those be used?

" Node Name: fmriprep_23_0_wf.single_subject_sid001731_wf.func_preproc_task_movie_run_07_wf.initial_boldref_wf.enhance_and_skullstrip_bold_wf.n4_correct

File: /out/fmriprep/sub-sid001731/log/20230525-234352_fe038ce0-65c1-431d-ae02-8bea95598acd/crash-20230526-001122-f0065x7-n4_correct-3ac28077-ea4f-44de-9da4-8859df89e846.txt
Working Directory: /out/work/fmriprep_23_0_wf/single_subject_sid001731_wf/func_preproc_task_movie_run_07_wf/initial_boldref_wf/enhance_and_skullstrip_bold_wf/n4_correct
Inputs:

  • args:``
  • bias_image:``
  • bspline_fitting_distance: 200.0
  • bspline_order:``
  • convergence_threshold:``
  • copy_header: True
  • dimension: 3
  • environ: {'NSLOTS': '1'}
  • histogram_sharpening:``
  • input_image:``
  • mask_image:``
  • n_iterations:``
  • num_threads: 1
  • output_image:``
  • rescale_intensities: True
  • save_bias: False
  • shrink_factor:``
  • weight_image:``

Traceback (most recent call last): File “/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py”, line 67, in run_node result[“result”] = node.run(updatehash=updatehash) File “/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py”, line 527, in run result = self._run_interface(execute=True) File “/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py”, line 645, in _run_interface return self._run_command(execute) File “/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py”, line 771, in _run_command raise NodeExecutionError(msg) nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node n4_correct. Cmdline: N4BiasFieldCorrection --bspline-fitting [ 200 ] -d 3 --input-image /out/work/fmriprep_23_0_wf/single_subject_sid001731_wf/func_preproc_task_movie_run_07_wf/initial_boldref_wf/gen_avg/sub-sid001731_task-movie_run-07_bold_average.nii.gz --output sub-sid001731_task-movie_run-07_bold_average_corrected.nii.gz -r --weight-image /out/work/fmriprep_23_0_wf/single_subject_sid001731_wf/func_preproc_task_movie_run_07_wf/initial_boldref_wf/enhance_and_skullstrip_bold_wf/check_hdr/tpl-MNI152NLin2009cAsym_res-01_label-brain_probseg_trans_hdr.nii.gz Stdout: Stderr: Traceback: Traceback (most recent call last): File “/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py”, line 453, in aggregate_outputs setattr(outputs, key, val) File “/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/traits_extension.py”, line 330, in validate value = super(File, self).validate(objekt, name, value, return_pathlike=True) File “/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/traits_extension.py”, line 135, in validate self.error(objekt, name, str(value)) File “/opt/conda/lib/python3.9/site-packages/traits/base_trait_handler.py”, line 74, in error raise TraitError( traits.trait_errors.TraitError: The ‘output_image’ trait of a _FixN4BiasFieldCorrectionOutputSpec instance must be a pathlike object or string representing an existing file, but a value of ‘/out/work/fmriprep_23_0_wf/single_subject_sid001731_wf/func_preproc_task_movie_run_07_wf/initial_boldref_wf/enhance_and_skullstrip_bold_wf/n4_correct/sub-sid001731_task-movie_run-07_bold_average_corrected.nii.gz’ was specified. During handling of the above exception, another exception occurred: Traceback (most recent call last): File “/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py”, line 398, in run runtime = self._post_run_hook(runtime) File “/opt/conda/lib/python3.9/site-packages/nipype/interfaces/mixins/fixheader.py”, line 127, in _post_run_hook outputs = self.aggregate_outputs(runtime=runtime).get_traitsfree() File “/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py”, line 460, in aggregate_outputs raise FileNotFoundError(msg) FileNotFoundError: No such file or directory ‘/out/work/fmriprep_23_0_wf/single_subject_sid001731_wf/func_preproc_task_movie_run_07_wf/initial_boldref_wf/enhance_and_skullstrip_bold_wf/n4_correct/sub-sid001731_task-movie_run-07_bold_average_corrected.nii.gz’ for output ‘output_image’ of a FixN4BiasFieldCorrection interface "

10

I’m also having the same issue. fMRIPrep finished preprocessing for all subjects except one (44 out of 45) and is giving me the same error as above, though for all of this one subjects’ runs.

  • fMRIPrep version: 23.0.2
  • fMRIPrep command: apptainer exec ${filepath} fmriprep ${root_dir} ${output_dir} participant --participant-label ${subjID} -w ${work_dir}
    –output-spaces MNI152Lin fsaverage --bold2t1w-dof 6 --cifti-output 91k -vv --n_cpus 16 --omp-nthreads 8 --nthreads 12 --mem_mb 30000 --fs-license-file /home/mc2835/project/license.txt

I’m attaching a crash file from one of the runs here. Any advice on how to fix this would be much appreciated!

Megha
crash-20230605-142549-mc2835-n4_correct-9f324884-918f-44c8-b355-2ce5851eb3f9.txt (4.9 KB)

11

Hi @megha ,

I wonder if this could be due to a lack of memory. Do you have the possibility to increase --mem_mb 30000 to a higher value?

12

Hi @jsein

Note that in my example above there is no explicit limit on memory. Do you know if there’s a default value that might need to be increased for particular cases?

13

Indeed the error message shows that the N4BiasFieldCorrection command with the mean bold image as input failed to produce an output. I wonder if that could come from a limit with the memory. I am not sure of this but if you can, maybe you can try --mem_mb 48000 or 64000?

14

Bias-field correcting a mean BOLD image really shouldn’t use a ton of memory. I wonder if you’d be willing to try with nipreps/fmriprep:unstable to see if this will still be in issue in 23.1.0. We’ve upgraded ANTs, which resolved another N4 issue that cropped up recently.

15

Indeed changing the mem-mb did not change the outcome. Same runs still error in the same way.
In case it helps somehow, I found some warnings preceding the error that seam relevant. Pasted below.

I am hesitant to try a new version at this point becauseI have over 100 participants successfully processed with the same fmriprep version/command.

If there are no alternative troubleshooting steps to take, I can try to run the erroring subject separately.

“230605-18:45:12,174 nipype.workflow INFO:
[Node] Finished “n4_correct”, elapsed time 0.439168s.
230605-18:45:12,174 nipype.workflow WARNING:
Storing result file without outputs
230605-18:45:12,179 nipype.workflow INFO:
[Node] Finished “n4_correct”, elapsed time 0.452177s.
230605-18:45:12,179 nipype.workflow WARNING:
Storing result file without outputs
230605-18:45:12,182 nipype.workflow WARNING:
[Node] Error on “fmriprep_23_0_wf.single_subject_sid001429_wf.func_preproc_task_movie_run_06_wf.initial_boldref_wf.enhance_and_skullstrip_bold_wf.n4_correct” (/out/work/fmriprep_23_0_wf/single_subject_sid001429_wf/func_preproc_task_movie_run_06_wf/initial_boldref_wf/enhance_and_skullstrip_bold_wf/n4_correct)
230605-18:45:12,187 nipype.workflow WARNING:
[Node] Error on “fmriprep_23_0_wf.single_subject_sid001429_wf.func_preproc_task_movie_run_07_wf.initial_boldref_wf.enhance_and_skullstrip_bold_wf.n4_correct” (/out/work/fmriprep_23_0_wf/single_subject_sid001429_wf/func_preproc_task_movie_run_07_wf/initial_boldref_wf/enhance_and_skullstrip_bold_wf/n4_correct)
230605-18:45:12,723 nipype.workflow ERROR:
Node n4_correct failed to run on host m08.hpcc.dartmouth.edu.
230605-18:45:12,726 nipype.workflow ERROR:”

16

Thank you for reporting. Did you remove the temporary files when launching a new iteration?
I think I remember falling into this issue some some time ago, and I think it was when I used a local installation of fmriprep when our HPC was temporarily out of order. Everything came back to normal when th HPC was back in business and we could launch again the singularity image of fmriprep. Do you use fmriprep with singularity or docker?
EDIT: looking at the log output, it looks like a HPC, so I guess singularity…

Could you look at the input of the N4BiasFieldCorrection command to see if the image look normal?

17

I use singularity and I always remove the entire subject folder from my work folder (and from output folder) before re-running.

I will need more guidance for which specific files to look at.
Regardless, I don’t understand why would it ever do “Storing result files without outputs”

18

I will need more guidance for which specific files to look at.

If you go to the working directory where the error was found (/out/work/fmriprep_23_0_wf/single_subject_sid001429_wf/func_preproc_task_movie_run_07_wf/initial_boldref_wf/enhance_and_skullstrip_bold_wf/n4_correct, modulo any adjustments for bound/mounted locations), then you should find a command.txt file. From there you can see what the inputs to the command were. If you view the inputs in a viewer like mricron or fsleyes, you might see an obvious issue with the inputs. Then it’s a matter of chasing the directories back until you find where files go from looking sensible to wrong.

(In some cases, the node might be a Python function instead of a shell command. In that case, you’ll want to look at _report/report.rst instead of command.txt to see the inputs.)

19

The input is /out/work/fmriprep_23_0_wf/single_subject_sid001429_wf/func_preproc_task_movie_run_07_wf/initial_boldref_wf/gen_avg/sub-sid001429_task-movie_run-07_bold_average.nii.gz
The file looks pretty sensible to me. The only noticeable difference between this one and the bold_average from other runs that did not crash was how far they are from center of the coordinate space. Is it possible that there is a tolerance for how far of a translation can be applied to the data - that is the whole brain needs to be moved further to be centered on the axis.
Though I’m not sure why that would interact with the n4 process of all things.

20

Also, I loaded the same run that errored and at a first glance the preproc_bold.nii.gz for the same run looks fine