Exception raised while executing Node enhance_pre_sdc

Akram · January 15, 2024, 11:25pm

Hi everyone,

I ran qsiprep version 0.20.1 on 1 subject with 3 sessions each with 3 runs (BIDS valid), with the following command:

docker run -ti --rm \
    -v /Users/akram/proj/bids:/data  \
    -v /Users/akram/proj/outbids:/output \
    -v ${FREESURFER_HOME}/license.txt:/opt/freesurfer/license.txt \
    pennbbl/qsiprep:latest \
    /data /output participant \
    --ignore fieldmaps \
    --output-resolution 1.2 \
    --fs-license-file /opt/freesurfer/license.txt

and I get the following error:

Node: qsiprep_wf.single_subject_002_wf.dwi_preproc_ses_01_wf.hmc_sdc_wf.enhance_pre_sdc
Working directory: /tmp/work/qsiprep_wf/single_subject_002_wf/dwi_preproc_ses_01_wf/hmc_sdc_wf/enhance_pre_sdc

Node inputs:

b0_file = <undefined>

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node enhance_pre_sdc.

Traceback:
	Traceback (most recent call last):
	  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 397, in run
	    runtime = self._run_interface(runtime)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/qsiprep/interfaces/nilearn.py", line 258, in _run_interface
	    bias_corrected, bias_img = biascorrect(input_img, cwd=runtime.cwd)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/qsiprep/interfaces/nilearn.py", line 358, in biascorrect
	    new_img = load_img(out_fname)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/nilearn/image/image.py", line 1334, in load_img
	    return check_niimg(img, wildcards=wildcards, dtype=dtype)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/nilearn/_utils/niimg_conversions.py", line 306, in check_niimg
	    raise ValueError(f"File not found: '{niimg}'")
	ValueError: File not found: '/tmp/work/qsiprep_wf/single_subject_002_wf/dwi_preproc_ses_01_wf/hmc_sdc_wf/enhance_pre_sdc/tmpcm2y6s_2.nii.gz'

Any idea on how to proceed?

Thanks,
Akram

Steven · January 16, 2024, 12:09am

Hi @Akram,

For future posts like this, please use the Software Support category, which also gives you a post template for you to provide information we can use to better help you.

Please include a working directory with the -w flag (make sure to bind it to the docker container too) and try again.

Can you also return the output from the BIDS validator?

Best,
Steven

Akram · January 16, 2024, 1:10am

Thank you @Steven.
Of course.

I included a working directory, this time it crashed. One crash file is similar to what I shared above and the other one is below

Node: qsiprep_wf.single_subject_002_wf.dwi_preproc_ses_01_wf.hmc_sdc_wf.pre_eddy_b0_ref_wf.synthstrip_wf.pad_before_synthstrip_wf.resample_skulled_to_reference
Working directory: /out/work/qsiprep_wf/single_subject_002_wf/dwi_preproc_ses_01_wf/hmc_sdc_wf/pre_eddy_b0_ref_wf/synthstrip_wf/pad_before_synthstrip_wf/resample_skulled_to_reference

Node inputs:

args = <undefined>
default_value = 0.0
dimension = 3
environ = {'NSLOTS': '1'}
float = False
input_image = <undefined>
input_image_type = <undefined>
interpolation = BSpline
interpolation_parameters = <undefined>
invert_transform_flags = <undefined>
num_threads = 1
out_postfix = _trans
output_image = <undefined>
print_out_composite_warp_file = <undefined>
reference_image = <undefined>
transforms = ['identity']

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node resample_skulled_to_reference.

Cmdline:
	antsApplyTransforms --default-value 0 --dimensionality 3 --float 0 --input /out/work/qsiprep_wf/single_subject_002_wf/dwi_preproc_ses_01_wf/hmc_sdc_wf/gather_inputs/topup_imain.nii.gz --interpolation BSpline --output topup_imain_trans.nii.gz --reference-image /out/work/qsiprep_wf/single_subject_002_wf/dwi_preproc_ses_01_wf/hmc_sdc_wf/pre_eddy_b0_ref_wf/synthstrip_wf/pad_before_synthstrip_wf/prepare_synthstrip_reference/topup_imain_resample_autobox_SynthStripGrid.nii --transform identity
Stdout:

Stderr:
	Illegal instruction
Traceback:
	Traceback (most recent call last):
	  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 453, in aggregate_outputs
	    setattr(outputs, key, val)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
	    value = super(File, self).validate(objekt, name, value, return_pathlike=True)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
	    self.error(objekt, name, str(value))
	  File "/usr/local/miniconda/lib/python3.10/site-packages/traits/base_trait_handler.py", line 74, in error
	    raise TraitError(
	traits.trait_errors.TraitError: The 'output_image' trait of an ApplyTransformsOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/out/work/qsiprep_wf/single_subject_002_wf/dwi_preproc_ses_01_wf/hmc_sdc_wf/pre_eddy_b0_ref_wf/synthstrip_wf/pad_before_synthstrip_wf/resample_skulled_to_reference/topup_imain_trans.nii.gz' <class 'str'> was specified.

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 400, in run
	    outputs = self.aggregate_outputs(runtime)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 460, in aggregate_outputs
	    raise FileNotFoundError(msg)
	FileNotFoundError: No such file or directory '/out/work/qsiprep_wf/single_subject_002_wf/dwi_preproc_ses_01_wf/hmc_sdc_wf/pre_eddy_b0_ref_wf/synthstrip_wf/pad_before_synthstrip_wf/resample_skulled_to_reference/topup_imain_trans.nii.gz' for output 'output_image' of a ApplyTransforms interface

Here is the output from the BIDS validator, with no error:

	1: [WARN] The recommended file /README is very small. Please consider expanding it with additional information about the dataset. (code: 213 - README_FILE_SMALL)


        Summary:                 Available Tasks:                     Available Modalities: 
        79 Files, 692.1MB        midt                                 MRI                   
        1 - Subject              rest                                                       
        3 - Sessions             stop                                                       
                                 TODO: full task name for stop                              
                                 RestingState                                               
                                 TODO: full task name for midt

Let me know if I f I should share other details.

Steven · January 16, 2024, 1:21am

Can you try not putting the work directory in the output directory?

Akram · January 16, 2024, 1:32am

I changed the command as below, still has the same errors:

docker run -ti --rm \
    -v /Users/akram/proj/bids:/data  \
    -v /Users/akram/proj/outbids:/output \
    -v /Users/akram/proj/workbids:/work/ \
    -v ${FREESURFER_HOME}/license.txt:/opt/freesurfer/license.txt \
    pennbbl/qsiprep:latest \
    /data /output participant -w /work/ \
    --ignore fieldmaps \
    --output-resolution 1.2 \
    --fs-license-file /opt/freesurfer/license.txt

Steven · January 16, 2024, 1:39am

How much memory are you giving docker?

Akram · January 16, 2024, 1:45am

Docker: Memory Limit: 16 GB,
Swap: 1 GB
Virtual disk limit: 64 GB
and these are the run details: Running with omp_nthreads=7, nthreads=8

Steven · January 16, 2024, 2:00am

Do you have more memory you can give to Docker? And is there any other part of the log that contains errors?

Akram · January 16, 2024, 1:36pm

Unfortunately no. Yes this error:

Node: _n4_correct0
Working directory: /work/qsiprep_wf/single_subject_002_wf/anat_preproc_wf/anat_template_wf/n4_correct/mapflow/_n4_correct0

Node inputs:

args = <undefined>
bias_image = <undefined>
bspline_fitting_distance = 200.0
bspline_order = <undefined>
convergence_threshold = 1e-07
copy_header = True
dimension = 3
environ = {'NSLOTS': '7'}
histogram_sharpening = <undefined>
input_image = /work/qsiprep_wf/single_subject_002_wf/anat_preproc_wf/anat_template_wf/anat_conform/mapflow/_anat_conform0/sub-002_ses-01_run-01_T1w_lps.nii.gz
mask_image = <undefined>
n_iterations = [50, 50, 50, 50]
num_threads = 7
output_image = <undefined>
rescale_intensities = False
save_bias = False
shrink_factor = 4
weight_image = <undefined>

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node _n4_correct0.

Cmdline:
	N4BiasFieldCorrection --bspline-fitting [ 200 ] -d 3 --input-image /work/qsiprep_wf/single_subject_002_wf/anat_preproc_wf/anat_template_wf/anat_conform/mapflow/_anat_conform0/sub-002_ses-01_run-01_T1w_lps.nii.gz --convergence [ 50x50x50x50, 1e-07 ] --output sub-002_ses-01_run-01_T1w_lps_corrected.nii.gz --shrink-factor 4
Stdout:

Stderr:
	Illegal instruction
Traceback:
	Traceback (most recent call last):
	  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 453, in aggregate_outputs
	    setattr(outputs, key, val)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
	    value = super(File, self).validate(objekt, name, value, return_pathlike=True)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
	    self.error(objekt, name, str(value))
	  File "/usr/local/miniconda/lib/python3.10/site-packages/traits/base_trait_handler.py", line 74, in error
	    raise TraitError(
	traits.trait_errors.TraitError: The 'output_image' trait of a N4BiasFieldCorrectionOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/work/qsiprep_wf/single_subject_002_wf/anat_preproc_wf/anat_template_wf/n4_correct/mapflow/_n4_correct0/sub-002_ses-01_run-01_T1w_lps_corrected.nii.gz' <class 'str'> was specified.

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 398, in run
	    runtime = self._post_run_hook(runtime)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/interfaces/mixins/fixheader.py", line 127, in _post_run_hook
	    outputs = self.aggregate_outputs(runtime=runtime).get_traitsfree()
	  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 460, in aggregate_outputs
	    raise FileNotFoundError(msg)
	FileNotFoundError: No such file or directory '/work/qsiprep_wf/single_subject_002_wf/anat_preproc_wf/anat_template_wf/n4_correct/mapflow/_n4_correct0/sub-002_ses-01_run-01_T1w_lps_corrected.nii.gz' for output 'output_image' of a N4BiasFieldCorrection interface

Steven · January 16, 2024, 1:40pm

Can you provide the full output log? All of these seem like different (but related) errors.

Akram · January 16, 2024, 1:42pm

I don’t see one output log, where could I see it? I have three crash logs that I’ve already shared here.

Steven · January 16, 2024, 1:49pm

If you are running this in terminal, the full output from the terminal.

Akram · January 16, 2024, 2:27pm

I guess this would be the complete terminal log after running the command.

Steven · January 16, 2024, 2:49pm

Can you switch to the latest stable release, 0.20.0, try a fresh working directory, and running just one of the DWIs (e.g., make a copy of the data and delete two of the sessions)?

Akram · January 17, 2024, 10:03pm

Hi,
I tried to run it locally with the conditions you mentioned but the errors didn’t resolve. Since memory probably was an issue I ran it with singularity (keeping everything else the same) on the HPC and it worked fine this time.
Thank you