QSIPrep error during eddy

Summary of what happened:

QSIPrep crashed during eddy

Command used (and if a helper script was used, a link to the helper script or the command generated):

docker run -ti --rm \
    -v /data/BIDS-AHBA/bids:/data \
    -v /data/BIDS-AHBA/output:/output \
    -v /home/freesurfer/license.txt:/opt/freesurfer/license.txt \
    pennlinc/qsiprep:latest \
    /data /output participant \
    --fs-license-file /opt/freesurfer/license.txt \
    --output-resolution 1.2

Version:

latest

Environment (Docker, Singularity / Apptainer, custom installation):

docker

Data formatted according to a validatable standard? Please provide the output of the validator:

validator no error

Relevant log outputs (up to 20 lines):

Node: qsiprep_1_0_wf.sub_10619_wf.dwi_preproc_dir_AP_wf.hmc_sdc_wf.eddy
Working directory: /tmp/work/qsiprep_1_0_wf/sub_10619_wf/dwi_preproc_dir_AP_wf/hmc_sdc_wf/eddy

Node inputs:

args = 
cnr_maps = True
dont_peas = False
dont_sep_offs_move = False
environ = {'FSLOUTPUTTYPE': 'NIFTI_GZ', 'OMP_NUM_THREADS': '8'}
estimate_move_by_susceptibility = <undefined>
fep = False
field = <undefined>
field_mat = <undefined>
flm = quadratic
fudge_factor = 10.0
fwhm = <undefined>
in_acqp = <undefined>
in_bval = <undefined>
in_bvec = <undefined>
in_file = <undefined>
in_index = <undefined>
in_mask = <undefined>
in_topup_fieldcoef = <undefined>
in_topup_movpar = <undefined>
initrand = <undefined>
interp = spline
is_shelled = True
json = <undefined>
mbs_ksp = <undefined>
mbs_lambda = <undefined>
mbs_niter = <undefined>
method = jac
mporder = <undefined>
multiband_factor = <undefined>
multiband_offset = <undefined>
niter = 5
num_threads = 8
nvoxhp = 1000
out_base = eddy_corrected
outlier_nstd = <undefined>
outlier_nvox = <undefined>
outlier_pos = <undefined>
outlier_sqr = <undefined>
outlier_type = <undefined>
output_type = NIFTI_GZ
repol = True
residuals = False
session = <undefined>
slice2vol_interp = <undefined>
slice2vol_lambda = <undefined>
slice2vol_niter = <undefined>
slice_order = <undefined>
slm = linear
use_cuda = False

Traceback (most recent call last):
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 66, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 525, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 643, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 769, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node eddy.

Cmdline:
	eddy_cpu  --cnr_maps --field=/tmp/work/qsiprep_1_0_wf/sub_10619_wf/dwi_preproc_dir_AP_wf/hmc_sdc_wf/topup/fieldmap_HZ --field_mat=/tmp/work/qsiprep_1_0_wf/sub_10619_wf/dwi_preproc_dir_AP_wf/hmc_sdc_wf/topup_to_eddy_reg/topup_reg_image_flirt.mat --flm=quadratic --ff=10.0 --acqp=/tmp/work/qsiprep_1_0_wf/sub_10619_wf/dwi_preproc_dir_AP_wf/hmc_sdc_wf/gather_inputs/eddy_acqp.txt --bvals=/data/sub-10619/dwi/sub-10619_dir-AP_dwi.bval --bvecs=/data/sub-10619/dwi/sub-10619_dir-AP_dwi.bvec --imain=/tmp/work/qsiprep_1_0_wf/sub_10619_wf/dwi_preproc_dir_AP_wf/pre_hmc_wf/merge_and_denoise_wf/dwi_denoise_dir_AP_dwi_wf/denoiser/sub-10619_dir-AP_dwi_denoised.nii.gz --index=/tmp/work/qsiprep_1_0_wf/sub_10619_wf/dwi_preproc_dir_AP_wf/hmc_sdc_wf/gather_inputs/eddy_index.txt --mask=/tmp/work/qsiprep_1_0_wf/sub_10619_wf/dwi_preproc_dir_AP_wf/hmc_sdc_wf/transform_mask_to_eddy/topup_imain_corrected_avg_trans_mask_trans_flirt.nii.gz --interp=spline --data_is_shelled --resamp=jac --niter=5 --nthr=8 --nvoxhp=1000 --out=/tmp/work/qsiprep_1_0_wf/sub_10619_wf/dwi_preproc_dir_AP_wf/hmc_sdc_wf/eddy/eddy_corrected --repol --slm=linear
Stdout:

	Warning: In a future release the first argument will have to be "diffusion" when using eddy on diffusion data, i.e.
	eddy diffusion --imain='my_ima' --acqp='my_acqp' ...

	Warning: Writing of individual text files will be discontinued in favour of a single .json file in future versions

	EDDY::ReplacementManager::Update: Mismatched DiffStatsVector object
	EDDY::EddyHelperClasses.cpp:::  void EDDY::ReplacementManager::Update(const EDDY::DiffStatsVector&):  Exception thrown
	EDDY::eddy.cpp:::  EDDY::DiffStatsVector EDDY::detect_outliers(const EddyCommandLineOptions&, ScanType, std::shared_ptr<const DWIPredictionMaker>, const NEWIMAGE::volume<float>&, const ECScanManager&, unsigned int, unsigned int, ReplacementManager&):  Exception thrown
	EDDY::eddy.cpp:::  EDDY::DiffStatsVector EDDY::DetectOutliers(const EddyCommandLineOptions&, ScanType, std::shared_ptr<const DWIPredictionMaker>, const NEWIMAGE::volume<float>&, const ECScanManager&, ReplacementManager&):  Exception thrown
	EDDY::eddy.cpp:::  EDDY::ReplacementManager* EDDY::Register(const EddyCommandLineOptions&, ScanType, unsigned int, const std::vector<float, std::allocator<float> >&, SecondLevelECModelType, bool, ECScanManager&, ReplacementManager*, NEWMAT::Matrix&, NEWMAT::Matrix&):  Exception thrown
	EDDY::: Eddy failed with message EDDY::eddy.cpp:::  EDDY::ReplacementManager* EDDY::DoVolumeToVolumeRegistration(const EddyCommandLineOptions&, ECScanManager&):  Exception thrown
Stderr:

Traceback:
	Traceback (most recent call last):
	  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 457, in aggregate_outputs
	    setattr(outputs, key, val)
	  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 325, in validate
	    value = super().validate(objekt, name, value, return_pathlike=True)
	  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
	    self.error(objekt, name, str(value))
	  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/traits/base_trait_handler.py", line 74, in error
	    raise TraitError(
	traits.trait_errors.TraitError: The 'out_corrected' trait of an ExtendedEddyOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/tmp/work/qsiprep_1_0_wf/sub_10619_wf/dwi_preproc_dir_AP_wf/hmc_sdc_wf/eddy/eddy_corrected.nii.gz' <class 'str'> was specified.

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 404, in run
	    outputs = self.aggregate_outputs(runtime)
	  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 464, in aggregate_outputs
	    raise FileNotFoundError(msg)
	FileNotFoundError: No such file or directory '/tmp/work/qsiprep_1_0_wf/sub_10619_wf/dwi_preproc_dir_AP_wf/hmc_sdc_wf/eddy/eddy_corrected.nii.gz' for output 'out_corrected' of a ExtendedEddy interface

Screenshots / relevant information:


Hi @Catherine_Tang,

Please specify a working directory in your qsiprep command (-w in qsiprep, and make sure it is also mounted in the conatiner).

Best,
Steven

HI Steven,

I reran it as instructed, but still got the same error. I closely monitor the realtime output and found a warning that happened before this:

250824-18:46:38,183 nipype.workflow INFO:
         [Node] Executing "seg2msks" <nipype.interfaces.utility.wrappers.Function>
250824-18:46:38,227 nipype.interface WARNING:
         Changing /output/sub-10619/anat/sub-10619_space-ACPC_dseg.nii.gz dtype from uint32 to int16
250824-18:46:38,888 nipype.workflow INFO:
         [Node] Finished "seg2msks", elapsed time 0.704068s.
250824-18:46:40,163 nipype.workflow INFO:
         [Node] Setting-up "qsiprep_1_0_wf.sub_10619_wf.anat_preproc_wf.seg_rpt" in "/work/qsiprep_1_0_wf/sub_10619_wf/anat_preproc_wf/seg_rpt".
250824-18:46:40,186 nipype.workflow INFO:
         [Node] Executing "seg_rpt" <niworkflows.interfaces.reportlets.masks.ROIsPlot>
250824-18:46:59,937 nipype.workflow INFO:
         [Node] Finished "seg_rpt", elapsed time 19.749633s.
250824-18:47:27,120 nipype.workflow INFO:
         [Node] Finished "eddy", elapsed time 571.39545s.
250824-18:47:27,120 nipype.workflow WARNING:
         Storing result file without outputs
250824-18:47:27,122 nipype.workflow WARNING:
         [Node] Error on "qsiprep_1_0_wf.sub_10619_wf.dwi_preproc_dir_AP_wf.hmc_sdc_wf.eddy" (/work/qsiprep_1_0_wf/sub_10619_wf/dwi_preproc_dir_AP_wf/hmc_sdc_wf/eddy)
250824-18:47:28,87 nipype.workflow ERROR:
         Node eddy failed to run on host a90bb992e32c.
250824-18:47:28,97 nipype.workflow ERROR:
         Saving crash info to /output/sub-10619/log/20250824-183112_a44ade5d-c449-4043-9be5-e33817b9d136/crash-20250824-184728-root-eddy-3e71fcb5-ee7e-40ce-9ae6-bde4ad080f25.txt
Traceback (most recent call last):
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 66, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 525, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 643, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 769, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node eddy.

Hi @Catherine_Tang,

In the future please surround your code/terminal outputs with tickmarks (or use the </> button in the text editor) for readability. You can see I edited your post for you this time.

It looks like you didn’t provide the full traceback. May you provide the rest of the error message?

Best,
Steven

Got it, thanks!

250824-18:46:35,923 nipype.workflow INFO:
         [Node] Finished "rigid_acpc_resample_aseg", elapsed time 101.835061s.
250824-18:46:36,186 nipype.workflow INFO:
         [Node] Setting-up "qsiprep_1_0_wf.sub_10619_wf.anat_preproc_wf.acpc_aseg_to_dseg" in "/work/qsiprep_1_0_wf/sub_10619_wf/anat_preproc_wf/acpc_aseg_to_dseg".
250824-18:46:36,189 nipype.workflow INFO:
         [Node] Executing "acpc_aseg_to_dseg" <nipype.interfaces.mrtrix3.connectivity.LabelConvert>
250824-18:46:36,306 nipype.interface WARNING:
         Changing /output/sub-10619/anat/sub-10619_space-ACPC_desc-aseg_dseg.nii.gz dtype from float64 to int16
250824-18:46:37,562 nipype.workflow INFO:
         [Node] Finished "acpc_aseg_to_dseg", elapsed time 1.219924s.
250824-18:46:38,175 nipype.workflow INFO:
         [Node] Setting-up "qsiprep_1_0_wf.sub_10619_wf.anat_preproc_wf.seg2msks" in "/work/qsiprep_1_0_wf/sub_10619_wf/anat_preproc_wf/seg2msks".
250824-18:46:38,183 nipype.workflow INFO:
         [Node] Executing "seg2msks" <nipype.interfaces.utility.wrappers.Function>
250824-18:46:38,227 nipype.interface WARNING:
         Changing /output/sub-10619/anat/sub-10619_space-ACPC_dseg.nii.gz dtype from uint32 to int16
250824-18:46:38,888 nipype.workflow INFO:
         [Node] Finished "seg2msks", elapsed time 0.704068s.
250824-18:46:40,163 nipype.workflow INFO:
         [Node] Setting-up "qsiprep_1_0_wf.sub_10619_wf.anat_preproc_wf.seg_rpt" in "/work/qsiprep_1_0_wf/sub_10619_wf/anat_preproc_wf/seg_rpt".
250824-18:46:40,186 nipype.workflow INFO:
         [Node] Executing "seg_rpt" <niworkflows.interfaces.reportlets.masks.ROIsPlot>
250824-18:46:59,937 nipype.workflow INFO:
         [Node] Finished "seg_rpt", elapsed time 19.749633s.
250824-18:47:27,120 nipype.workflow INFO:
         [Node] Finished "eddy", elapsed time 571.39545s.
250824-18:47:27,120 nipype.workflow WARNING:
         Storing result file without outputs
250824-18:47:27,122 nipype.workflow WARNING:
         [Node] Error on "qsiprep_1_0_wf.sub_10619_wf.dwi_preproc_dir_AP_wf.hmc_sdc_wf.eddy" (/work/qsiprep_1_0_wf/sub_10619_wf/dwi_preproc_dir_AP_wf/hmc_sdc_wf/eddy)
250824-18:47:28,87 nipype.workflow ERROR:
         Node eddy failed to run on host a90bb992e32c.
250824-18:47:28,97 nipype.workflow ERROR:
         Saving crash info to /output/sub-10619/log/20250824-183112_a44ade5d-c449-4043-9be5-e33817b9d136/crash-20250824-184728-root-eddy-3e71fcb5-ee7e-40ce-9ae6-bde4ad080f25.txt
Traceback (most recent call last):
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 66, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 525, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 643, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 769, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node eddy.

Cmdline:
        eddy_cpu  --cnr_maps --field=/work/qsiprep_1_0_wf/sub_10619_wf/dwi_preproc_dir_AP_wf/hmc_sdc_wf/topup/fieldmap_HZ --field_mat=/work/qsiprep_1_0_wf/sub_10619_wf/dwi_preproc_dir_AP_wf/hmc_sdc_wf/topup_to_eddy_reg/topup_reg_image_flirt.mat --flm=quadratic --ff=10.0 --acqp=/work/qsiprep_1_0_wf/sub_10619_wf/dwi_preproc_dir_AP_wf/hmc_sdc_wf/gather_inputs/eddy_acqp.txt --bvals=/data/sub-10619/dwi/sub-10619_dir-AP_dwi.bval --bvecs=/data/sub-10619/dwi/sub-10619_dir-AP_dwi.bvec --imain=/work/qsiprep_1_0_wf/sub_10619_wf/dwi_preproc_dir_AP_wf/pre_hmc_wf/merge_and_denoise_wf/dwi_denoise_dir_AP_dwi_wf/denoiser/sub-10619_dir-AP_dwi_denoised.nii.gz --index=/work/qsiprep_1_0_wf/sub_10619_wf/dwi_preproc_dir_AP_wf/hmc_sdc_wf/gather_inputs/eddy_index.txt --mask=/work/qsiprep_1_0_wf/sub_10619_wf/dwi_preproc_dir_AP_wf/hmc_sdc_wf/transform_mask_to_eddy/topup_imain_corrected_avg_trans_mask_trans_flirt.nii.gz --interp=spline --data_is_shelled --resamp=jac --niter=5 --nthr=8 --nvoxhp=1000 --out=/work/qsiprep_1_0_wf/sub_10619_wf/dwi_preproc_dir_AP_wf/hmc_sdc_wf/eddy/eddy_corrected --repol --slm=linear
Stdout:

        Warning: In a future release the first argument will have to be "diffusion" when using eddy on diffusion data, i.e.
        eddy diffusion --imain='my_ima' --acqp='my_acqp' ...

        Warning: Writing of individual text files will be discontinued in favour of a single .json file in future versions

        EDDY::ReplacementManager::Update: Mismatched DiffStatsVector object
        EDDY::EddyHelperClasses.cpp:::  void EDDY::ReplacementManager::Update(const EDDY::DiffStatsVector&):  Exception thrown
        EDDY::eddy.cpp:::  EDDY::DiffStatsVector EDDY::detect_outliers(const EddyCommandLineOptions&, ScanType, std::shared_ptr<const DWIPredictionMaker>, const NEWIMAGE::volume<float>&, const ECScanManager&, unsigned int, unsigned int, ReplacementManager&):  Exception thrown
        EDDY::eddy.cpp:::  EDDY::DiffStatsVector EDDY::DetectOutliers(const EddyCommandLineOptions&, ScanType, std::shared_ptr<const DWIPredictionMaker>, const NEWIMAGE::volume<float>&, const ECScanManager&, ReplacementManager&):  Exception thrown
        EDDY::eddy.cpp:::  EDDY::ReplacementManager* EDDY::Register(const EddyCommandLineOptions&, ScanType, unsigned int, const std::vector<float, std::allocator<float> >&, SecondLevelECModelType, bool, ECScanManager&, ReplacementManager*, NEWMAT::Matrix&, NEWMAT::Matrix&):  Exception thrown
        EDDY::: Eddy failed with message EDDY::eddy.cpp:::  EDDY::ReplacementManager* EDDY::DoVolumeToVolumeRegistration(const EddyCommandLineOptions&, ECScanManager&):  Exception thrown
Stderr:

Traceback:
        Traceback (most recent call last):
          File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 457, in aggregate_outputs
            setattr(outputs, key, val)
          File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 325, in validate
            value = super().validate(objekt, name, value, return_pathlike=True)
          File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
            self.error(objekt, name, str(value))
          File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/traits/base_trait_handler.py", line 74, in error
            raise TraitError(
        traits.trait_errors.TraitError: The 'out_corrected' trait of an ExtendedEddyOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/work/qsiprep_1_0_wf/sub_10619_wf/dwi_preproc_dir_AP_wf/hmc_sdc_wf/eddy/eddy_corrected.nii.gz' <class 'str'> was specified.

        During handling of the above exception, another exception occurred:

        Traceback (most recent call last):
          File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 404, in run
            outputs = self.aggregate_outputs(runtime)
          File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 464, in aggregate_outputs
            raise FileNotFoundError(msg)
        FileNotFoundError: No such file or directory '/work/qsiprep_1_0_wf/sub_10619_wf/dwi_preproc_dir_AP_wf/hmc_sdc_wf/eddy/eddy_corrected.nii.gz' for output 'out_corrected' of a ExtendedEddy interface```

What kind of DWI data are these? Is it cleanly-shelled data or something else? Providing the b-vals might help clarify. Also what kind of resources (memory, CPU) are you devoting to the job? Was that the first error experiences in the log or did any precede it?

Hi @Catherine_Tang,

Do you mind sharing the new call in which you mount the working directory as @Steven suggested? The error you provide suggests that the following path does not exist inside the container:

traits.trait_errors.TraitError: The 'out_corrected' trait of an ExtendedEddyOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/work/qsiprep_1_0_wf/sub_10619_wf/dwi_preproc_dir_AP_wf/hmc_sdc_wf/eddy/eddy_corrected.nii.gz' <class 'str'> was specified.

I might be wrong about this, but I see in your former post that the working directory is defined under “/tmp/work/…” and not “/work/…”:

Could this be the issue?

HI Steven,

It’s multishell. Attached are the json files for the bvals and bvecs.
dwi.bval.txt (1.2 KB)
dwi.bvec.txt (7.2 KB)

I have 48 CPUs and I didn’t specify the number of CPUs or memory for the job, so in theory I think it should use whatever it can. For context, I’ve run fmriprep (freesurfere included) with this machine before and encountered no error.

The errors I pasted above looked to be the first error experiences, as no other log was written to the output log folder other than this.

Hello,

Yes, I did rerun it with a working directory specified, and can confirm the temp files were written to the newly defined working directory.

Hi @Catherine_Tang,

These b-vals do not look cleanly shelled, with several unique b-values some times very close to each other. The unique b-values are:

150, 300, 350, 500, 650, 800, 950, 1250, 1300, 1450, 1550, 1600, 1750, 1900, 2050, 2100, 2200, 2250, 2550, 2700, 2750, 2850, 2900, 3000, 3050, 3150, 3200, 3300, 3350, 3500, 3800, 3850, 3950, 4000

I do not think Eddy will work well for these data. Try --hmc-model 3dSHORE instead.

You could also try specifying your own custom eddy config file to turn the is_shelled checker by eddy off (false). But I am not sure it will be a good fit for Eddy. Are these DSI data?

Best,
Steven

Hi Steven,

Correct, they are DSI. So you would recommend 3dSHORE instead?

Yes, please use 3dSHORE for DSI data, not eddy. From the QSIPrep documentation:

The 3dSHORE (aka “SHORELine”) option works for multi-shell, Cartesian grid sampling (DSI) and random q-space sampling (CS-DSI).