QSIprep problem: Unable to perform HCP-style merge, different numbers of images

Software Support
1

Summary of what happened:

Hi, experts. I am using QSIPrep to preprocess DTI data from the HCP-development database and wish to include the --distortion-group-merge average option to average the corrected images from opposite PE directions.

I have encountered an issue where one sub is unable to complete the averaging process, with the error message: ‘Unable to perform HCP-style merge, different numbers of images.’ The rest of the subs have completed this process successfully. How should I address this issue for the affected individual without having to restart the entire preprocessing pipeline from scratch?

Command used (and if a helper script was used, a link to the helper script or the command generated):

export HOME=/home/user7/Datapool/
docker run -ti --rm \
    --gpus '"device=0"'  -e NVIDIA_DRIVER_CAPABILITIES=compute,utility -e NVIDIA_VISIBLE_DEVICES=all \
    -v $HOME/data:/data \
    -v $HOME/output:/output \
    -v /home/data/user7/HCD_QSIPrep_working_dir:/working_dir \
    -v ${FREESURFER_HOME}/license.txt:/usr/local/freesurfer/license.txt \
    pennbbl/qsiprep:0.20.0 \
    /data /output participant \
    --fs-license-file /usr/local/freesurfer/license.txt \
    --output-resolution 1.5 \
    --distortion-group-merge average \
    --skip-anat-based-spatial-normalization \
    --eddy-config /output/eddy_params.json \
    --nthreads 24 \
    --omp-nthreads 8 \
    -w /working_dir -v -v

Version:

pennbbl/qsiprep:0.20.0

Environment (Docker, Singularity / Apptainer, custom installation):

Docker

Relevant log outputs (up to 20 lines):

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node distortion_merger.

Traceback:
	Traceback (most recent call last):
	  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 397, in run
	    runtime = self._run_interface(runtime)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/qsiprep/interfaces/dwi_merge.py", line 169, in _run_interface
	    image_pairs, averaged_raw_bvec = find_image_pairs(original_bvecs, bvals, assignments)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/qsiprep/interfaces/dwi_merge.py", line 338, in find_image_pairs
	    raise Exception("Unable to perform HCP-style merge, different numbers of images")
	Exception: Unable to perform HCP-style merge, different numbers of images

Screenshots / relevant information:

2

Hi @ZitengHan,

Can you return the output of tree run on the failing subjects raw data directory and a passing subjects raw data directory so we can compare them?

Thanks,
Steven

1 Like
3

Thanks for your quick reply. The failing subject’s raw data directory:

(zeeresearch) [root@localhost sub-HCD2335344]# tree
.
├── anat
│   ├── sub-HCD2335344_T1w.json
│   ├── sub-HCD2335344_T1w.nii.gz
│   ├── sub-HCD2335344_T2w.json
│   └── sub-HCD2335344_T2w.nii.gz
├── dwi
│   ├── sub-HCD2335344_dir-AP_run-1_dwi.bval
│   ├── sub-HCD2335344_dir-AP_run-1_dwi.bvec
│   ├── sub-HCD2335344_dir-AP_run-1_dwi.json
│   ├── sub-HCD2335344_dir-AP_run-1_dwi.nii.gz
│   ├── sub-HCD2335344_dir-AP_run-1_sbref.json
│   ├── sub-HCD2335344_dir-AP_run-1_sbref.nii.gz
│   ├── sub-HCD2335344_dir-AP_run-2_dwi.bval
│   ├── sub-HCD2335344_dir-AP_run-2_dwi.bvec
│   ├── sub-HCD2335344_dir-AP_run-2_dwi.json
│   ├── sub-HCD2335344_dir-AP_run-2_dwi.nii.gz
│   ├── sub-HCD2335344_dir-AP_run-2_sbref.json
│   ├── sub-HCD2335344_dir-AP_run-2_sbref.nii.gz
│   ├── sub-HCD2335344_dir-PA_run-1_dwi.bval
│   ├── sub-HCD2335344_dir-PA_run-1_dwi.bvec
│   ├── sub-HCD2335344_dir-PA_run-1_dwi.json
│   ├── sub-HCD2335344_dir-PA_run-1_dwi.nii.gz
│   ├── sub-HCD2335344_dir-PA_run-1_sbref.json
│   └── sub-HCD2335344_dir-PA_run-1_sbref.nii.gz
└── fmap
    ├── sub-HCD2335344_dir-AP_epi.json
    ├── sub-HCD2335344_dir-AP_epi.nii.gz
    ├── sub-HCD2335344_dir-PA_epi.json
    └── sub-HCD2335344_dir-PA_epi.nii.gz

3 directories, 26 files

The failing subject’s out directory:

.
├── anat
│   ├── sub-HCD2335344_desc-aseg_dseg.nii.gz
│   ├── sub-HCD2335344_desc-brain_mask.nii.gz
│   ├── sub-HCD2335344_desc-preproc_T1w.nii.gz
│   ├── sub-HCD2335344_dseg.nii.gz
│   ├── sub-HCD2335344_from-orig_to-T1w_mode-image_xfm.txt
│   ├── sub-HCD2335344_from-T1wACPC_to-T1wNative_mode-image_xfm.mat
│   └── sub-HCD2335344_from-T1wNative_to-T1wACPC_mode-image_xfm.mat
├── dwi
│   ├── sub-HCD2335344_confounds.tsv
│   └── sub-HCD2335344_space-T1w_desc-eddy_cnr.nii.gz
└── log
    ├── 20240315-134639_27793e7e-9977-4712-bcfc-1293a88c64d9
    │   └── crash-20240319-213953-root-eddy-5975f398-33d4-4b19-8133-41f1352bf423.txt
    └── 20240322-152738_8372de11-9520-4287-9f7a-a5a4d979ac11
        └── crash-20240328-112207-root-distortion_merger-4336f0a9-bc57-4dfb-8136-fe200d0c9a0f.txt

5 directories, 11 files

The passing subject’s raw data directory:

.
├── anat
│   ├── sub-HCD0042420_T1w.json
│   ├── sub-HCD0042420_T1w.nii.gz
│   ├── sub-HCD0042420_T2w.json
│   └── sub-HCD0042420_T2w.nii.gz
├── dwi
│   ├── sub-HCD0042420_dir-AP_run-1_dwi.bval
│   ├── sub-HCD0042420_dir-AP_run-1_dwi.bvec
│   ├── sub-HCD0042420_dir-AP_run-1_dwi.json
│   ├── sub-HCD0042420_dir-AP_run-1_dwi.nii.gz
│   ├── sub-HCD0042420_dir-AP_run-1_sbref.json
│   ├── sub-HCD0042420_dir-AP_run-1_sbref.nii.gz
│   ├── sub-HCD0042420_dir-AP_run-2_dwi.bval
│   ├── sub-HCD0042420_dir-AP_run-2_dwi.bvec
│   ├── sub-HCD0042420_dir-AP_run-2_dwi.json
│   ├── sub-HCD0042420_dir-AP_run-2_dwi.nii.gz
│   ├── sub-HCD0042420_dir-AP_run-2_sbref.json
│   ├── sub-HCD0042420_dir-AP_run-2_sbref.nii.gz
│   ├── sub-HCD0042420_dir-PA_run-1_dwi.bval
│   ├── sub-HCD0042420_dir-PA_run-1_dwi.bvec
│   ├── sub-HCD0042420_dir-PA_run-1_dwi.json
│   ├── sub-HCD0042420_dir-PA_run-1_dwi.nii.gz
│   ├── sub-HCD0042420_dir-PA_run-1_sbref.json
│   ├── sub-HCD0042420_dir-PA_run-1_sbref.nii.gz
│   ├── sub-HCD0042420_dir-PA_run-2_dwi.bval
│   ├── sub-HCD0042420_dir-PA_run-2_dwi.bvec
│   ├── sub-HCD0042420_dir-PA_run-2_dwi.json
│   ├── sub-HCD0042420_dir-PA_run-2_dwi.nii.gz
│   ├── sub-HCD0042420_dir-PA_run-2_sbref.json
│   └── sub-HCD0042420_dir-PA_run-2_sbref.nii.gz
└── fmap
    ├── sub-HCD0042420_dir-AP_epi.json
    ├── sub-HCD0042420_dir-AP_epi.nii.gz
    ├── sub-HCD0042420_dir-AP_run-02_epi.json
    ├── sub-HCD0042420_dir-AP_run-02_epi.nii.gz
    ├── sub-HCD0042420_dir-AP_run-03_epi.json
    ├── sub-HCD0042420_dir-AP_run-03_epi.nii.gz
    ├── sub-HCD0042420_dir-PA_epi.json
    ├── sub-HCD0042420_dir-PA_epi.nii.gz
    ├── sub-HCD0042420_dir-PA_run-02_epi.json
    ├── sub-HCD0042420_dir-PA_run-02_epi.nii.gz
    ├── sub-HCD0042420_dir-PA_run-03_epi.json
    └── sub-HCD0042420_dir-PA_run-03_epi.nii.gz

3 directories, 40 files

The passing subject’s output directory:

.
├── anat
│   ├── sub-HCD0042420_desc-aseg_dseg.nii.gz
│   ├── sub-HCD0042420_desc-brain_mask.nii.gz
│   ├── sub-HCD0042420_desc-preproc_T1w.nii.gz
│   ├── sub-HCD0042420_dseg.nii.gz
│   ├── sub-HCD0042420_from-orig_to-T1w_mode-image_xfm.txt
│   ├── sub-HCD0042420_from-T1wACPC_to-T1wNative_mode-image_xfm.mat
│   └── sub-HCD0042420_from-T1wNative_to-T1wACPC_mode-image_xfm.mat
├── dwi
│   ├── sub-HCD0042420_confounds.tsv
│   ├── sub-HCD0042420_desc-ImageQC_dwi.csv
│   ├── sub-HCD0042420_dwiqc.json
│   ├── sub-HCD0042420_space-T1w_desc-brain_mask.nii.gz
│   ├── sub-HCD0042420_space-T1w_desc-eddy_cnr.nii.gz
│   ├── sub-HCD0042420_space-T1w_desc-preproc_dwi.b
│   ├── sub-HCD0042420_space-T1w_desc-preproc_dwi.bval
│   ├── sub-HCD0042420_space-T1w_desc-preproc_dwi.bvec
│   ├── sub-HCD0042420_space-T1w_desc-preproc_dwi.nii.gz
│   ├── sub-HCD0042420_space-T1w_desc-preproc_dwi.txt
│   └── sub-HCD0042420_space-T1w_dwiref.nii.gz
└── log
    └── 20240315-134639_27793e7e-9977-4712-bcfc-1293a88c64d9
        └── crash-20240319-211222-root-eddy-bdba106c-6adb-46a1-8688-e942d555a616.txt

4 directories, 19 files
4

It seems that I need to delete all the *_dir-AP_run-2* files from the failing subjects raw data directory.

5

I wouldn’t delete data. You can just remove the distortion-group-merge average argument.

7

I will try. In that case, will subs who have previously successfully completed the process restart the entire QSIPrep process?

8

Hi @ZitengHan,

Yes, a lot of steps will need to be repeated for other subjects.

Best,
Steven