--combine-all-dwis not recognized

Software Support
1

Summary of what happened:

hi qsiprep experts! i’m running qsiprep 0.22.0 and have 2 runs of DWI data (ABCD-style pulse seq) that both have 103 directions but opposite phase encode PA and AP. i believe this is the HPC approach to distortion correction and i see the instructions of these 2 commands to add. i’ve done this below but it breaks. any idea? if i remove that and the “–distortion-group-merge average” it runs fine. let me know if you need any more info, thanks, but also sorry in advance if i’m missing something obvious
-sam

Command used (and if a helper script was used, a link to the helper script or the command generated):

./qsiprep_0.22.0.sif motsf005 /path/user participant \
    --fs-license-file /path/user/license.txt \
    --output-resolution 1.7                 \
    --work-dir ${WORK_DIR}                  \
    --participant_label ${subjectID}        \
    --distortion-group-merge average        \
    --combine-all-dwis                      \
    --nthreads 4                            \
    --omp-nthreads 4                        \
    --stop-on-first-crash                   \
    -v -v

Version:

0.22.0

Environment (Docker, Singularity / Apptainer, custom installation):

Singularity

Data formatted according to a validatable standard? Please provide the output of the validator:

PASTE VALIDATOR OUTPUT HERE

Relevant log outputs (up to 20 lines):

qsiprep: error: unrecognized arguments: --combine-all-dwis

Screenshots / relevant information:

2

Hi @stbmtx,

That flag is deprecated, as combining is now default. There is now a --separate_all_dwis flag that tells qsiprep to not combine them.

Best,
Steven

3

thanks! so basically i just remove that, keep “–distortion-group-merge average” and it’s HCP-appropriate?

4

Yes that should be the case, as long as the gradients are the same for both directions.

1 Like
5

hello @Steven unfortunately that’s not working. I’ve run it a few ways now to try to zero in on the issue and here are some more details.

before all this i ran dcm2bids_3.2.0 to get the data into BIDS format and didn’t see any errors during that process. however if i remove “–skip_bids_validation” from my command it breaks with

*1: [ERR] Files with such naming scheme are not part of BIDS specification....*
and
*2: [ERR] Subject label in the filename doesn't match with the path of the file....*

otherwise, this finishes successfully:

./qsiprep_0.22.0.sif motsf005 /wynton/protected/home/rad-wynton-only/storrisi participant \
    --fs-license-file /wynton/protected/home/rad-wynton-only/storrisi/license.txt \
    --output-resolution 1.7                 \
    --work-dir ${WORK_DIR}                  \
    --participant_label ${subjectID}        \
    --distortion-group-merge none        \
    --skip_bids_validation                  \
    --nthreads 4                            \
    --omp-nthreads 4                        \
    --stop-on-first-crash                   \
    -v -v

so i think the BIDS validating thing is orthogonal to my main point but let me know if i have to go back and change that.

as for the current topic (HCP-style DWI processing with reverse PE), you can see here i have distortion-group-merge “none”, which works. but if i specify either “average” or “concat” it breaks. here’s some output error after having specified “average”, although i think it’s the same error for either:

241028-14:55:07,672 nipype.workflow IMPORTANT:
     Building QSIPrep's workflow:
           * BIDS dataset path: /path/storrisi/motsf005.
           * Participant list: ['005'].
           * Run identifier: 20241028-145425_7e884e66-f9eb-4adb-a28d-90547f8bc132.
241028-14:55:07,769 nipype.workflow INFO:
     Running nonlinear normalization to template
241028-14:55:07,798 nipype.workflow INFO:
     Combining all 2 dwis within the single available session
241028-14:55:07,827 nipype.workflow INFO:
     [{'dwi_series': ['/path/storrisi/motsf005/sub-005/dwi/sub-005_dir-PA_dwi.nii.gz'], 'dwi_series_pedir': 'j', 'fieldmap_info': {'suffix': 'rpe_series', 'rpe_series': ['/path/storrisi/motsf005/sub-005/dwi/sub-005_dir-AP_dwi.nii.gz']}, 'concatenated_bids_name': 'sub-005'}]
Process Process-2:
Traceback (most recent call last):
  File "/opt/conda/envs/qsiprep/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/conda/envs/qsiprep/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/qsiprep/cli/workflow.py", line 160, in build_workflow
    retval["workflow"] = workflow_builder()
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/qsiprep/workflows/base.py", line 82, in init_qsiprep_wf
    single_subject_wf = init_single_subject_wf(subject_id)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/qsiprep/workflows/base.py", line 295, in init_single_subject_wf
    merging_group_workflows[merged_group] = init_distortion_group_merge_wf(
TypeError: init_distortion_group_merge_wf() got an unexpected keyword argument 'template'

full disclosure i thought i was running QSIprep 0.22 but it technically says
Running QSIPrep version 0.22.1.dev0+gd9279db.d20240719

not sure if that’s relevant. and thanks again

-s

6

Hi @stbmtx,

In the future, please use the code formatting instead of blockquotes for terminal output / code (you can see I’ve been editing your posts to change it).

This is probably unrelated to current issue, but if you were to return full output I could help diagnose.

Regarding your error, you can see an issue was already raised for it: error when using option --distortion-group-merge · Issue #797 · PennLINC/qsiprep · GitHub . Have you tried the newest version? Also, I personally don’t like averaging the images, I prefer to just concatenate (as default) such as to not lose information.

Best,
Steven

7

hello @Steven

that thread seems incredibly relevant to my situation, thank you for sharing.
i built the ‘unstable’ version and my command is:

./qsiprep_unstable.sif motsf005 /wynton/protected/home/rad-wynton-only/storrisi participant \
    --fs-license-file /wynton/protected/home/rad-wynton-only/storrisi/license.txt \
    --output-resolution 1.7                 \
    --work-dir ${WORK_DIR}                  \
    --participant_label ${subjectID}        \
    --distortion-group-merge concat        \
    --skip_bids_validation                  \
    --nthreads 4                            \
    --omp-nthreads 4                        \
    --stop-on-first-crash                   \
    -v -v

in which the only differences to previously successful commands are referring to a different .sif and trying to specify --distortion-group-merge concat

however this breaks in a new way:

241028-18:55:27,918 nipype.workflow INFO:
     [{'dwi_series': ['/wynton/protected/home/rad-wynton-only/storrisi/motsf005/sub-005/dwi/sub-005_dir-PA_dwi.nii.gz'], 'dwi_series_pedir': 'j', 'fieldmap_info': {'suffix': 'rpe_series', 'rpe_series': ['/wynton/protected/home/rad-wynton-only/storrisi/motsf005/sub-005/dwi/sub-005_dir-AP_dwi.nii.gz']}, 'concatenated_bids_name': 'sub-005'}]
Process Process-2:
Traceback (most recent call last):
  File "/opt/conda/envs/qsiprep/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/conda/envs/qsiprep/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/qsiprep/cli/workflow.py", line 122, in build_workflow
    retval["workflow"] = init_qsiprep_wf()
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/qsiprep/workflows/base.py", line 82, in init_qsiprep_wf
    single_subject_wf = init_single_subject_wf(subject_id)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/qsiprep/workflows/base.py", line 303, in init_single_subject_wf
    merging_group_workflows[merged_group] = init_distortion_group_merge_wf(
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/qsiprep/workflows/dwi/distortion_group_merge.py", line 193, in init_distortion_group_merge_wf
    SeriesQC(output_file_name=config.execution.output_prefix), name="series_qc"
AttributeError: type object 'execution' has no attribute 'output_prefix'

should i have not built the “unstable” version? that version now prints as

Running QSIPrep version 0.23.1.dev6+g551bacd.d20241025

note that i built it a simple way on the particular HPC i’m using:

apptainer pull docker://pennbbl/qsiprep:unstable

thanks for your continued help

8

Hi @stbmtx,

Kind reminder to please format code and terminal outputs as code (</> button), instead of block quotes, for better readability. You can see I edited your last post again.

You can try the latest version, 0.23.0, but I would just leave the distortion group merge argument as default.

Best,
Steven

9

apologies for the formatting bungle

i tried the exact version 0.23.0 and it still doesn’t work

so i suppose i’m left having to --separate-all-dwis and no merging is possible for my 2-run data (one AP, one PA)?

i wonder why the main developer closed that former thread when the problem does not seem solved. perhaps i should wait for a more stable upcoming version?

being new to DWI, will -separate-all-dwis (A) be flat out incorrect with my data, (B) just mean i have to specify downstream reconstruction strategies a certain way, or (C) something else?

10

Hi @stbmtx,

This error may not be too hard to fix, and in fact you can simply try patching in a fix without waiting for a release. Download this file (qsiprep/qsiprep/workflows/dwi/distortion_group_merge.py at 551bacd66acd39d64bbf03b3a013921d71d9ac6a · PennLINC/qsiprep · GitHub) and change the output_file_name argument in line 193 to something else (you can even try an empty string, ""). It looks like what was there is referencing a config argument that has since been removed. When you have the file changed and ready, in your singularity preamble, add a bind string -B /PATH/TO/YOUR/distortion_group_merge.py:/opt/conda/envs/qsiprep/lib/python3.10/site-packages/qsiprep/workflows/dwi/distortion_group_merge.py, and run again.

If that works, you could even open a PR in the repository!

Even if this doesn’t work…

You still do not have to separate the images. It is just that the SDC will not be applied from the reverse encoded images. But you could in theory extract the average b0 or most representative/“best” b0 from each image and save it as a reverse phase encoded EPI in the fmap folder, re-enabling SDC. Last resort, though.

Not incorrect, unless you have a specific downstream use case that warrants separate preprocessed images.

Nope, QSIRecon should adapt to having multiple output images, if that’s what you need (see point A).

Best,
Steven