Qsiprep: Stuck on eddy with hcp-aging data

Software Support
,
1

Summary of what happened:

Hi, everyone! I’m testing a sample subject of HCP-Aging in Qsiprep for preprocessing and reconstruction. But Qsiprep seems get stuck on eddy_openmp for about 10 hours. And I found that under the $work directory, the report about Node eddy contains nothing but Original Inputs. I’m not sure whether it’s normal. By the way, I also tried to use eddy_cuda but failed due to mismatch of cuda version.

Command used (and if a helper script was used, a link to the helper script or the command generated):

singularity run --cleanenv /biolab/zhouzx/Code/simg/qsiprep/qsiprep.sif \
    /biolab/zhouzx/Code/Data/hcpa_bids /biolab/zhouzx/Code/Data/hcpa-test-out1 \
    participant \
    --participant-label HCD0001305 \
    --fs-license-file /biolab/zhouzx/freesurfer/freesurfer/license.txt \
    -w  /biolab/zhouzx/Code/Data/hcpa-test-work1 \
    --write-graph \
    --freesurfer-input /biolab/zhouzx/Code/Data/hcpa-out/sourcedata/freesurfer \
    --output-resolution 1.2 \
    -vv \
    --recon_spec mrtrix_multishell_msmt_ACT-hsvs \
    --mem-mb 180000 \
    --distortion-group-merge average \

And I use --distortion-group-merge average according to an issue. Besides it, most of the args are default.

Version:

0.19.0

Environment (Docker, Singularity, custom installation):

Singularity

Data formatted according to a validatable standard? Please provide the output of the validator:

tree
tree1632×1235 326 KB

I converted the unprocessed data to bids by modifying hcp2bids which can not be used to hcp-a directly. And one step I’m not sure is that I use the FieldMap for both dir99
and dir98. For example, the json of a fieldmap named with AP will be written as:

    "IntendedFor": [
        "dwi/sub-HCD0001305_acq-dir99_dir-AP_dwi.nii.gz",
        "dwi/sub-HCD0001305_acq-dir98_dir-AP_dwi.nii.gz"
    ]

Besides this, I think rest of the setting should be normal and it pass the exam by bids validator.

Relevant log outputs (up to 20 lines):

Output of top:

PROCESS USER      PR  NI    VIRT    RES    SHR �  %CPU %MEM     TIME+ COMMAND 
35279 zhouzx    20   0 12.807g 0.012t  11792 R 793.8  4.8   4519:48 eddy_openmp

Output of Qsiprep:

231002-00:42:10,646 nipype.workflow INFO:
         [Node] Setting-up "qsiprep_wf.single_subject_HCD0001305_wf.dwi_preproc_wf.hmc_sdc_wf.eddy" in "/biolab/zhouzx/Code/Data/hcpa-test-work1/qsiprep_wf/single_subject_HCD0001305_wf/dwi_preproc_wf/hmc_sdc_wf/eddy".
231002-00:42:10,648 nipype.workflow INFO:
         [Node] Executing "eddy" <qsiprep.interfaces.eddy.ExtendedEddy>
231002-00:42:11,796 nipype.workflow INFO:
         [MultiProc] Running 1 tasks, and 0 jobs ready. Free memory (GB): 175.58/175.78, Free processors: 56/64.
                     Currently running:
                       * qsiprep_wf.single_subject_HCD0001305_wf.dwi_preproc_wf.hmc_sdc_wf.eddy

Screenshots / relevant information:

report_eddy.txt (2.6 KB)

2

Hi @taka2223,

The HCP-A is a lot of data per subject, try just letting eddy run for longer.

Best,
Steven

1 Like
3

Thank you very much @Steven. Eddy has successfully finished after about 12 hours. And I wonder if I could utilize cuda and eddy_cuda to accelerate the procedure. What I have tried is the following command:

#!/bin/bash
SINGULARITYENV_LD_LIBRARY_PATH=/usr/local/cuda-11.0/extras/CUPTI/lib64:/usr/local/cuda-11.0/lib64:/usr/local/cuda-11.0/targets/x86_64-linux/lib/stubs
singularity run --cleanenv --nv /biolab/zhouzx/Code/simg/qsiprep/qsiprep.sif \
    /biolab/zhouzx/Code/Data/hcpa_bids /biolab/zhouzx/Code/Data/hcpa-test-out1 \
    participant \
    --participant-label HCD0001305 \
    --fs-license-file /biolab/zhouzx/freesurfer/freesurfer/license.txt \
    -w  /biolab/zhouzx/Code/Data/hcpa-test-work1 \
    --write-graph \
    --freesurfer-input /biolab/zhouzx/Code/Data/hcpa-out/sourcedata/freesurfer \
    --output-resolution 1.2 \
    -vv \
    --recon_spec mrtrix_multishell_msmt_ACT-hsvs \
    --mem-mb 180000 \
    --distortion-group-merge average \
    --eddy_config /biolab/zhouzx/Code/simg/eddy_params.json

And the GPU I used is A100 whose cuda version is 11.4 but Qsiprep or say eddy seems require cuda version below 10. The output log error is

eddy_cuda: error while loading shared libraries: libcublas.so.10: cannot open shared object file: No such file or directory

What do you think I should do in this situation?

4

This has been reported but there was no answer so far on this thread:

My advice would be to try version 0.18.1 until this issue is fixed for 0.19.0

1 Like