QSIPrep gets stuck at plot_connectivity step

Software Support
1

Summary of what happened:

Hi! Not sure, if this issue is related to this one, that’s why I am opening a new one. I am running QSIPrep (including preprocessing + reconstruction/fibertracking) on my dataset, but for certain subjects QSIprep get’s stuck at the plot_connectivity step.

Command used (and if a helper script was used, a link to the helper script or the command generated):

This is the .sbatch script (which calls a slurm job for one subject):

#!/bin/bash
#SBATCH --job-name=qsiprep   # name of the job
#SBATCH --cpus-per-task=16    # require 8 cores
#SBATCH --mem=30000          # reqiure max. 30000 MB
#SBATCH --nodelist=zilxhp05
#SBATCH --output=/pandora/data/INDICATE/BIDS_INDI/derivatives/johannes/output/slurm-qsiprep-%J.out
participant_label=${1}
#############################################################
## Run qsiprep reconstruction for one subject as a slurm job
#############################################################
# analysis-settings
data_dir_server="/pandora/data/INDICATE/BIDS_INDI"
cache_dir_server="/pandora/data/INDICATE/BIDS_INDI/derivatives/johannes/cache/qsiprep_tmp"
custom_atlases="/pandora/data/INDICATE/BIDS_INDI/derivatives/johannes/data/qsirecon_atlases"
output_dir_server="/pandora/data/INDICATE/BIDS_INDI/derivatives/johannes/output"
recon_spec_dir="/pandora/data/INDICATE/BIDS_INDI/derivatives/johannes/data/recon_spec"
nthreads=64
omp_nthreads=16
mem_mb=30000
# run qsiprep and qsirecon
singularity run --cleanenv \
-B $custom_atlases:/atlas/qsirecon_atlases \
-B $data_dir_server \
-B $output_dir_server \
-B $recon_spec_dir \
/csp-tools/singularity_images/qsiprep_0.16.1.sif \
$data_dir_server/rawdata $output_dir_server participant \
--recon_spec $recon_spec_dir/dsi_studio_gqi_with_GlasserTianS1.json \
--output-resolution 1 \
--skip-bids-validation \
--fs-license-file $data_dir_server/derivatives/license.txt \
--work-dir $cache_dir_server \
--nthreads $nthreads \
--omp_nthreads $omp_nthreads \
--mem_mb $mem_mb \
--participant_label ${participant_label}

Each slurm job is spawned using this script:

#!/bin/bash
raw_data=/pandora/data/INDICATE/BIDS_INDI/rawdata
cd $raw_data
for sub in sub-INDI*; do
  sbatch /pandora/data/INDICATE/BIDS_INDI/derivatives/johannes/code/qsiprep_subject.sbatch $sub
done

Version:

qisprep: 0.16.1
singularity: 3.7.1

Environment (Docker, Singularity, custom installation):

  • Singularity

Data formatted according to a validatable standard? Please provide the output of the validator:

  • Dataset confirms to BIDS-standard but there are some non-BIDS-files in there, so I use --skip-bids-validation

Relevant log outputs (up to 20 lines):

230510-12:02:45,436 nipype.workflow INFO:
	 [Node] Setting-up "qsirecon_wf.sub-INDI132M2_dsistudio_pipeline.sub_INDI132M2_ses_01_acq_multishell_dir_pa_space_T1w_desc_preproc_recon_wf.streamline_connectivity.plot_connectivity" in "/pandora/data/INDICATE/BIDS_INDI/derivatives/johannes/cache/qsiprep_tmp/qsirecon_wf/sub-INDI132M2_dsistudio_pipeline/sub_INDI132M2_ses_01_acq_multishell_dir_pa_space_T1w_desc_preproc_recon_wf/streamline_connectivity/plot_connectivity".
230510-12:02:45,442 nipype.workflow INFO:
	 [Node] Outdated cache found for "qsirecon_wf.sub-INDI132M2_dsistudio_pipeline.sub_INDI132M2_ses_01_acq_multishell_dir_pa_space_T1w_desc_preproc_recon_wf.streamline_connectivity.plot_connectivity".
230510-12:02:45,504 nipype.workflow INFO:
	 [Node] Executing "plot_connectivity" <qsiprep.interfaces.reports.ConnectivityReport>
230510-12:02:48,175 nipype.workflow INFO:
	 [Node] Finished "plot_connectivity", elapsed time 2.6388220000000002s.

Screenshots / relevant information:

I checked the output for the subjects where this happens, but I cannot see any errors. I am using SLURM to spawn one job for each subject but as some jobs never finish QSIPrep does not get executed for the remaining subjects. E.g. this is the current folder structure of a subject where the job is currently stuck:

(csp_wiesner_johannes) johannes.wiesner@zilxhp01:/pandora/data/INDICATE/BIDS_INDI/derivatives/johannes/output/qsiprep/sub-INDI132M2$ tree
.
├── anat
│   ├── sub-INDI132M2_desc-brain_mask.nii.gz
│   ├── sub-INDI132M2_desc-preproc_T1w.nii.gz
│   ├── sub-INDI132M2_dseg.nii.gz
│   ├── sub-INDI132M2_from-MNI152NLin2009cAsym_to-T1w_mode-image_xfm.h5
│   ├── sub-INDI132M2_from-T1w_to-MNI152NLin2009cAsym_mode-image_xfm.h5
│   ├── sub-INDI132M2_label-CSF_probseg.nii.gz
│   ├── sub-INDI132M2_label-GM_probseg.nii.gz
│   ├── sub-INDI132M2_label-WM_probseg.nii.gz
│   ├── sub-INDI132M2_space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz
│   ├── sub-INDI132M2_space-MNI152NLin2009cAsym_desc-preproc_T1w.nii.gz
│   ├── sub-INDI132M2_space-MNI152NLin2009cAsym_dseg.nii.gz
│   ├── sub-INDI132M2_space-MNI152NLin2009cAsym_label-CSF_probseg.nii.gz
│   ├── sub-INDI132M2_space-MNI152NLin2009cAsym_label-GM_probseg.nii.gz
│   └── sub-INDI132M2_space-MNI152NLin2009cAsym_label-WM_probseg.nii.gz
├── figures
│   ├── sub-INDI132M2_seg_brainmask.svg
│   ├── sub-INDI132M2_ses-01_acq-multishell_dir-pa_carpetplot.svg
│   ├── sub-INDI132M2_ses-01_acq-multishell_dir-pa_coreg.svg
│   ├── sub-INDI132M2_ses-01_acq-multishell_dir-pa_desc-resampled_b0ref.svg
│   ├── sub-INDI132M2_ses-01_acq-multishell_dir-pa_dwi_denoise_ses_01_acq_multishell_dir_pa_dwi_wf_biascorr.svg
│   ├── sub-INDI132M2_ses-01_acq-multishell_dir-pa_dwi_denoise_ses_01_acq_multishell_dir_pa_dwi_wf_denoising.svg
│   ├── sub-INDI132M2_ses-01_acq-multishell_dir-pa_sampling_scheme.gif
│   ├── sub-INDI132M2_ses-02_acq-multishell_dir-pa_carpetplot.svg
│   ├── sub-INDI132M2_ses-02_acq-multishell_dir-pa_coreg.svg
│   ├── sub-INDI132M2_ses-02_acq-multishell_dir-pa_desc-resampled_b0ref.svg
│   ├── sub-INDI132M2_ses-02_acq-multishell_dir-pa_dwi_denoise_ses_02_acq_multishell_dir_pa_dwi_wf_biascorr.svg
│   ├── sub-INDI132M2_ses-02_acq-multishell_dir-pa_dwi_denoise_ses_02_acq_multishell_dir_pa_dwi_wf_denoising.svg
│   ├── sub-INDI132M2_ses-02_acq-multishell_dir-pa_sampling_scheme.gif
│   └── sub-INDI132M2_t1_2_mni.svg
├── ses-01
│   ├── anat
│   │   └── sub-INDI132M2_ses-01_from-orig_to-T1w_mode-image_xfm.txt
│   └── dwi
│       ├── sub-INDI132M2_ses-01_acq-multishell_dir-pa_confounds.tsv
│       ├── sub-INDI132M2_ses-01_acq-multishell_dir-pa_desc-ImageQC_dwi.csv
│       ├── sub-INDI132M2_ses-01_acq-multishell_dir-pa_desc-SliceQC_dwi.json
│       ├── sub-INDI132M2_ses-01_acq-multishell_dir-pa_dwiqc.json
│       ├── sub-INDI132M2_ses-01_acq-multishell_dir-pa_space-T1w_desc-brain_mask.nii.gz
│       ├── sub-INDI132M2_ses-01_acq-multishell_dir-pa_space-T1w_desc-eddy_cnr.nii.gz
│       ├── sub-INDI132M2_ses-01_acq-multishell_dir-pa_space-T1w_desc-preproc_dwi.b
│       ├── sub-INDI132M2_ses-01_acq-multishell_dir-pa_space-T1w_desc-preproc_dwi.bval
│       ├── sub-INDI132M2_ses-01_acq-multishell_dir-pa_space-T1w_desc-preproc_dwi.bvec
│       ├── sub-INDI132M2_ses-01_acq-multishell_dir-pa_space-T1w_desc-preproc_dwi.nii.gz
│       └── sub-INDI132M2_ses-01_acq-multishell_dir-pa_space-T1w_dwiref.nii.gz
└── ses-02
    ├── anat
    │   └── sub-INDI132M2_ses-02_from-orig_to-T1w_mode-image_xfm.txt
    └── dwi
        ├── sub-INDI132M2_ses-02_acq-multishell_dir-pa_confounds.tsv
        ├── sub-INDI132M2_ses-02_acq-multishell_dir-pa_desc-ImageQC_dwi.csv
        ├── sub-INDI132M2_ses-02_acq-multishell_dir-pa_desc-SliceQC_dwi.json
        ├── sub-INDI132M2_ses-02_acq-multishell_dir-pa_dwiqc.json
        ├── sub-INDI132M2_ses-02_acq-multishell_dir-pa_space-T1w_desc-brain_mask.nii.gz
        ├── sub-INDI132M2_ses-02_acq-multishell_dir-pa_space-T1w_desc-eddy_cnr.nii.gz
        ├── sub-INDI132M2_ses-02_acq-multishell_dir-pa_space-T1w_desc-preproc_dwi.b
        ├── sub-INDI132M2_ses-02_acq-multishell_dir-pa_space-T1w_desc-preproc_dwi.bval
        ├── sub-INDI132M2_ses-02_acq-multishell_dir-pa_space-T1w_desc-preproc_dwi.bvec
        ├── sub-INDI132M2_ses-02_acq-multishell_dir-pa_space-T1w_desc-preproc_dwi.nii.gz
        └── sub-INDI132M2_ses-02_acq-multishell_dir-pa_space-T1w_dwiref.nii.gz
2

Hi @JohannesWiesner,

I have relabeled your post as Software Support and added in the corresponding post template. Please post software-related topics under this category in the future.

Filling in the missing information by editing your post may help us help you. Also, I think plot_connectivity is one of, if not the last, step in the process, so I am also curious as to what files are missing from the subjects who get stuck?

Best,
Steven

1 Like
3

Are you running with singluarity/apptainer? If so did you use the --containall flag? I wonder if your user’s matplotlib is being used instead of the one included in the container.

4

Thanks for the hint, updated my question correspondingly!

5

See update: Currently, I only use singularity run --cleanenv to make sure that singularity does not use any host software (at least that’s how I learned it from fmriprep)

6

a number of users have been having trouble with this because singularity can keep your user and home info and sometimes there are python libraries in there. cleanenv will only exclude environment variables, containall prevents auto configuring the home directory

7

Ah, interesting! I will try this next week and report if it worked.

8

@mattcieslak , @Steven : I couldn’t access our server for a while, sorry for the late reply. I added --containall to my script, but it didn’t change anything. There are currently three jobs running (for ~48hrs) and all of these are again stuck at the "plot_connectivity" step.