Running MRIQC group w/o bids_dir

Software Support
1

Summary of what happened:

I am running MRIQC version 23.1.0rc0. The problem that I am running into is that the group level analysis requires and queries the bids_dir. In my case, the bids_dir will have some validation issues that are known. In the output_dir I have all of my individual subjects MRIQC outputs (those without errors are not included). So I am only trying to run the summary on these that I have output. Given the MRIQC queries and requires the bids_dir, I get an error.

Command used (and if a helper script was used, a link to the helper script or the command generated):

singularity run --cleanenv \
-B ${bids_dir}:/bids_dir \
-B ${mriqc_in}:/output_dir \
-B ${work_dir}/work_dir/mriqc_v23_1_0_grp:/wd \
${sif_img} \
/bids_dir /output_dir group \
--session-id ${session} \
-vv \
-w /wd

Version:

23.1.0rc0

Environment (Docker, Singularity, custom installation):

Singularity

Data formatted according to a validatable standard? Please provide the output of the validator:

Relevant log outputs (up to 20 lines):

Does anyone have a workaround that they have used? I tried adding --participants-labels ${sub}, whereby sub is a list of IDs without prefix ‘sub-’ (sub=$(cat ${dir}/mriqc_completed.csv | awk -F'\t' '{ print $1 }')) but that gave me an error

mriqc: error: Querying BIDS dataset at </bids_dir> got an empty result.
Even though the $sub value is a space separated variable of the IDs

Screenshots / relevant information:

2

Hi @mdemi ,

MRIQC is a bids-app and this requires a bids-valid directory to run. Can you print the output of tree on your root directory?

Best,
Steven

3

The bids_dir has 10k subjects so the tree on bids-root directory would be quite large. Would an abbreviated be okay?

4

Sure, maybe just tree on one subject’s directory.

5

Here is n = 1 tree of subject in bids-root dir:

sub-EXAMPLE
└── ses-SESSIONA
├── anat
│ ├── sub-EXAMPLE_ses-SESSIONA_T1w.json
│ ├── sub-EXAMPLE_ses-SESSIONA_T1w.nii.gz
│ ├── sub-EXAMPLE_ses-SESSIONA_T2w.json
│ └── sub-EXAMPLE_ses-SESSIONA_T2w.nii.gz
├── dwi
│ ├── sub-EXAMPLE_ses-SESSIONA_dwi.bval
│ ├── sub-EXAMPLE_ses-SESSIONA_dwi.bvec
│ ├── sub-EXAMPLE_ses-SESSIONA_dwi.json
│ └── sub-EXAMPLE_ses-SESSIONA_dwi.nii.gz
├── fmap
│ ├── sub-EXAMPLE_ses-SESSIONA_acq-dwi_dir-AP_epi.json
│ ├── sub-EXAMPLE_ses-SESSIONA_acq-dwi_dir-AP_epi.nii.gz
│ ├── sub-EXAMPLE_ses-SESSIONA_dir-AP_run-01_epi.json
│ ├── sub-EXAMPLE_ses-SESSIONA_dir-AP_run-01_epi.nii.gz
│ ├── sub-EXAMPLE_ses-SESSIONA_dir-AP_run-02_epi.json
│ ├── sub-EXAMPLE_ses-SESSIONA_dir-AP_run-02_epi.nii.gz
│ ├── sub-EXAMPLE_ses-SESSIONA_dir-AP_run-03_epi.json
│ ├── sub-EXAMPLE_ses-SESSIONA_dir-AP_run-03_epi.nii.gz
│ ├── sub-EXAMPLE_ses-SESSIONA_dir-AP_run-04_epi.json
│ ├── sub-EXAMPLE_ses-SESSIONA_dir-AP_run-04_epi.nii.gz
│ ├── sub-EXAMPLE_ses-SESSIONA_dir-PA_run-01_epi.json
│ ├── sub-EXAMPLE_ses-SESSIONA_dir-PA_run-01_epi.nii.gz
│ ├── sub-EXAMPLE_ses-SESSIONA_dir-PA_run-02_epi.json
│ ├── sub-EXAMPLE_ses-SESSIONA_dir-PA_run-02_epi.nii.gz
│ ├── sub-EXAMPLE_ses-SESSIONA_dir-PA_run-03_epi.json
│ ├── sub-EXAMPLE_ses-SESSIONA_dir-PA_run-03_epi.nii.gz
│ ├── sub-EXAMPLE_ses-SESSIONA_dir-PA_run-04_epi.json
│ └── sub-EXAMPLE_ses-SESSIONA_dir-PA_run-04_epi.nii.gz
└── func
├── sub-EXAMPLE_ses-SESSIONA_task-rest_run-01_bold.json
├── sub-EXAMPLE_ses-SESSIONA_task-rest_run-01_bold.nii.gz
├── sub-EXAMPLE_ses-SESSIONA_task-rest_run-02_bold.json
├── sub-EXAMPLE_ses-SESSIONA_task-rest_run-02_bold.nii.gz
├── sub-EXAMPLE_ses-SESSIONA_task-rest_run-03_bold.json
├── sub-EXAMPLE_ses-SESSIONA_task-rest_run-03_bold.nii.gz
├── sub-EXAMPLE_ses-SESSIONA_task-rest_run-04_bold.json
├── sub-EXAMPLE_ses-SESSIONA_task-rest_run-04_bold.nii.gz
├── sub-EXAMPLE_ses-SESSIONA_task-SST_run-01_bold.json
├── sub-EXAMPLE_ses-SESSIONA_task-SST_run-01_bold.nii.gz
├── sub-EXAMPLE_ses-SESSIONA_task-SST_run-02_bold.json
└── sub-EXAMPLE_ses-SESSIONA_task-SST_run-02_bold.nii.gz

6 directories, 38 files

6

What are the BIDS validation issues?

7

Working on the fixes, but approx 250 subjects have .json and/or .nii.gz files that are empty. Which causes an MRIQC decoding error

8

One workaround to try is making a fake BIDS directory containing just symlinks/shortcuts with subject data you want to analyze, and then running MRIQC on that. That way it will not be taking up much extra space on your system (presumably a HPC if you have 10k subjects). Just make sure to bind the original and symlinked drives in Singularity.

1 Like
9

Thanks for the recommendation, Steven! When you say “just make sure to bind the original and symlinked drives in singularity”, what do you mean by that? Sorry, semi-singularity newb.

10 
singularity run -e -B ${original_bids_dir},${fake_bids_dir}

Basically, you will input your fake directory to be analyzed, but Singularity needs to be able to find the original data (i.e. the target of the shortcuts), so both drives need to be available.

1 Like
11

Edit: I’m a bad reader and should feel bad. Sorry for the noise.

1 Like
12

Created the symbolic links using:
for folder in $(find ${input}/${sub}/ses-${ses} -mindepth 1 -maxdepth 1 -type d -not -name "dwi" ) ; do ln -s "$folder" "${sub}/ses-${ses}/"; done; done

Got an error similar to when using the --participant-labels, e.g., mriqc: error: Querying BIDS dataset at </bids_dir> got an empty result. Please, check out your currently set filters. Retrying atm by removing the --session_id

Perhaps something wrong with my singularity command?

singularity run --cleanenv \
-B ${in_dir}:${bids_dir} \
-B ${bids_dir}:/bids_dir \
-B ${mriqc_in}:/output_dir \
-B ${work_dir}/work_dir/mriqc_v23_1_0_grp:/wd \
${sif_img} \
/bids_dir /output_dir group \
--session-id ses-2YearFollowUpYArm1 \
-vv \
-w /wd
13

Hi @mdemi,

Can you try running singularity shell into your container (using the same -e and -B arguments) and do ls /bids_dir and see if your BIDS folder shows up as expected?

Best,
Steven

15

Thanks – tried it and I get a list of subject directories as expected…

singularity shell --cleanenv \
-B ${in_dir}:${bids_dir} \
-B ${bids_dir}:/bids_dir \
-B ${mriqc_in}:/output_dir \
-B ${work_dir}/work_dir/mriqc_v23_1_0_grp:/wd \
${sif_img}

However, ls /bids_dir/sub-EXAMPLE/ses-EXAMPLE/anat shows no files. Not sure if that is expected.

16

Which of your names arguments are the original and fake bids directories?

17

@Steven – The original of bids subject directories: in_dir. The fake sym linked of subject direcotirs: bids_dir

18

Ah so here’s the thing. Your symlinks may be pointing to something with full path of your original bids directory, BUT in the container, the path is renamed! Try not renaming your directories when you bind in your singularity command.

19

Thanks, @Steven ! So, do you recommend something like the below?

singularity run --cleanenv \
-B ${in_dir}:${bids_dir} \
-B ${mriqc_in}:/output_dir \
-B ${work_dir}/work_dir/mriqc_v23_1_0_grp:/wd \
${sif_img} \
/home/mdemiden/bids_dir /output_dir group \
-vv \
-w /wd
20

Remove the colon statements in your -B arguments, and replace /output_dir and /wd with their paths

21 
singularity run --cleanenv \
-B ${real_bids_dir},${fake_bids_dir},${mriqc_dir},${work_dir} \
${sif_img} \
${fake_bids_dir} ${mriqc_dir} group \
-vv \
-w ${work_dir}
1 Like