Bids-database-dir ValueError

3

Hi Steven,

Thanks for the quick and very helpful response. This all makes sense to me. I followed your recommendation and edited the command as follows.

#User inputs:
bids_root_dir=/lbc/lbc1/PREVENT_AD/PREVENT_AD_BIDS
bids_deriv_dir=/lbc/lbc1/PREVENT_AD/derivatives/rsFC/rsFC_fmriprep/derivatives
fs_dir=/lbc/lbc1/PREVENT_AD/derivatives/freesurfer
pybids_dir=/lbc/lbc1/PREVENT_AD/derivatives/rsFC/pybids_db
subj=0196

#Begin:

export TEMPLATEFLOW_HOME=/home/bic/collhugh/.cache/templateflow
export FS_LICENSE=/lbc/lbc1/PREVENT_AD/derivatives/rsFC/rsFC_fmriprep/license.txt

#Run fmriprep
docker run -ti --rm \
-v $bids_root_dir:/data:ro \
-v $bids_deriv_dir:/out \
nipreps/fmriprep:21.0.0 \
/data /out \
participant \
--participant-label $subj \
--skip-bids-validation \
--bids-database-dir $pybids_dir \
--md-only-boilerplate \
--fs-no-reconall \
--fs-subjects-dir $fs_dir \
--output-spaces T1w \
--ignore slicetiming fieldmaps sbref t2w flair \
--dummy-scans 4 \
--skull-strip-t1w force \
--me-output-echos \
--stop-on-first-crash \
--low-mem

It’s still appearing to hang on the pybids indexing without giving errors. I think this is probably because the pybids directory does not live in /out and is therefore not seen by Docker. So, I added a -v flag for the pybids directory (like the workdir in the last example here). But, this results in the earlier “root dir does not exist” error. I moved the database to /out in case -v can’t be used for any location, but got the same error. I’ll keep troubleshooting as it’s probably my misunderstanding of how docker works, but would welcome any suggestions.

Just to note, the derivatives folder - including pre-run freesurfer segmentations - lives outside of the BIDS directory. This is on purpose.

4

By doing this

you are introducing the same error as the wrapper function did. When you mount your BIDS directory to docker, you don’t want to rename it in the docker environment (in your case), since your pybids directory has paths using the original name. Also, yes good catch, you will also want to mount the pybids directory with an additional -v command, forgot to say that earlier.

In short, use the command above, but change

to this:

-v $bids_root_dir \
-v $bids_deriv_dir \
-v $pybids_dir \
nipreps/fmriprep:21.0.0 \
$bids_root_dir $bids_deriv_dir \
5

Additional suggestion (doesn’t relate to the error):

In this case, I would delete the --fs-no-reconall argument. If the FS outputs are already available, recon-all will not be run, so you wouldn’t have to worry about it being run again and wasting time. But, for example, if you misspecified your fs-dir and kept that no reconall argument in, then fMRIPrep would not have any FS outputs to use which would likely detriment the anatomical workflows.

6

Oh! I interpreted the mismatch in the -v flag in your first response about the renaming to be a singularity/docker difference not the core issue. Sorry about that! I will follow both suggestions and update how it goes.

7

Great! Just to explain it a bit further, the -v docker argument mounts a drive to the docker environment, so you can use your own files in there.

-v $bids_root_dir is saying “I want to have $bids_root_dir in my docker environment”.
-v $bids_root_dir:/data is saying "I want to have $bids_root_dir in my docker environment renamed to /data"
-v $bids_root_dir:/data:ro is saying “I want tot have $bids_root_dir in my docker environment renamed to /data and with read-only permissions”

1 Like
8

Thanks for the further explanation! Well, I got past the old error to get a new one - command and error pasted below. Looks like it might be similar to some recent posts (here and here and here).

I will double check the BIDS validation done by my colleague, but I think it is likely that there is still some reason why Docker cannot see my BIDS directory. I haven’t seen many cases where the bids-database-dir argument and docker were used together. Perhaps relevant that I am not the owner/creator of the BIDS directory but I do have permissions through the group. And I do not have root access to Docker, which is why I included the -u $(id -u) argument earlier. However, that argument is only accepted by the wrapper. I am also setting all this up through ssh.

Command

#!/bin/bash

#User inputs:
bids_root_dir=/lbc/lbc1/PREVENT_AD/PREVENT_AD_BIDS
bids_deriv_dir=/lbc/lbc1/PREVENT_AD/derivatives/rsFC/rsFC_fmriprep/derivatives
fs_dir=/lbc/lbc1/PREVENT_AD/derivatives/freesurfer
pybids_dir=/lbc/lbc1/PREVENT_AD/derivatives/rsFC/pybids_db
subj=0196

#Begin:

export TEMPLATEFLOW_HOME=/home/bic/collhugh/.cache/templateflow
export FS_LICENSE=/lbc/lbc1/PREVENT_AD/derivatives/rsFC/rsFC_fmriprep/license.txt

#Run fmriprep
docker run -ti --rm -v $bids_root_dir \
-v $bids_deriv_dir \
-v $pybids_dir \
-v $fs_dir \
nipreps/fmriprep:21.0.0 \
$bids_root_dir $bids_deriv_dir \
participant \
--participant-label $subj \
--bids-database-dir $pybids_dir \
--skip-bids-validation \
--md-only-boilerplate \
--fs-subjects-dir $fs_dir/sub-${subj} \
--output-spaces T1w \
--ignore slicetiming fieldmaps sbref t2w flair \
--dummy-scans 4 \
--skull-strip-t1w force \
--me-output-echos \
--stop-on-first-crash \
--low-mem

Output
collhugh@albany:/lbc/lbc1/PREVENT_AD/derivatives/rsFC/rsFC_fmriprep/code$ ./docker_PAD.sh
/opt/conda/lib/python3.8/site-packages/bids/config.py:39: FutureWarning: Setting ‘extension_initial_dot’ will be removed in pybids 0.16.
warnings.warn(“Setting ‘extension_initial_dot’ will be removed in pybids 0.16.”,
Traceback (most recent call last):
File “/opt/conda/bin/fmriprep”, line 8, in
sys.exit(main())
File “/opt/conda/lib/python3.8/site-packages/fmriprep/cli/run.py”, line 38, in main
parse_args()
File “/opt/conda/lib/python3.8/site-packages/fmriprep/cli/parser.py”, line 668, in parse_args
config.from_dict(vars(opts))
File “/opt/conda/lib/python3.8/site-packages/fmriprep/config.py”, line 664, in from_dict
execution.load(settings)
File “/opt/conda/lib/python3.8/site-packages/fmriprep/config.py”, line 238, in load
cls.init()
File “/opt/conda/lib/python3.8/site-packages/fmriprep/config.py”, line 481, in init
cls._layout = BIDSLayout(
File “/opt/conda/lib/python3.8/site-packages/bids/layout/layout.py”, line 126, in init
root, description = validate_root(root, validate)
File “/opt/conda/lib/python3.8/site-packages/bids/layout/validation.py”, line 73, in validate_root
raise BIDSValidationError(
bids.exceptions.BIDSValidationError: ‘dataset_description.json’ is missing from project root. Every valid BIDS dataset must have this file.
Example contents of ‘dataset_description.json’:
{“Name”: “Example dataset”, “BIDSVersion”: “1.0.2”}

dataset_description.json
{
“Acknowledgements”: “Data from the PREVENT-AD”,
“Authors”: [
“PREVENT-AD group”,
“formatted by Giulia Baracchini”
],
“BIDSVersion”: “1.6.0”,
“DatasetDOI”: “”,
“Funding”: [
“”
],
“HowToAcknowledge”: “”,
“License”: “”,
“Name”: “”,
“ReferencesAndLinks”: [
“”
],
“TaskName”: “description”
}

9

A few things:

  1. Sometimes environment variable names do not get passed into the docker/singularity environment properly. So, what I would do is mount the license with -v $FS_LICENSE:/license.txt in the docker syntax, and then include --fs-license-file /license.txt in the fmriprep command, just to remove the ambiguity.
  2. Which version of pybids did you use to create the database dir? You should make sure it is the same version as the one used in the fmriprep container (for fmriprep 21.0.0, pybids is version 0.14.0).
  3. Yes, you should make sure the dataset is BIDS valid.
  4. If the -u argument is wrapper specific, then you should remove it from your command here.
  5. You can always try running a single subject without the database dir and see if it gets past that stage, just as a test. You don’t need a pybids database premade, it just makes startup quicker and saves work directory space, especially for larger datasets.
10
  1. Done (no change to error)
  2. Just installed 0.14.0 from 0.13.2 and re-created database in my test BIDS directory (one subject), but the ‘missing dataset_description.json’ error persists. Will run updated pybids on the full BIDS directory (~200 subjects overnight).
  3. Will check.
  4. Already done (no change to error).
  5. Just to clarify, I have the full BIDS directory (~200 subjects) and a test BIDS directory that is valid but only contains one subject for testing purposes. I used the test directory originally to verify my script worked and then set it to run on one subject from the full directory. Just the change in root directory caused the appearance of hanging. I assumed pybids was trying to index all 200 people and so to avoid that each time, I should pre-index. Oddly, if I use the current code minus pybids on my one person test directory, I still get the ’ missing dataset_description.json’. So now it definitely seems that the change from the wrapper to docker itself is the source of that error.

Thanks for your advice, will post if/when I have more information.

11

Quick update, if I drop pybids for now and follow the renaming format for docker, fmriprep starts successfully on the one person test BIDS directory. So there’s something that isn’t working to have docker look directly for the directory paths with/without pybids.

-v $bids_root_dir:/data is saying "I want to have $bids_root_dir in my docker environment renamed to /data

Here is the command: 
#Run fmriprep
docker run -ti --rm --user $(id -u) \
-v $bids_root_dir:/data \
-v $bids_deriv_dir:/out \
-v $fs_dir:/fssubjects \
-v $FS_LICENSE:/license.txt \
nipreps/fmriprep:21.0.0 \
/data /out \
participant \
--participant-label $subj \
--skip-bids-validation \
--fs-license-file /license.txt \
--md-only-boilerplate \
--fs-subjects-dir /fssubjects/sub-${subj} \
--output-spaces T1w \
--ignore slicetiming \
--dummy-scans 4 \
--skull-strip-t1w force \
--me-output-echos \
--low-mem
12

This is why it is not working. The pybids database you pass into fmriprep should correspond to the same paths used for the BIDS dir input into fmriprep. Pass the full BIDS datadir root into fmriprep, using the pybids database made from that full BIDS dataset.

Alternatively, you can also make a temporary single-subject database (I do this very often), but in that case, do not pass in a pybids database. A pybids database will be created automatically by fmriprep for that single-subject dataset when fmriprep begins.

13

Actually…I just added pybids in and it threw the ‘missing dataset_description.json’ error. Going to come back to this tomorrow.

docker run -ti --rm --user $(id -u) \
-v $bids_root_dir:/data \
-v $bids_deriv_dir:/out \
-v $fs_dir:/fssubjects \
-v $pybids_dir:/pybids_db \
-v $FS_LICENSE:/license.txt \
nipreps/fmriprep:21.0.0 \
/data /out \
participant \
--participant-label $subj \
--skip-bids-validation \
--bids-database-dir /pybids_db \
--fs-license-file /license.txt \
--md-only-boilerplate \
--fs-subjects-dir /fssubjects/sub-${subj} \
--output-spaces T1w \
--ignore slicetiming \
--dummy-scans 4 \
--skull-strip-t1w force \
--me-output-echos \
--low-mem
14

More things:

  1. fs-subjects-dir should point to the folder where all Freesurfer folders live. That is, it should contain a folder called $SUBJECTNAME that contains that subject’s freesurfer outputs. So in your case, --fs-subjects-dir /fssubjects should be appropriate, leaving out the subject specific part.
  2. Again, if you are using a pybids database, do not rename the mounts in the -v commands.
15

Make sure $bids_root_dir points to the full BIDS root directory, and the pybids_dir is made using the same pybids version that fMRIPrep uses, and is made on the same full BIDS root directory, then try the following:

docker run -ti --rm --user $(id -u) \
-v $bids_root_dir \
-v $bids_deriv_dir \
-v $fs_dir \
-v $pybids_dir \
-v $FS_LICENSE:/license.txt \
nipreps/fmriprep:21.0.0 \
$bids_root_dir $bids_deriv_dir \
participant \
--participant-label $subj \
--skip-bids-validation \
--bids-database-dir $pybids_dir \
--fs-license-file /license.txt \
--md-only-boilerplate \
--fs-subjects-dir $fs_dir \
--output-spaces T1w \
--ignore slicetiming \
--dummy-scans 4 \
--skull-strip-t1w force \
--me-output-echos \
--low-mem
16

Okay, I will re-run the full BIDS directory with pybids 0.14.0 (will take a couple hours) then try this solution ensuring that $bids_root_dir and pybids_dir are based on the same full directory. Thank you very much for the quick and helpful replies!

1 Like
17

Same ‘missing dataset_description.json’ error if I used the 0.14.0 pybids full database and the direct calls to my paths. The dataset is bids valid, but contains many different types of scans and some that are not yet included in bids (put in bids ignore file). So, if I don’t pre-index it hangs for a very very long time on that stage - I’ve actually not had it start fmriprep when I run just sub-0196 but using the full directory path. I’ll let the not pre-indexed code run for a full 24 hours just to see if it ever gets going. But if that happens each time, it’s not practical. I suppose I can do the temporary single subject dirs as a workaround, if needed.

18

Can you confirm the dataset description is not in the bidsignore file? Can you also print the outputs of the tree Command run on the bids data directory roof, and then on a single subject?

19

Yes, the dataset description file is not in bidsignore.

Here is the file tree for sub-0196 (using the -I flag to ignore a directory containing non-BIDS formatted files e.g., QSM dicoms that are in bidsignore).

collhugh@albany:/lbc/lbc1/PREVENT_AD$ tree PREVENT_AD_BIDS/sub-0196 -I 'nonBIDSified'
PREVENT_AD_BIDS/sub-0196
└── ses-01
    ├── anat
    │   ├── sub-0196_ses-01_acq-MPRAGE_T1w.json
    │   ├── sub-0196_ses-01_acq-MPRAGE_T1w.nii
    │   ├── sub-0196_ses-01_acq-space_T2w.json
    │   ├── sub-0196_ses-01_acq-space_T2w.nii
    │   ├── sub-0196_ses-01_FLAIR.json
    │   ├── sub-0196_ses-01_FLAIR.nii
    │   ├── sub-0196_ses-01_inv-1_MP2RAGE.json
    │   ├── sub-0196_ses-01_inv-1_MP2RAGE.nii
    │   ├── sub-0196_ses-01_inv-2_MP2RAGE.json
    │   ├── sub-0196_ses-01_inv-2_MP2RAGE.nii
    │   ├── sub-0196_ses-01_T1map.json
    │   ├── sub-0196_ses-01_T1map.nii
    │   ├── sub-0196_ses-01_UNIT1.json
    │   └── sub-0196_ses-01_UNIT1.nii
    ├── dwi
    │   ├── sub-0196_ses-01_dir-PA_dwi.bval
    │   ├── sub-0196_ses-01_dir-PA_dwi.bvec
    │   ├── sub-0196_ses-01_dir-PA_dwi.json
    │   └── sub-0196_ses-01_dir-PA_dwi.nii
    ├── fmap
    │   ├── sub-0196_ses-01_dir-AP_epi.json
    │   └── sub-0196_ses-01_dir-AP_epi.nii
    └── func
        ├── sub-0196_ses-01_task-rest_run-01_echo-01_bold.json
        ├── sub-0196_ses-01_task-rest_run-01_echo-01_bold.nii
        ├── sub-0196_ses-01_task-rest_run-01_echo-02_bold.json
        ├── sub-0196_ses-01_task-rest_run-01_echo-02_bold.nii
        ├── sub-0196_ses-01_task-rest_run-01_echo-03_bold.json
        └── sub-0196_ses-01_task-rest_run-01_echo-03_bold.nii

5 directories, 26 files

And here is the file tree one level deep for the bids data dir:

collhugh@albany:/lbc/lbc1/PREVENT_AD$ tree PREVENT_AD_BIDS -I 'nonBIDSified' -L 1
PREVENT_AD_BIDS
├── allsubs.txt
├── dataset_description.json
├── fsissues
├── ignore_derivatives
├── README.md
├── scripts
├── sub-0002
├── sub-0005
├── sub-0010
├── sub-0015
├── sub-0018
├── sub-0020
├── sub-0026
├── sub-0032
├── sub-0033
├── sub-0037
├── sub-0039
├── sub-0040
├── sub-0041
├── sub-0047
├── sub-0051
├── sub-0054
├── sub-0062
├── sub-0079
├── sub-0080
├── sub-0082
├── sub-0085
├── sub-0086
├── sub-0101
├── sub-0108
├── sub-0112
├── sub-0125
├── sub-0128
├── sub-0139
├── sub-0157
├── sub-0159
├── sub-0160
├── sub-0165
├── sub-0166
├── sub-0172
├── sub-0173
├── sub-0174
├── sub-0177
├── sub-0179
├── sub-0181
├── sub-0184
├── sub-0186
├── sub-0187
├── sub-0190
├── sub-0191
├── sub-0194
├── sub-0196
├── sub-0198
├── sub-0203
├── sub-0212
├── sub-0225
├── sub-0228
├── sub-0230
├── sub-0235
├── sub-0243
├── sub-0249
├── sub-0252
├── sub-0254
├── sub-0263
├── sub-0264
├── sub-0268
├── sub-0271
├── sub-0276
├── sub-0278
├── sub-0279
├── sub-0282
├── sub-0284
├── sub-0287
├── sub-0292
├── sub-0297
├── sub-0304
├── sub-0308
├── sub-0311
├── sub-0317
├── sub-0319
├── sub-0333
├── sub-0336
├── sub-0343
├── sub-0347
├── sub-0348
├── sub-0349
├── sub-0350
├── sub-0353
├── sub-0354
├── sub-0360
├── sub-0363
├── sub-0365
├── sub-0366
├── sub-0371
├── sub-0374
├── sub-0375
├── sub-0376
├── sub-0377
├── sub-0380
├── sub-0381
├── sub-0384
├── sub-0387
├── sub-0390
├── sub-0391
├── sub-0392
├── sub-0393
├── sub-0394
├── sub-0395
├── sub-0403
├── sub-0404
├── sub-0409
├── sub-0414
├── sub-0425
├── sub-0427
├── sub-0431
├── sub-0432
├── sub-0442
├── sub-0448
├── sub-0453
├── sub-0457
├── sub-0458
├── sub-0461
├── sub-0462
├── sub-0469
├── sub-0473
├── sub-0476
├── sub-0488
├── sub-0492
├── sub-0494
├── sub-0501
├── sub-0504
├── sub-0505
├── sub-0509
├── sub-0510
├── sub-0520
├── sub-0521
├── sub-0527
├── sub-0528
├── sub-0529
├── sub-0531
├── sub-0534
├── sub-0537
├── sub-0538
├── sub-0541
├── sub-0544
├── sub-0546
├── sub-0550
├── sub-0552
├── sub-0555
├── sub-0569
├── sub-0572
├── sub-0574
├── sub-0580
├── sub-0587
├── sub-0588
├── sub-0589
├── sub-0595
├── sub-0598
├── sub-0599
├── sub-0602
├── sub-0603
├── sub-0606
├── sub-0607
├── sub-0608
├── sub-0609
├── sub-0611
├── sub-0622
├── sub-0631
├── sub-0632
├── sub-0633
├── sub-0634
├── sub-0648
├── sub-0649
├── sub-0653
├── sub-0663
├── sub-0665
├── sub-0668
├── sub-0669
├── sub-0672
├── sub-0674
├── sub-0681
├── sub-0688
├── sub-0694
├── sub-0696
├── sub-0703
├── sub-0706
├── sub-0709
├── temp
└── weirdsubjs

186 directories, 3 files
20

And the contents of bidsignore

/sub-*/nonBIDSified/**
/scripts/**
/temp/**
/weirdsubjs/**
/ignore_derivatives/**
/fsissues/**
/allsubs.txt/
21

My labmate just brought up the good point that pybids might be indexing the nonBIDSified folders for each subject that have many QSM dicoms and nii files for scan types not in BIDS. So I wonder if there’s a way to tell pybids to ignore those files (they are already in bidsignore), which might speed up the process.

22

pybids should ignore files in bidsignore too. With all of those non bids-compliant files, I think it would be okay to try the single-subject bids directory. It would go something like this (adapt to your needs):

workdir=/path/to/scratch #Set this as your scratch directory
mkdir -p $workdir/${subj}_db/${subj}/func #Make the single-subject directory in scratch space
mkdir -p $workdir/${subj}_db/${subj}/anat
cp $bids_root_dir/dataset_description.json $workdir/${subj}_db/ #Copy dataset_description
cp $bids_root_dir/$subj/func/* $workdir/${subj}_db/${subj}/func #Copy functional files
cp $bids_root_dir/$subj/anat/* $workdir/${subj}_db/${subj}/anat #Copy anatomical files

Then, in the fMRIPrep command, simply substitute the input directory argument (previously occupied by $bids_root_dir) with the single subject temporary directory ($workdir/${subj}_db/).

Does this make sense?