Hello,
I’m working on a longitudinal study and our dataset is growing quickly (hooray!) However, I’ve noticed that as we acquire more data, it is taking longer and longer for BIDS Apps like fmriprep to get going. My guess is that fmriprep and many other programs start by creating a layout of the BIDS directory (using PyBIDS or a similar tool).
For fmriprep, we are using version 22.0.2 in a singularity image (we strongly prefer to keep this version because of the longitudinal nature of our data). I’m using the flags --skip-bids-validation and --bids-filter-file to try and speed up the BIDS directoy layout creation, but there doesn’t seem to be any speed up.
My next idea was to try generate the default fmriprep layout as a file using the PyBIDS command: pybids layout /PATH/TO/BIDS/DIRECTORY/ /PATH/TO/DATABASE/FILE --no-validate --index-medata Unfortunately, I’m several hours in and have yet to generate a file. I was thinking I could automate a procedure to create one of these files every night as we acquire more data. But now I’m skeptical if it could even run in a few hours at night.
Our database consists of ~150 subjects who have each completed between 1-3 multimodal imaging sessions (modality folders for anat, func, fmap, pet, etc.) This is a large dataset by many standards, but I assumed BIDS apps could run on much larger datasets (e.g. HCP). I’m eager to hear from others about similar problems? Perhaps I’m missing a simple solution? Thanks!