Summary of what happened:
Hi! In the past, I have used QSIPrep 0.19.0 to preprocess our DWI data (BIDS compliant, includes dataset_description.json and participants.tsv), but when I have updated it to QSIPrep 1.1.0, I now encounter a memory issue where there is no space left on the device to complete the job. As an attempt to rectify, I have preprocessed the DWI data using the older version (0.19.0) and then run a few reconstruction pipelines with QSIRecon (using the --recon-spec flags mrtrix_multishell_msmt_pyafq_tractometry and dsi_studio_autotrack separately) I get an error saying the data are not accessible, and “DatasetType key not in ~/bids/v1/dataset_description.json. Assuming ‘derivative’.”
Command used (and if a helper script was used, a link to the helper script or the command generated):
Of note, I have manually changed the file paths for confidentiality (using ~) but they are leading to the right places. I have changed the derivatives directory between having it end at derivatives and qsiprep (the latter shown below) with the same output errors. So it must not be a problem with the file paths from my understanding. Below, I show (I) the file paths I am pointing things to; (II) the original QSIPrep 0.19.0 script I was using in the past; (III) the new QSIPrep 1.0.0 code I am using that is yielding errors; and (IV) the new QSIRecon 1.1.0 code I am using that is yielding errors. I run this command entirely (QSIPrep and QSIRecon separately) by submitting it to a BATCH system with 1 core processor, 22 nodes (a few more than I ask in each script to allow for memory fluctuation), 128GB (since I ask for 96GB in QSIPrep and 120GB in QSIRecon–I have messed with these values by decreasing them all, to no avail). I have also messed with the --output-resolution and --low-mem flags, to no avail.
(I)
sub=$1 #subject ID, uses Step1_QSIPrep.sh to loop through subject IDs from a specified BIDS directory.
expdir="~AFQ_Demo/qsiprep" #Path to where QSIPrep output should go to
fsdir="${expdir}/freesurfer" #FreeSurfer directory, copy freesurfer output over.
bids="${expdir}/bids/v1" #where your BIDS directory of raw neuroimaging data are stored. Check that all the files are there and all that.
derivatives="${expdir}/derivatives/qsiprep" #where output from QSIPrep is to be stored.
work="${expdir}/work" #${expdir}/work" #change back
fslicense="~default/apps/fs_license.txt" #FreeSurfer license location
(II)
cmd="singularity run --containall --writable-tmpfs \
-B ${bids},${derivatives},${work},${fsdir},${fslicense}:/opt/freesurfer/license.txt \
/storage/group/edc5208/default/apps/qsiprep-0.19.0.sif \
${bids} ${derivatives} participant \
--mem_mb 96000 --nthreads 11 \
--fs-license-file /opt/freesurfer/license.txt --participant_label $sub \
--output-resolution 2 -w ${work}"
echo $cmd
eval $cmd
(III)
cmd="apptainer run --containall --writable-tmpfs \
-B ${bids},${derivatives},${work},${fsdir},${fslicense}:/opt/freesurfer/license.txt \
/storage/group/edc5208/default/apps/qsiprep-1.0.0.sif \
${bids} ${derivatives} participant \
--mem 120000 --low-mem --nthreads 22 \
--fs-license-file /opt/freesurfer/license.txt --participant-label $sub \
--output-resolution 2 -w ${work}"
echo $cmd
eval $cmd
(IV)
cmd="apptainer run --containall --writable-tmpfs \
-B ${bids},${derivatives},${work},${fsdir},${fslicense}:/opt/freesurfer/license.txt \
/storage/group/edc5208/default/apps/qsirecon-1.1.0.sif \
${bids} ${derivatives} participant \
--fs-license-file /opt/freesurfer/license.txt --participant-label $sub \
--mem 120000 --low-mem --nthreads 22 \
--output-resolution 2 -w ${work} \
--recon-spec mrtrix_multishell_msmt_pyafq_tractometry" #some yaml file, look for in Eric's apps.
echo $cmd
eval $cmd
Version:
QSIPrep 1.0.0, QSIRecon 1.1.0Environment (Docker, Singularity / Apptainer, custom installation):
Apptainer for QSIPrep 1.0.0/QSIRecon 1.1.0; Singularity run for QSIPrep 0.19.0.Data formatted according to a validatable standard? Please provide the output of the validator:
Yes, data are BIDS compliant and formatted appropriately for QSIPrep, including folders bids, derivatives, and work at the same level. The work folder I delete its contents after each run to ensure it is empty.**I have removed the entire file path for confidentiality reasons, but the file paths are accurate.
250620-10:02:26,756 nipype.workflow IMPORTANT:
Building QSIPrep's workflow:
* BIDS dataset path: /storage/group/edc5208/default/sunderland/AFQ_Demo/qsiprep/bids/v1.
* Participant list: ['205511'].
* Run identifier: 20250620-100207_99f5fedd-abc9-4776-8230-46b9f6fa827b.
250620-10:02:27,74 nipype.workflow IMPORTANT:
Collected data:
dwi:
- ~qsiprep/bids/v1/sub-205511/ses-v1/dwi/sub-205511_ses-v1_dir-AP_dwi.nii.gz
flair: []
fmap:
- ~qsiprep/bids/v1/sub-205511/ses-v1/fmap/sub-205511_ses-v1_dir-PA_epi.nii.gz
roi: []
sbref: []
t1w:
- ~qsiprep/bids/v1/sub-205511/ses-v1/anat/sub-205511_ses-v1_T1w.nii.gz
t2w: []
Relevant log outputs (up to 20 lines):
QSIPrep 1.0.0: OSError: [Errno 28] No space left on device.
QSIRecon 1.1.0 (run using QSIPrep 0.19.0 data):
250619-14:18:05,181 nipype.workflow INFO:
No dwi files found for 205511
250619-14:18:05,420 nipype.workflow INFO:
QSIRecon workflow graph with 0 nodes built successfully.
250619-14:18:20,544 nipype.workflow IMPORTANT:
QSIRecon started!
250619-14:18:20,640 nipype.workflow CRITICAL:
QSIRecon failed: Graph has no nodes or edges
250619-14:18:20,671 nipype.utils WARNING:
DatasetType key not in ~qsiprep/bids/v1/dataset_description.json. Assuming ‘derivative’.
250619-14:18:20,680 nipype.utils WARNING:
Previous output generated by version 1.0.1.dev0+gee9aa2e.d20250115 found.
250619-14:18:21,155 nipype.utils WARNING:
DatasetType key not in ~qsiprep/bids/v1/dataset_description.json. Assuming ‘derivative’.
Screenshots / relevant information:
I. The dataset_description.json file it is referring to within my BIDS directory has the following keys formatted appropriately as it has been for other runs: BIDSVersion, Name, and Funding. From what I have read online, other elements are not needed, but I would be happy to amend this file.
II. At the level of derivatives, there is a dataset_description.json file formatted appropriately, the subject’s .html file indicating QSIPrep 0.19.0 ran without error, and the subject folder containing anat, and then within the session/run, an anat and dwi folder. All contents therein are appropriate compared to previous runs of QSIPrep with other data, and the data do not appear warped/corrupted in any way.
III. I had read that the flag “qsiprep output” in the dataset_description.json file may be problematic, and I changed that to just :qsiprep" but that did not work either.
IV. I am unsure what the flag DatasetType is referring to. I have not used such key in past iterations of the dataset_description.json using this dataset and other datasets with QSIPrep (at least vers. 0.19.0) and QSIRecon.
Thank you for your help!