Error using qsiprep: "'dataset_description.json' is missing from project root. "

Continuing the discussion from Dataset_description.json missing from project root, file exists in project root:

Summary of what happened:

I am trying to run qsiprep on a BIDS validated dataset (with a few warnings but they don’t seem relevant to the issue I am having here).

I checked and /opt/conda directory does not exist on my laptop. It may be in the container?
On my end, I use VSCode Version: 1.95.2 (Universal)
I created a /.conda virtual environment with Python 3.9.20
And my laptop is Apple M3 Pro with Mac OS Sonoma 14.7.1

Command used (and if a helper script was used, a link to the helper script or the command generated):

import os
import os.path as op
# 
# qsiprep (preprocessing)
version = "latest"
bids_dir = op.join(op.expanduser('~'), 'Documents/research/ampb_mt_tractometry_analysis/ampb')
output_dir = op.join(op.expanduser('~'), 'Documents/research/ampb_mt_tractometry_analysis/ampb/derivatives')

participants = [f for f in os.listdir(bids_dir) if f not in {".DS_Store", "dataset_description.json", "code","derivatives"}]
participants_cleaned = [f.replace("sub-", "") for f in participants]
print(participants_cleaned)

for participant in participants_cleaned:
    cmd = f"docker run --rm " \
        f"--volume {bids_dir} " \
        f"--volume {output_dir} " \
        f"--volume $FREESURFER_HOME/license.txt " \
        f"pennlinc/qsiprep:{version} " \
        f"--skip-bids-validation " \
        f"{bids_dir} {output_dir} " \
        f"participant --participant-label {participant} " \
        f"--ignore fieldmaps " \
        f"--output-resolution 1.5 " \
        f"--fs-license-file $FREESURFER_HOME/license.txt " 

    os.system(cmd)

Version:

1.0.0rc1

Environment (Docker, Singularity / Apptainer, custom installation):

I am using Docker, I installed Docker desktop from the website
Docker version 27.3.1, build ce12230
I then pulled pennlinc/qsiprep:latest from DockerHub.

Data formatted according to a validatable standard? Please provide the output of the validator:

 deno run -A jsr:@bids/validator /Users/ldaumail3/Documents/Research/AMPB_MT_tractometry_analysis/AMPB 
	[WARNING] README_FILE_MISSING The recommended file /README is missing.
See Section 03 (Modality agnostic files) of the BIDS specification.

		/dataset_description.json

	Please visit https://neurostars.org/search?q=README_FILE_MISSING for existing conversations about this issue.

	[WARNING] TOO_FEW_AUTHORS The 'Authors' field of 'dataset_description.json' should contain an array of values -
with one author per value.
This was triggered based on the presence of only one author field.
Please ignore if all contributors are already properly listed.

		/dataset_description.json

	Please visit https://neurostars.org/search?q=TOO_FEW_AUTHORS for existing conversations about this issue.

	[WARNING] JSON_KEY_RECOMMENDED A JSON file is missing a key listed as recommended.
		DatasetType
		/dataset_description.json

		GeneratedBy
		/dataset_description.json

		SourceDatasets
		/dataset_description.json

	Please visit https://neurostars.org/search?q=JSON_KEY_RECOMMENDED for existing conversations about this issue.

	[WARNING] SIDECAR_KEY_RECOMMENDED A data file's JSON sidecar is missing a key listed as recommended.
		GradientSetType
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-PA_epi.nii.gz
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-AP_epi.nii.gz

		961 more files with the same issue

		MRTransmitCoilSequence
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-PA_epi.nii.gz
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-AP_epi.nii.gz

		961 more files with the same issue

		MatrixCoilMode
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-PA_epi.nii.gz
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-AP_epi.nii.gz

		961 more files with the same issue

		CoilCombinationMethod
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-PA_epi.nii.gz
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-AP_epi.nii.gz

		961 more files with the same issue

		PulseSequenceType
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-PA_epi.nii.gz
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-AP_epi.nii.gz

		961 more files with the same issue

		MTState
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-PA_epi.nii.gz
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-AP_epi.nii.gz

		961 more files with the same issue

		SpoilingState
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-PA_epi.nii.gz
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-AP_epi.nii.gz

		961 more files with the same issue

		NumberShots
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-PA_epi.nii.gz
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-AP_epi.nii.gz

		961 more files with the same issue

		ParallelReductionFactorInPlane
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-PA_epi.nii.gz
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-AP_epi.nii.gz

		810 more files with the same issue

		ParallelReductionFactorOutOfPlane
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-PA_epi.nii.gz
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-AP_epi.nii.gz

		961 more files with the same issue

		ParallelAcquisitionTechnique
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-PA_epi.nii.gz
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-AP_epi.nii.gz

		961 more files with the same issue

		PartialFourierDirection
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-PA_epi.nii.gz
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-AP_epi.nii.gz

		961 more files with the same issue

		MixingTime
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-PA_epi.nii.gz
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-AP_epi.nii.gz

		961 more files with the same issue

		InversionTime
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-PA_epi.nii.gz
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-AP_epi.nii.gz

		826 more files with the same issue

		DwellTime
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-PA_epi.nii.gz
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-AP_epi.nii.gz

		961 more files with the same issue

		SliceEncodingDirection
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-PA_epi.nii.gz
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-AP_epi.nii.gz

		562 more files with the same issue

		B0FieldIdentifier
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-PA_epi.nii.gz
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-AP_epi.nii.gz

		606 more files with the same issue

		InstitutionalDepartmentName
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-PA_epi.nii.gz
		/sub-NSxLxIUx1994/ses-02/fmap/sub-NSxLxIUx1994_ses-02_acq-opphase_dir-AP_epi.nii.gz

		961 more files with the same issue

		StimulusPresentation
		/sub-NSxLxIUx1994/ses-02/func/sub-NSxLxIUx1994_ses-02_task-ampb_run-3_events.tsv
		/sub-NSxLxIUx1994/ses-02/func/sub-NSxLxIUx1994_ses-02_task-ampb_run-6_events.tsv

		322 more files with the same issue

		NumberOfVolumesDiscardedByScanner
		/sub-NSxLxIUx1994/ses-02/func/sub-NSxLxIUx1994_ses-02_task-ampb_run-4_bold.nii.gz
		/sub-NSxLxIUx1994/ses-02/func/sub-NSxLxIUx1994_ses-02_task-ampb_run-1_bold.nii.gz

		322 more files with the same issue

		NumberOfVolumesDiscardedByUser
		/sub-NSxLxIUx1994/ses-02/func/sub-NSxLxIUx1994_ses-02_task-ampb_run-4_bold.nii.gz
		/sub-NSxLxIUx1994/ses-02/func/sub-NSxLxIUx1994_ses-02_task-ampb_run-1_bold.nii.gz

		322 more files with the same issue

		DelayTime
		/sub-NSxLxIUx1994/ses-02/func/sub-NSxLxIUx1994_ses-02_task-ampb_run-4_bold.nii.gz
		/sub-NSxLxIUx1994/ses-02/func/sub-NSxLxIUx1994_ses-02_task-ampb_run-1_bold.nii.gz

		322 more files with the same issue

		AcquisitionDuration
		/sub-NSxLxIUx1994/ses-02/func/sub-NSxLxIUx1994_ses-02_task-ampb_run-4_bold.nii.gz
		/sub-NSxLxIUx1994/ses-02/func/sub-NSxLxIUx1994_ses-02_task-ampb_run-1_bold.nii.gz

		322 more files with the same issue

		DelayAfterTrigger
		/sub-NSxLxIUx1994/ses-02/func/sub-NSxLxIUx1994_ses-02_task-ampb_run-4_bold.nii.gz
		/sub-NSxLxIUx1994/ses-02/func/sub-NSxLxIUx1994_ses-02_task-ampb_run-1_bold.nii.gz

		322 more files with the same issue

		Instructions
		/sub-NSxLxIUx1994/ses-02/func/sub-NSxLxIUx1994_ses-02_task-ampb_run-4_bold.nii.gz
		/sub-NSxLxIUx1994/ses-02/func/sub-NSxLxIUx1994_ses-02_task-ampb_run-1_bold.nii.gz

		322 more files with the same issue

		TaskDescription
		/sub-NSxLxIUx1994/ses-02/func/sub-NSxLxIUx1994_ses-02_task-ampb_run-4_bold.nii.gz
		/sub-NSxLxIUx1994/ses-02/func/sub-NSxLxIUx1994_ses-02_task-ampb_run-1_bold.nii.gz

		322 more files with the same issue

		CogAtlasID
		/sub-NSxLxIUx1994/ses-02/func/sub-NSxLxIUx1994_ses-02_task-ampb_run-4_bold.nii.gz
		/sub-NSxLxIUx1994/ses-02/func/sub-NSxLxIUx1994_ses-02_task-ampb_run-1_bold.nii.gz

		322 more files with the same issue

		CogPOID
		/sub-NSxLxIUx1994/ses-02/func/sub-NSxLxIUx1994_ses-02_task-ampb_run-4_bold.nii.gz
		/sub-NSxLxIUx1994/ses-02/func/sub-NSxLxIUx1994_ses-02_task-ampb_run-1_bold.nii.gz

		322 more files with the same issue

		EffectiveEchoSpacing
		/sub-NSxLxIUx1994/ses-01/anat/sub-NSxLxIUx1994_ses-01_acq-MEMPRvNav_rec-RMS_T1w.nii.gz
		/sub-NSxLxIUx1994/ses-01/anat/sub-NSxLxIUx1994_ses-01_acq-MEMPRvNav_echo-2_T1w.nii.gz

		397 more files with the same issue

		PhaseEncodingDirection
		/sub-NSxLxIUx1994/ses-01/anat/sub-NSxLxIUx1994_ses-01_acq-MEMPRvNav_rec-RMS_T1w.nii.gz
		/sub-NSxLxIUx1994/ses-01/anat/sub-NSxLxIUx1994_ses-01_acq-MEMPRvNav_echo-2_T1w.nii.gz

		381 more files with the same issue

		TotalReadoutTime
		/sub-NSxLxIUx1994/ses-01/anat/sub-NSxLxIUx1994_ses-01_acq-MEMPRvNav_rec-RMS_T1w.nii.gz
		/sub-NSxLxIUx1994/ses-01/anat/sub-NSxLxIUx1994_ses-01_acq-MEMPRvNav_echo-2_T1w.nii.gz

		397 more files with the same issue

		MultibandAccelerationFactor
		/sub-NSxLxIUx1994/ses-01/anat/sub-NSxLxIUx1994_ses-01_acq-MEMPRvNav_rec-RMS_T1w.nii.gz
		/sub-NSxLxIUx1994/ses-01/anat/sub-NSxLxIUx1994_ses-01_acq-MEMPRvNav_echo-2_T1w.nii.gz

		461 more files with the same issue

		Manufacturer
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-headMTw_RB1COR.nii
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-bodyMTw_RB1COR.nii

		246 more files with the same issue

		ManufacturersModelName
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-headMTw_RB1COR.nii
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-bodyMTw_RB1COR.nii

		246 more files with the same issue

		DeviceSerialNumber
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-headMTw_RB1COR.nii
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-bodyMTw_RB1COR.nii

		246 more files with the same issue

		StationName
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-headMTw_RB1COR.nii
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-bodyMTw_RB1COR.nii

		246 more files with the same issue

		SoftwareVersions
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-headMTw_RB1COR.nii
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-bodyMTw_RB1COR.nii

		246 more files with the same issue

		MagneticFieldStrength
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-headMTw_RB1COR.nii
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-bodyMTw_RB1COR.nii

		246 more files with the same issue

		ReceiveCoilName
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-headMTw_RB1COR.nii
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-bodyMTw_RB1COR.nii

		246 more files with the same issue

		ReceiveCoilActiveElements
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-headMTw_RB1COR.nii
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-bodyMTw_RB1COR.nii

		246 more files with the same issue

		ScanningSequence
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-headMTw_RB1COR.nii
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-bodyMTw_RB1COR.nii

		246 more files with the same issue

		SequenceVariant
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-headMTw_RB1COR.nii
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-bodyMTw_RB1COR.nii

		246 more files with the same issue

		ScanOptions
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-headMTw_RB1COR.nii
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-bodyMTw_RB1COR.nii

		246 more files with the same issue

		SequenceName
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-headMTw_RB1COR.nii
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-bodyMTw_RB1COR.nii

		246 more files with the same issue

		PulseSequenceDetails
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-headMTw_RB1COR.nii
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-bodyMTw_RB1COR.nii

		246 more files with the same issue

		PartialFourier
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-headMTw_RB1COR.nii
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-bodyMTw_RB1COR.nii

		246 more files with the same issue

		EchoTime
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-headMTw_RB1COR.nii
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-bodyMTw_RB1COR.nii

		246 more files with the same issue

		FlipAngle
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-headMTw_RB1COR.nii
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-bodyMTw_RB1COR.nii

		246 more files with the same issue

		InstitutionName
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-headMTw_RB1COR.nii
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-bodyMTw_RB1COR.nii

		246 more files with the same issue

		InstitutionAddress
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-headMTw_RB1COR.nii
		/sub-NSxGxBAx1970/ses-04/fmap/sub-NSxGxBAx1970_ses-04_acq-bodyMTw_RB1COR.nii

		246 more files with the same issue

	Please visit https://neurostars.org/search?q=SIDECAR_KEY_RECOMMENDED for existing conversations about this issue.


          Summary:                          Available Tasks:        Available Modalities:
          2703 Files, 44.1 GB               ampb                    MRI                  
          27 - Subjects 6 - Sessions        ptlocal                                      
                                            mtlocal                                      

	If you have any questions, please post on https://neurostars.org/tags/bids.


Relevant log outputs (up to 20 lines):

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Traceback (most recent call last):
  File "/opt/conda/envs/qsiprep/bin/qsiprep", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/qsiprep/cli/run.py", line 42, in main
    parse_args()
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/qsiprep/cli/parser.py", line 709, in parse_args
    config.from_dict({})
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/qsiprep/config.py", line 721, in from_dict
    execution.load(settings, init=initialize("execution"), ignore=ignore)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/qsiprep/config.py", line 251, in load
    cls.init()
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/qsiprep/config.py", line 496, in init
    cls._layout = BIDSLayout(
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/bids/layout/layout.py", line 135, in __init__
    root, description = validate_root(root, validate)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/bids/layout/validation.py", line 75, in validate_root
    raise BIDSValidationError(
bids.exceptions.BIDSValidationError: 'dataset_description.json' is missing from project root. Every valid BIDS dataset must have this file.
Example contents of 'dataset_description.json': 
{"Name": "Example dataset", "BIDSVersion": "1.0.2"}

Screenshots / relevant information:


Hi @ldaumail,

What are the contents of your dataset_description.json?

Best,
Steven

Hi @Steven,

Thank you for your prompt reply!

Content is as follows:

{
    "Acknowledgements": "",
    "Authors": [],
    "BIDSVersion": "1.6.0",
    "DatasetDOI": "",
    "Funding": [],
    "HowToAcknowledge": "",
    "License": "",
    "Name": "VisDep-AMPB",
    "ReferencesAndLinks": [],
    "rule_id": "bids_project",
    "template": "project"
}

Hi @ldaumail,

Can you fix the dataset to be BIDS valid by removing the subjects folder? Also how were these data converted to BIDS? And what vendor did they come from. That is a worrying amount of warnings. Can you also try adding the -ti flag to the docker preamble? And specify a work dir (-w qsiprep flag).

Will you be eventually working working on a compute cluster? If so I recommend troubleshooting there. Some tools used in qsiprep rely on tensorflow versions that do not work on Apple silicon.

Best,
Steven

Hi @Steven,

Thanks for your suggestions.

I added -ti and a work directory. Now this is the error I am getting:
"
the input device is not a TTY
"

I will look into the BIDS data formatting, I did not convert this raw data into bids myself.

Best,
-Loïc