Nibabies: Local variable 'raw sources' error

Summary of what happened:

Hello,

I am running the nibabies pipeline using singularity. My data passes the BIDS check, but I am getting an error: Local Variable 'raw sources' referenced before assignment. I am running nibabies after running BIBSnet to get the derivatives path.

Also, I don’t know what the None refers to in the participant list that is output after I submit the command.

Command used (and if a helper script was used, a link to the helper script or the command generated):

singularity run --cleanenv -B $path/BIBSnet_inputs:/data:ro -B $path/BIBSnet_outputs_nibabies:/out -B /usr/local/freesurfer/6.0.0/license.txt:/opt/freesurfer/license.txt:ro $path/scripts/Kelly/infant/nibabies/nibabies-23.1.0.sif /data /out participant --derivatives $path/BIBSnet_outputs/bibsnet/ --anat-only

Version:

23.1.0

Environment (Docker, Singularity / Apptainer, custom installation):

Singularity

Data formatted according to a validatable standard? Please provide the output of the validator:

bids-validator@1.11.0
	1: [WARN] You should define 'SliceTiming' for this file. If you don't provide this information slice time correction will not be possible. 'Slice Timing' is the time at which each slice was acquired within each volume (frame) of the acquisition. Slice timing is not slice order -- rather, it is a list of times containing the time (in seconds) of each slice acquisition in relation to the beginning of volume acquisition. (code: 13 - SLICE_TIMING_NOT_DEFINED)
		./sub-007/ses-02/func/sub-007_ses-02_task-langloc_bold.nii.gz

	Please visit https://neurostars.org/search?q=SLICE_TIMING_NOT_DEFINED for existing conversations about this issue.

	2: [WARN] The recommended file /README is missing. See Section 03 (Modality agnostic files) of the BIDS specification. (code: 101 - README_FILE_MISSING)

	Please visit https://neurostars.org/search?q=README_FILE_MISSING for existing conversations about this issue.

	3: [WARN] The Authors field of dataset_description.json should contain an array of fields - with one author per field. This was triggered because there are no authors, which will make DOI registration from dataset metadata impossible. (code: 113 - NO_AUTHORS)

	Please visit https://neurostars.org/search?q=NO_AUTHORS for existing conversations about this issue.

        Summary:                  Available Tasks:        Available Modalities: 
        41 Files, 383.27MB                                MRI                   
        11 - Subjects                                                           
        2 - Sessions                                                            


	If you have any questions, please post on https://neurostars.org/tags/bids.
(node:1295) Warning: Closing directory handle on garbage collection
(Use `node --trace-warnings ...` to show where the warning was created)

Relevant log outputs (up to 20 lines):

INFO:    underlay of /etc/localtime required more than 50 (109) bind mounts

240122-16:07:10,345 nipype.workflow IMPORTANT:
	 
    Running nibabies version 23.1.0:
      * BIDS dataset path: /data.
      * Participant list: **[['007', None]**, ['008', None], ['009', None], ['010', None], ['012', None], ['015', None], ['016', None], ['017', None], ['018', None], ['019', None], ['053', None]].
      * Run identifier: 20240122-160654_5ef980ae-9020-4cff-9c17-f299d895183f.
      * Output spaces: MNIInfant.
      * Pre-run FreeSurfer's SUBJECTS_DIR: /out/sourcedata/freesurfer.
Traceback (most recent call last):
  File "/opt/conda/envs/nibabies/bin/nibabies", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/envs/nibabies/lib/python3.10/site-packages/nibabies/cli/run.py", line 63, in main
    retval = build_workflow(config_file)
  File "/opt/conda/envs/nibabies/lib/python3.10/site-packages/nibabies/cli/workflow.py", line 80, in build_workflow
    retval["workflow"] = init_nibabies_wf(config.execution.unique_labels)
  File "/opt/conda/envs/nibabies/lib/python3.10/site-packages/nibabies/workflows/base.py", line 130, in init_nibabies_wf
    single_subject_wf = init_single_subject_wf(
  File "/opt/conda/envs/nibabies/lib/python3.10/site-packages/nibabies/workflows/base.py", line 371, in init_single_subject_wf
    else init_infant_single_anat_wf(**wf_args)
  File "/opt/conda/envs/nibabies/lib/python3.10/site-packages/nibabies/workflows/anatomical/base.py", line 656, in init_infant_single_anat_wf
    anat_derivatives_wf = init_anat_derivatives_wf(
  File "/opt/conda/envs/nibabies/lib/python3.10/site-packages/nibabies/workflows/anatomical/outputs.py", line 510, in init_anat_derivatives_wf
    (inputnode, raw_sources, [(source_files, 'in_files')]),
**UnboundLocalError: local variable 'raw_sources' referenced before assignment**

Screenshots / relevant information:

Hi @kellyhi08,

For future posts like this, please use the Software Support category which will provide a template to help organize your post. You can see I have relabeled your post and reorganized it to the template.

Pinging @mgxd who might be best suited to help.

Best,
Steven

Appreciate the help @Steven
@mgxd - any chance you have an idea of how to fix this?

I have been playing around with the command, and I think it has something to do with my file structure, even though I am not getting any errors related to the BIDS validator. My structure is as follows:

- data 
    - participants.tsv --> this contains the participant_id and age column
     - sub-01
          - sub-01_sessions.tsv --> this has session_id and age
          - sess-01
                - sub-01_ses-01_T1w.nii.gz
                - sub-01_ses-01_T2w.nii.gz
          - sess-01
                  - sub-01_ses-02_T1w.nii.gz
                  - sub-01_ses-02_T2w.nii.gz

The reason I think it is the file structure is because if I reorganize all the files to not have session folders, but only all unique subject IDS, then it works:

- data 
    - participants.tsv --> this contains the participant_id and age column
     - sub-0101
                - sub-0101_T1w.nii.gz
                - sub-0101_T2w.nii.gz
     - sub-0102
                - sub-0102_T1w.nii.gz
               - sub-0102_T2w.nii.gz

Is there any way to solve this error without reorganizing all my files?

Should be ses, not sess

Sorry, that is a typo in my post above, not my file structure. I am using ses for all

1 Like

any other idea? Should the output list the subject id’s and then ‘None’:

Running nibabies version 23.1.0: * BIDS dataset path: /data. * Participant list: [['007', None], ['008', None], ['009', None], ['010', None], ['012', None], ['015', None], ['016', None], ['017', None], ['018', None], ['019', None], ['053', None]].

Hi @kellyhi08 - The list you are seeing is composed of <subject, session> pairs. If you do not have sessions in your dataset, then the Nones should be present. However, it looks like you do, so seems like something may be falling through the cracks. Can you try running tree on your dataset and share the output?

Yes, here is the output:

tree BIBSnet_inputs/
BIBSnet_inputs/
\u251c\u2500\u2500 dataset_description.json
\u251c\u2500\u2500 participants.tsv
\u251c\u2500\u2500 sub-007
\u2502   \u251c\u2500\u2500 ses-01
\u2502   \u2502   \u2514\u2500\u2500 anat
\u2502   \u2502       \u251c\u2500\u2500 sub-007_ses-01_T1w.nii.gz
\u2502   \u2502       \u2514\u2500\u2500 sub-007_ses-01_T2w.nii.gz
\u2502   \u251c\u2500\u2500 ses-02
\u2502   \u2502   \u2514\u2500\u2500 anat
\u2502   \u2502       \u251c\u2500\u2500 sub-007_ses-02_T1w.nii.gz
\u2502   \u2502       \u2514\u2500\u2500 sub-007_ses-02_T2w.nii.gz
\u2502   \u2514\u2500\u2500 sub-007_sessions.tsv
\u251c\u2500\u2500 sub-008
\u2502   \u251c\u2500\u2500 ses-01
\u2502   \u2502   \u2514\u2500\u2500 anat
\u2502   \u2502       \u251c\u2500\u2500 sub-008_ses-01_T1w.nii.gz
\u2502   \u2502       \u2514\u2500\u2500 sub-008_ses-01_T2w.nii.gz
\u2502   \u2514\u2500\u2500 sub-008_sessions.tsv
\u251c\u2500\u2500 sub-009
\u2502   \u251c\u2500\u2500 ses-01
\u2502   \u2502   \u2514\u2500\u2500 anat
\u2502   \u2502       \u251c\u2500\u2500 sub-009_ses-01_T1w.nii.gz
\u2502   \u2502       \u2514\u2500\u2500 sub-009_ses-01_T2w.nii.gz
\u2502   \u251c\u2500\u2500 ses-02
\u2502   \u2502   \u2514\u2500\u2500 anat
\u2502   \u2502       \u251c\u2500\u2500 sub-009_ses-02_T1w.nii.gz
\u2502   \u2502       \u2514\u2500\u2500 sub-009_ses-02_T2w.nii.gz
\u2502   \u2514\u2500\u2500 sub-009_sessions.tsv
\u251c\u2500\u2500 sub-010
\u2502   \u251c\u2500\u2500 ses-01
\u2502   \u2502   \u2514\u2500\u2500 anat
\u2502   \u2502       \u251c\u2500\u2500 sub-010_ses-01_T1w.nii.gz
\u2502   \u2502       \u2514\u2500\u2500 sub-010_ses-01_T2w.nii.gz
\u2502   \u2514\u2500\u2500 sub-010_sessions.tsv
\u251c\u2500\u2500 sub-012
\u2502   \u251c\u2500\u2500 ses-01
\u2502   \u2502   \u2514\u2500\u2500 anat
\u2502   \u2502       \u251c\u2500\u2500 sub-012_ses-01_T1w.nii.gz
\u2502   \u2502       \u2514\u2500\u2500 sub-012_ses-01_T2w.nii.gz
\u2502   \u2514\u2500\u2500 sub-012_sessions.tsv
\u251c\u2500\u2500 sub-015
\u2502   \u251c\u2500\u2500 ses-01
\u2502   \u2502   \u2514\u2500\u2500 anat
\u2502   \u2502       \u251c\u2500\u2500 sub-015_ses-01_T1w.nii.gz
\u2502   \u2502       \u2514\u2500\u2500 sub-015_ses-01_T2w.nii.gz
\u2502   \u2514\u2500\u2500 sub-015_sessions.tsv
\u251c\u2500\u2500 sub-016
\u2502   \u251c\u2500\u2500 ses-01
\u2502   \u2502   \u2514\u2500\u2500 anat
\u2502   \u2502       \u251c\u2500\u2500 sub-016_ses-01_T1w.nii.gz
\u2502   \u2502       \u2514\u2500\u2500 sub-016_ses-01_T2w.nii.gz
\u2502   \u2514\u2500\u2500 sub-016_sessions.tsv
\u251c\u2500\u2500 sub-017
\u2502   \u251c\u2500\u2500 ses-01
\u2502   \u2502   \u2514\u2500\u2500 anat
\u2502   \u2502       \u251c\u2500\u2500 sub-017_ses-01_T1w.nii.gz
\u2502   \u2502       \u2514\u2500\u2500 sub-017_ses-01_T2w.nii.gz
\u2502   \u2514\u2500\u2500 sub-017_sessions.tsv
\u251c\u2500\u2500 sub-018
\u2502   \u251c\u2500\u2500 ses-01
\u2502   \u2502   \u2514\u2500\u2500 anat
\u2502   \u2502       \u251c\u2500\u2500 sub-018_ses-01_T1w.nii.gz
\u2502   \u2502       \u2514\u2500\u2500 sub-018_ses-01_T2w.nii.gz
\u2502   \u2514\u2500\u2500 sub-018_sessions.tsv
\u251c\u2500\u2500 sub-019
\u2502   \u251c\u2500\u2500 ses-01
\u2502   \u2502   \u2514\u2500\u2500 anat
\u2502   \u2502       \u251c\u2500\u2500 sub-019_ses-01_T1w.nii.gz
\u2502   \u2502       \u2514\u2500\u2500 sub-019_ses-01_T2w.nii.gz
\u2502   \u2514\u2500\u2500 sub-019_sessions.tsv
\u2514\u2500\u2500 sub-053
    \u251c\u2500\u2500 ses-01
    \u2502   \u2514\u2500\u2500 anat
    \u2502       \u251c\u2500\u2500 sub-053_ses-01_T1w.nii.gz
    \u2502       \u2514\u2500\u2500 sub-053_ses-01_T2w.nii.gz
    \u2514\u2500\u2500 sub-053_sessions.tsv

37 directories, 39 files

Hmm, nothing jumps out. Here are a few extra steps to try:

  • Add -vvv to the nibabies command to enable debug logging - this may be able to track down the problem.
  • Specify a single participant ID (ie --participant-label 007), optionally also specifying a single session (ie --session-id 01), and see if the problem persists. This is the recommended way of running multiple subjects / sessions as it will process each combination as a separate process, allowing parallelization and finer-grain control of resources

I have tried that. What I ran:

singularity run --cleanenv -B $path/BIBSnet_inputs:/data:ro -B $path/BIBSnet_outputs/nibabies:/out -B /usr/local/freesurfer/7.2.0/license.txt:/opt/freesurfer/license.txt:ro $path/nibabies/nibabies-23.1.0.sif /data /out participant --derivatives $path/BIBSnet_outputs/bibsnet/ --anat-only -vvv --participant-label 007  --session-id 01

This is the new error. Also, note that sub-007 is a folder in this directory: $path/BIBSnet_inputs

singularity run --cleanenv -B /fs/ess/PAS1305/BIBSnet_inputs:/data:ro -B /fs/ess/PAS1305/BIBSnet_outputs/nibabies:/out -B /usr/local/freesurfer/7.2.0/license.txt:/opt/freesurfer/license.txt:ro /fs/ess/PAS1305/scripts/Kelly/infant/nibabies/nibabies-23.1.0.sif /data /out participant --derivatives /fs/ess/PAS1305/BIBSnet_outputs/bibsnet/ --anat-only -vvv --participant-label 007  --session-id 01
INFO:    underlay of /etc/localtime required more than 50 (109) bind mounts
240126-14:20:17,319 cli INFO:
	 Making sure the input data is BIDS compliant (warnings can be ignored in most cases).
bids-validator@1.11.0
	1: [WARN] The recommended file /README is missing. See Section 03 (Modality agnostic files) of the BIDS specification. (code: 101 - README_FILE_MISSING)

	Please visit https://neurostars.org/search?q=README_FILE_MISSING for existing conversations about this issue.

	2: [WARN] The Authors field of dataset_description.json should contain an array of fields - with one author per field. This was triggered because there are no authors, which will make DOI registration from dataset metadata impossible. (code: 113 - NO_AUTHORS)

	Please visit https://neurostars.org/search?q=NO_AUTHORS for existing conversations about this issue.

        Summary:                  Available Tasks:        Available Modalities: 
        39 Files, 134.41MB                                MRI                   
        11 - Subjects                                                           
        2 - Sessions                                                            


	If you have any questions, please post on https://neurostars.org/tags/bids.
(node:27630) Warning: Closing directory handle on garbage collection
(Use `node --trace-warnings ...` to show where the warning was created)
usage: nibabies [-h] [--version] [--skip_bids_validation] [--participant-label PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]] [-s SESSION_ID [SESSION_ID ...]]
                [-t TASK_ID] [--echo-idx ECHO_IDX] [--bids-filter-file FILE] [--anat-derivatives PATH] [--bids-database-dir PATH] [--nprocs NPROCS]
                [--omp-nthreads OMP_NTHREADS] [--mem MEMORY_GB] [--low-mem] [--use-plugin FILE] [--anat-only] [--boilerplate_only] [--md-only-boilerplate]
                [--error-on-aroma-warnings] [-v] [--ignore {fieldmaps,slicetiming,sbref,t2w,flair} [{fieldmaps,slicetiming,sbref,t2w,flair} ...]]
                [--longitudinal] [--output-spaces [OUTPUT_SPACES ...]] [--me-output-echos] [--bold2t1w-init {register,header}] [--bold2t1w-dof {6,9,12}]
                [--force-bbr] [--force-no-bbr] [--medial-surface-nan] [--project-goodvoxels] [--slice-time-ref SLICE_TIME_REF] [--dummy-scans DUMMY_SCANS]
                [--random-seed _RANDOM_SEED] [--use-aroma] [--aroma-melodic-dimensionality AROMA_MELODIC_DIM] [--return-all-components]
                [--fd-spike-threshold REGRESSORS_FD_TH] [--dvars-spike-threshold REGRESSORS_DVARS_TH] [--skull-strip-template SKULL_STRIP_TEMPLATE]
                [--skull-strip-fixed-seed] [--skull-strip-t1w {auto,skip,force}] [--fmap-bspline] [--fmap-no-demean] [--use-syn-sdc] [--force-syn]
                [--fs-license-file FILE] [--fs-subjects-dir PATH] [--no-submm-recon] [--cifti-output [{91k,170k}] | --fs-no-reconall]
                [--output-layout {bids,legacy}] [-w WORK_DIR] [--clean-workdir] [--resource-monitor] [--reports-only] [--config-file FILE] [--write-graph]
                [--stop-on-first-crash] [--notrack] [--debug {compcor,registration,fieldmaps,all} [{compcor,registration,fieldmaps,all} ...]] [--sloppy]
                [--age-months AGE_MONTHS] [--segmentation-atlases-dir SEGMENTATION_ATLASES_DIR] [--fd-radius FD_RADIUS] [-d DERIVATIVES [DERIVATIVES ...]]
                [--deriv-filter-file FILE] [--force-reconall] [--surface-recon-method {infantfs,freesurfer,mcribs}]
                bids_dir output_dir {participant}
nibabies: error: One or more participant labels were not found in the BIDS directory: 007.

Okay, we narrowed down the problem to the pybids layout. Some things to fix / check:

  1. In your command, it shows --derivatives /fs/ess/PAS1305/BIBSnet_outputs/bibsnet/, which is not a valid path within the container. You will need to add a new binding (-B /fs/ess/PAS1305/BIBSnet_outputs/bibsnet:/derivatives:ro) and update the derivatives argument (--derivatives /derivatives)
  2. Ensure the DatasetType field of your top-level dataset_description.json is either omitted or set to raw.

This has fixed the problem!! Thank you very much. I really appreciate your help.

1 Like