ValueError: 'epi_bold_run-001_fmap' is not in list / Exception raised while executing Node output_select

Summary of what happened:

Hi there,

I tried running a first round of fmriprep on my data.

Command used (and if a helper script was used, a link to the helper script or the command generated):

I ran the following command using Docker on a Mac OS catalina v12.6.2 with 12 cpus

for subject in subjects.participant_id:
    subject = subject[4:]
    if not op.isdir(f"{outdir}/sub-{subject}"):
        cmd = f"docker run --memory=\"22g\" --rm " \
              f"-v {indir}:/data:ro " \
              f"-v {outdir}:/out "\
              f"-v {workdir}:/work " \
              f"-v $FREESURFER_HOME:/fsDir " \
              f"nipreps/fmriprep:{version} " \
              f"--resource-monitor " \
              f"--n_cpus 8 " \
              f"--mem-mb 22000 " \
              f"--low-mem " \
              f"--stop-on-first-crash " \
              f"--use-aroma " \
              f"--fs-license-file /fsDir/license.txt " \
              f"--output-spaces fsaverage " \
              f"-w /work " \
              f"/data /out/ participant --participant-label {subject}"  # indir, outdir, analysis level
        os.system(cmd)

Version:

Fmriprep version is v23.0.1

Environment (Docker, Singularity, custom installation):

Docker.

Data formatted according to a validatable standard? Please provide the output of the validator:

and the BIDS validator report is:

Error 1: [Code 23] TSV_EMPTY_CELL
Click here for more information about this issue

Empty cell in TSV file detected: The proper way of labeling missing values is "n/a".

Though I silenced this error in the .bidsignore, as this error relates to empty columns in
the sub-XXXX_ses-XX_scans.tsv file, as acquisition time column and operator column is empty.

When running fmriprep, the only warning I get is:

bids-validator@1.8.0
	1: [WARN] The recommended file /README is very small. Please consider expanding it with additional information about the dataset. (code: 213 - README_FILE_SMALL)
	Please visit https://neurostars.org/search?q=README_FILE_SMALL for existing conversations about this issue.
        Summary:                Available Tasks:                        Available Modalities: 
        92 Files, 1.02GB        TODO: full task name for phantom        MRI                   
        2 - Subjects                                                                          
        1 - Session                                                                           
	If you have any questions, please post on https://neurostars.org/tags/bids.

Relevant log outputs (up to 20 lines):

I got the following error:

230421-23:22:48,786 nipype.workflow INFO:
	 [Node] Finished "fmap_rpt", elapsed time 10.881908s.
230421-23:22:49,957 nipype.workflow WARNING:
	 Storing result file without outputs
230421-23:22:49,965 nipype.workflow WARNING:
	 [Node] Error on "fmriprep_23_0_wf.single_subject_F019_wf.func_preproc_ses_01_task_phantom_dir_AP_run_001_wf.output_select" (/work/fmriprep_23_0_wf/single_subject_F019_wf/func_preproc_ses_01_task_phantom_dir_AP_run_001_wf/output_select)
230421-23:22:49,971 nipype.workflow ERROR:
	 Node output_select failed to run on host 90d0f01ea9e7.
230421-23:22:49,981 nipype.workflow ERROR:
	 Saving crash info to /out/sub-F019/log/20230421-224919_ad4cbd62-0a3a-4534-aa93-b7d04d08c09a/crash-20230421-232249-root-output_select-ab190351-aae6-48dd-8ed7-0f748b17430f.txt
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 344, in _send_procs_to_workers
    self.procs[jobid].run(updatehash=updatehash)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node output_select.
Traceback:
	Traceback (most recent call last):
	  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 401, in run
	    outputs = self.aggregate_outputs(runtime)
	  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 430, in aggregate_outputs
	    predicted_outputs = self._list_outputs()  # Predictions from _list_outputs
	  File "/opt/conda/lib/python3.9/site-packages/niworkflows/interfaces/utility.py", line 217, in _list_outputs
	    index = self.inputs.keys.index(self.inputs.key)
	ValueError: 'epi_bold_run-001_fmap' is not in list
230421-23:22:50,28 nipype.workflow CRITICAL:
	 fMRIPrep failed: Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 344, in _send_procs_to_workers
    self.procs[jobid].run(updatehash=updatehash)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node output_select.
Traceback:
	Traceback (most recent call last):
	  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 401, in run
	    outputs = self.aggregate_outputs(runtime)
	  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 430, in aggregate_outputs
	    predicted_outputs = self._list_outputs()  # Predictions from _list_outputs
	  File "/opt/conda/lib/python3.9/site-packages/niworkflows/interfaces/utility.py", line 217, in _list_outputs
	    index = self.inputs.keys.index(self.inputs.key)
	ValueError: 'epi_bold_run-001_fmap' is not in list

Screenshots / relevant information:

I think this is related to the way I dealt with B0FieldIdentifier/B0FieldSource.

In my bold .json files, I inserted the following:

        if "B0FieldIdentifier" not in boldJson:
            identifier = 'epi_bold_%s_fmap'%runNum
            boldJson["B0FieldIdentifier"] = identifier
        if "B0FieldSource" not in boldJson:
            source = 'epi_bold_%s_fmap'%runNum
            boldJson["B0FieldSource"] = source

In my corresponding topup epi .json files, I inserted this:

        if "IntendedFor" not in epiJson:
            nifti_target = f"{subject}/{jsonpath[:-5]}.nii.gz"  # link bold run nifti path to the corresponding epi json file for topup
            epiJson["IntendedFor"] = nifti_target[9:]
        if "B0FieldIdentifier" not in epiJson:
            identifier = 'epi_bold_%s_fmap' % runNum
            epiJson["B0FieldIdentifier"] = identifier

What did I do wrong?

Thanks!

Hi @ldaumail,

I have relabeled your post as Software Support and reorganized it according to the appropriate post template. In the future please submit software questions under this post category. Please provide the missing information (version and BIDS validation report) by editing your post which will help us assist you better.

Best,
Steven

Thanks for the heads up, I just edited my post

Can you print the output of tree on a single subject’s data directory?

Additionally, I do not know much about the B0 source/field json fields, but would this thread help? FmriPrep SDC Fails to Associate To Bold Image, Preprocess Working - #4 by toomanycats

/Users/tong_processor/Desktop/Loic/phantom_mri/data/derivatives/fmriprep-v23.0.1/sub-F019/figures
/Users/tong_processor/Desktop/Loic/phantom_mri/data/derivatives/fmriprep-v23.0.1/sub-F019/figures/sub-F019_ses-01_task-phantom_dir-AP_run-001_desc-validation_bold.html
/Users/tong_processor/Desktop/Loic/phantom_mri/data/derivatives/fmriprep-v23.0.1/sub-F019/figures/sub-F019_ses-01_desc-summary_T1w.html
/Users/tong_processor/Desktop/Loic/phantom_mri/data/derivatives/fmriprep-v23.0.1/sub-F019/figures/sub-F019_ses-01_task-phantom_dir-AP_run-006_desc-validation_bold.html
/Users/tong_processor/Desktop/Loic/phantom_mri/data/derivatives/fmriprep-v23.0.1/sub-F019/figures/sub-F019_ses-01_task-phantom_dir-AP_run-003_desc-validation_bold.html
/Users/tong_processor/Desktop/Loic/phantom_mri/data/derivatives/fmriprep-v23.0.1/sub-F019/figures/sub-F019_ses-01_task-phantom_dir-AP_run-004_desc-validation_bold.html
/Users/tong_processor/Desktop/Loic/phantom_mri/data/derivatives/fmriprep-v23.0.1/sub-F019/figures/sub-F019_ses-01_task-phantom_dir-AP_run-005_desc-validation_bold.html
/Users/tong_processor/Desktop/Loic/phantom_mri/data/derivatives/fmriprep-v23.0.1/sub-F019/figures/sub-F019_ses-01_desc-about_T1w.html
/Users/tong_processor/Desktop/Loic/phantom_mri/data/derivatives/fmriprep-v23.0.1/sub-F019/figures/sub-F019_ses-01_task-phantom_dir-AP_run-002_desc-validation_bold.html
/Users/tong_processor/Desktop/Loic/phantom_mri/data/derivatives/fmriprep-v23.0.1/sub-F019/figures/sub-F019_ses-01_desc-conform_T1w.html
/Users/tong_processor/Desktop/Loic/phantom_mri/data/derivatives/fmriprep-v23.0.1/sub-F019/log
/Users/tong_processor/Desktop/Loic/phantom_mri/data/derivatives/fmriprep-v23.0.1/sub-F019/log/20230422-165354_885177f1-494e-4a74-b751-e9aee98961b0
/Users/tong_processor/Desktop/Loic/phantom_mri/data/derivatives/fmriprep-v23.0.1/sub-F019/log/20230422-165354_885177f1-494e-4a74-b751-e9aee98961b0/fmriprep.toml
/Users/tong_processor/Desktop/Loic/phantom_mri/data/derivatives/fmriprep-v23.0.1/sub-F019/log/20230422-165354_885177f1-494e-4a74-b751-e9aee98961b0/crash-20230422-172403-root-output_select-fe5bace1-61f6-49c7-8160-58aa9373c087.txt
/Users/tong_processor/Desktop/Loic/phantom_mri/data/derivatives/fmriprep-v23.0.1/sub-F019/ses-01
/Users/tong_processor/Desktop/Loic/phantom_mri/data/derivatives/fmriprep-v23.0.1/sub-F019/ses-01/fmap
/Users/tong_processor/Desktop/Loic/phantom_mri/data/derivatives/fmriprep-v23.0.1/sub-F019/ses-01/fmap/sub-F019_ses-01_acq-b0shimmed_run-001_fmapid-auto00000_desc-magnitude_fieldmap.nii.gz
/Users/tong_processor/Desktop/Loic/phantom_mri/data/derivatives/fmriprep-v23.0.1/sub-F019/ses-01/anat
/Users/tong_processor/Desktop/Loic/phantom_mri/data/derivatives/fmriprep-v23.0.1/sub-F019/ses-01/anat/sub-F019_ses-01_run-001_from-orig_to-T1w_mode-image_xfm.txt
/Users/tong_processor/Desktop/Loic/phantom_mri/data/derivatives/fmriprep-v23.0.1/sub-F019/ses-01/anat/sub-F019_ses-01_acq-mprage_from-orig_to-T1w_mode-image_xfm.txt
/Users/tong_processor/Desktop/Loic/phantom_mri/data/derivatives/fmriprep-v23.0.1/sub-F019/ses-01/func
/Users/tong_processor/Desktop/Loic/phantom_mri/data/derivatives/fmriprep-v23.0.1/sub-F019/ses-01/func/sub-F019_ses-01_task-phantom_dir-AP_run-002_from-scanner_to-boldref_mode-image_xfm.txt
/Users/tong_processor/Desktop/Loic/phantom_mri/data/derivatives/fmriprep-v23.0.1/sub-F019/ses-01/func/sub-F019_ses-01_task-phantom_dir-AP_run-003_from-scanner_to-boldref_mode-image_xfm.txt
/Users/tong_processor/Desktop/Loic/phantom_mri/data/derivatives/fmriprep-v23.0.1/sub-F019/ses-01/func/sub-F019_ses-01_task-phantom_dir-AP_run-001_from-scanner_to-boldref_mode-image_xfm.txt
/Users/tong_processor/Desktop/Loic/phantom_mri/data/derivatives/fmriprep-v23.0.1/sub-F019/ses-01/func/sub-F019_ses-01_task-phantom_dir-AP_run-005_from-scanner_to-boldref_mode-image_xfm.txt
/Users/tong_processor/Desktop/Loic/phantom_mri/data/derivatives/fmriprep-v23.0.1/sub-F019/ses-01/func/sub-F019_ses-01_task-phantom_dir-AP_run-006_from-scanner_to-boldref_mode-image_xfm.txt
/Users/tong_processor/Desktop/Loic/phantom_mri/data/derivatives/fmriprep-v23.0.1/sub-F019/ses-01/func/sub-F019_ses-01_task-phantom_dir-AP_run-004_from-scanner_to-boldref_mode-image_xfm.txt

I meant the tree of the raw data. Additionally, have you tried not using the B0 fields and instead using IntendedFor?

Sorry please see below:

/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/.DS_Store
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/.DS_Store
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/sub-F019_ses-01_scans.tsv
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/fmap
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/fmap/sub-F019_ses-01_acq-b0shimmed_run-001_fieldmap.nii.gz
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/fmap/sub-F019_ses-01_acq-phtopup_dir-PA_run-004_epi.json
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/fmap/sub-F019_ses-01_acq-phtopup_dir-PA_run-005_epi.json
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/fmap/sub-F019_ses-01_acq-phtopup_dir-PA_run-003_epi.nii.gz
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/fmap/sub-F019_ses-01_acq-phtopup_dir-PA_run-006_epi.nii.gz
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/fmap/sub-F019_ses-01_acq-phtopup_dir-PA_run-001_epi.nii.gz
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/fmap/sub-F019_ses-01_acq-phtopup_dir-PA_run-004_epi.nii.gz
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/fmap/sub-F019_ses-01_acq-phtopup_dir-PA_run-003_epi.json
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/fmap/sub-F019_ses-01_acq-phtopup_dir-PA_run-002_epi.json
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/fmap/sub-F019_ses-01_acq-b0shimmed_run-001_fieldmap.json
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/fmap/sub-F019_ses-01_acq-funcNoEPI_run-001_magnitude.json
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/fmap/sub-F019_ses-01_acq-b0shimmed_run-001_magnitude.json
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/fmap/sub-F019_ses-01_acq-phtopup_dir-PA_run-002_epi.nii.gz
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/fmap/sub-F019_ses-01_acq-phtopup_dir-PA_run-006_epi.json
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/fmap/sub-F019_ses-01_acq-phtopup_dir-PA_run-005_epi.nii.gz
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/fmap/sub-F019_ses-01_acq-funcNoEPI_run-001_magnitude.nii.gz
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/fmap/sub-F019_ses-01_acq-b0shimmed_run-001_magnitude.nii.gz
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/fmap/sub-F019_ses-01_acq-phtopup_dir-PA_run-001_epi.json
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/anat
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/anat/sub-F019_ses-01_run-001_T1w.nii.gz
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/anat/sub-F019_ses-01_acq-mprage_T1w.nii.gz
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/anat/.DS_Store
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/anat/sub-F019_ses-01_acq-mprage_T1w.json
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/anat/sub-F019_ses-01_run-001_T1w.json
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/func
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/func/sub-F019_ses-01_task-phantom_dir-AP_run-006_bold.json
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/func/sub-F019_ses-01_task-phantom_dir-AP_run-006_bold.nii.gz
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/func/sub-F019_ses-01_task-phantom_dir-AP_run-002_events.tsv
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/func/.DS_Store
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/func/sub-F019_ses-01_task-phantom_dir-AP_run-003_bold.nii.gz
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/func/sub-F019_ses-01_task-phantom_dir-AP_run-005_events.tsv
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/func/sub-F019_ses-01_task-phantom_dir-AP_run-001_bold.json
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/func/sub-F019_ses-01_task-phantom_dir-AP_run-005_bold.nii.gz
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/func/sub-F019_ses-01_task-phantom_dir-AP_run-003_events.tsv
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/func/sub-F019_ses-01_task-phantom_dir-AP_run-002_bold.nii.gz
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/func/sub-F019_ses-01_task-phantom_dir-AP_run-004_bold.json
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/func/sub-F019_ses-01_task-phantom_dir-AP_run-002_bold.json
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/func/sub-F019_ses-01_task-phantom_dir-AP_run-006_events.tsv
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/func/sub-F019_ses-01_task-phantom_dir-AP_run-004_bold.nii.gz
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/func/sub-F019_ses-01_task-phantom_dir-AP_run-005_bold.json
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/func/sub-F019_ses-01_task-phantom_dir-AP_run-001_events.tsv
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/func/sub-F019_ses-01_task-phantom_dir-AP_run-003_bold.json
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/func/sub-F019_ses-01_task-phantom_dir-AP_run-001_bold.nii.gz
/Users/tong_processor/Desktop/Loic/phantom_mri/data/BIDS_conv/Nifti_topup/sub-F019/ses-01/func/sub-F019_ses-01_task-phantom_dir-AP_run-004_events.tsv

Thank you. Did the IntendedFor change help?

hi, yes it did thanks!