pyBids derivative validation error running fitlins on fmriprep cifti output

jsmentch · September 22, 2020, 11:55pm

2 part question, if anyone can help, I am trying to run fitlins using cifti outputs from fmriprep.

First: I’m getting an error in fitlins from the pybids validator:

Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/neuro/bin/fitlins", line 8, in <module>
    sys.exit(main())
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/fitlins/cli/run.py", line 271, in main
    sys.exit(run_fitlins(sys.argv[1:]))
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/fitlins/cli/run.py", line 218, in run_fitlins
    reset_database=reset_database)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/bids/layout/layout.py", line 209, in __init__
    reset_database=reset_database
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/bids/layout/layout.py", line 604, in add_derivatives
    MANDATORY_DERIVATIVES_FIELDS['PipelineDescription.Name'])
bids.exceptions.BIDSDerivativesValidationError: Every valid BIDS-derivatives dataset must have a PipelineDescription.Name field set inside 'dataset_description.json'.
Example: {'PipelineDescription': {'Name': 'Example pipeline'}}

but my derivative dataset_description has a name field that is filled out:
Do I need to add another field or change the formatting? I’ve experimented with this but was not successful yet.

{
    "Name": "fMRIPrep - fMRI PREProcessing workflow",
    "BIDSVersion": "1.4.0",
    "DatasetType": "derivative",
    "GeneratedBy": [
        {
            "Name": "fMRIPrep",
            "Version": "20.2.0rc0",
            "CodeURL": "https://github.com/poldracklab/fmriprep/archive/20.2.0rc0.tar.gz"
        }
    ],
    "HowToAcknowledge": "Please cite our paper (https://doi.org/10.1038/s41592-018-0235-4), and include the generated citation boilerplate within the \
Methods section of the text."
}

Second question, does anyone know how to specify that I would like to run fitlins using the cifti data and not the standard nifti data? Is there something in particular to specify in the --space or --desc-label flags? I’ve been unable to figure this out from going through fitlins docs/github page

This is the command I was trying to run using singularity

singularity run --cleanenv -B /om2,/scratch/ fitlins-latest.simg /om2/user/jsmentch/data/datalad/ds001110 out/ participant
-v
–participant-label 19
-m /om2/user/jsmentch/neuroscout/SherlockMerlin/neuroscout-bundles/MNQZW/model.json
–desc-label dtseries
-d /om2/user/jsmentch/data/datalad/ds001110/derivatives
-w /scratch/Fri/jsmentch

(I’m reusing a model from neuroscout so may need to fix that part up )

Thanks!

effigies · September 23, 2020, 2:00pm

We’re going to need to update PyBIDS to handle BIDS 1.4.0 derivatives.

To select CIFTIs, use --space fsLR.

jsmentch · September 24, 2020, 12:40am

Ah, fsLR makes a lot of sense, thank you!

For now, before PyBIDS is updated, I’m trying to modify the dataset_description.json file to conform with PyBIDS as per the github issue but running into some errors. I’ve modified it to have a PipelineDescription.Name see below, and a few permutations of that but it doesn’t seem to be working… Do you know if there is another way to squeeze this by the validator or what might be going wrong?

{
    "Name": "fMRIPrep - fMRI PREProcessing workflow",
    "BIDSVersion": "1.4.0",
    "DatasetType": "derivative",
    "PipelineDescription": [
        {
            "Name": "fMRIPrep"
        }
    ],
    "GeneratedBy": [
        {
            "Name": "fMRIPrep",
            "Version": "20.2.0rc0",
            "CodeURL": "https://github.com/poldracklab/fmriprep/archive/20.2.0r\
c0.tar.gz"
        }
    ],
    "HowToAcknowledge": "Please cite our paper (https://doi.org/10.1038/s41592-\
018-0235-4), and include the generated citation boilerplate within the Methods \
section of the text."
}

Error:

AttributeError: 'list' object has no attribute 'get'
bash-4.2$ singularity run --cleanenv -B /om2/user/jsmentch,/scratch fitlins-latest.simg /om2/user/jsmentch/data/datalad/ds001110 out/ participant -v --participant-label 19 -m /om2/user/jsmentch/neuroscout/SherlockMerlin/neuroscout-bundles/MNQZW/model.json --space fsLR -d /om2/user/jsmentch/data/datalad/ds001110/derivatives/fmriprep -w /scratch/Fri/jsmentch
Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/neuro/bin/fitlins", line 8, in <module>
    sys.exit(main())
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/fitlins/cli/run.py", line 271, in main
    sys.exit(run_fitlins(sys.argv[1:]))
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/fitlins/cli/run.py", line 218, in run_fitlins
    reset_database=reset_database)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/bids/layout/layout.py", line 209, in __init__
    reset_database=reset_database
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/bids/layout/layout.py", line 597, in add_derivatives
    'PipelineDescription', {}).get('Name')
AttributeError: 'list' object has no attribute 'get'

effigies · September 24, 2020, 1:49pm

PipelineDescription was just one JSON object. It got changed to a list when renamed to GeneratedBy.

jsmentch · September 25, 2020, 7:11pm

Thanks, it seems I was formatting the .json file incorrectly with pipeline description as a list. I got past this error by replacing GeneratedBy with this if anyone else runs into this:

"PipelineDescription":      {
            "Name": "fMRIPrep"
        },

Since then, I’ve been running into more errors like this:

Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/neuro/bin/fitlins", line 8, in <module>
    sys.exit(main())
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/fitlins/cli/run.py", line 271, in main
    sys.exit(run_fitlins(sys.argv[1:]))
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/fitlins/cli/run.py", line 263, in run_fitlins
    analysis.setup(**selectors)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/bids/analysis/analysis.py", line 98, in setup
    step.setup(input_contrasts, drop_na=drop_na, **selectors)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/bids/analysis/analysis.py", line 256, in setup
    **kwargs)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/bids/layout/layout.py", line 960, in get_collections
    skip_empty=skip_empty, **kwargs)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/bids/variables/io.py", line 93, in load_variables
    dataset = _load_time_variables(layout, dataset, scope=scope, **_kwargs)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/bids/variables/io.py", line 186, in _load_time_variables
    raise ValueError("No functional images that match criteria found.")
ValueError: No functional images that match criteria found.

This is when i set --space to fsLR or fsLR_den-91k , neither of which finds the file(sub-19_task-MerlinMovie_space-fsLR_den-91k_bold.dtseries.nii) I’m wondering if maybe this is due to it having the double extension .dtseries.nii? But I see there was a fix for that already.

I don’t encounter this error if I set it to --space MNI152NLin2009cAsym

jsmentch · September 28, 2020, 10:14pm

still unable to run fitlins with the cifti output, but another error I am receiving which neglected to mention is:
TypeError: __init__() got an unexpected keyword argument 'initializer'

Are there other differences for running CIFTI through besides specifying --space fsLR? Can’t find any more info on this.

The full error I’m receiving is:

bash-4.2$ singularity run --cleanenv -B /om2/user/jsmentch,/scratch fitlins-latest.simg /om2/user/jsmentch/data/datalad/ds001110 out/ participant -v --participant-label 19
-m /om2/user/jsmentch/neuroscout/SherlockMerlin/neuroscout-bundles/MNQZW/model-19.json --space fsLR -d /om2/user/jsmentch/data/datalad/ds001110/derivatives/fmriprep -w /scr
atch/Fri/jsmentch
200928-16:34:58,983 nipype.utils WARNING:
         A newer version (1.5.1) of nipy/nipype is available. You are using 1.5.0
200928-16:34:59,474 nipype.workflow INFO:
         Workflow fitlins_wf settings: ['check', 'execution', 'logging', 'monitoring']
200928-16:34:59,526 nipype.workflow INFO:
         Running in parallel.
200928-16:34:59,536 nipype.workflow INFO:
         [MultiProc] Running 0 tasks, and 1 jobs ready. Free memory (GB): 226.54/226.54, Free processors: 40/40.
200928-16:35:00,151 nipype.workflow INFO:
         [Node] Setting-up "fitlins_wf.loader" in "/scratch/Fri/jsmentch/fitlins_wf/loader".
200928-16:35:00,209 nipype.workflow INFO:
         [Node] Running "loader" ("fitlins.interfaces.bids.LoadBIDSModel")
200928-16:35:01,529 nipype.workflow WARNING:
         Storing result file without outputs
200928-16:35:01,535 nipype.workflow INFO:
         [MultiProc] Running 1 tasks, and 0 jobs ready. Free memory (GB): 226.34/226.54, Free processors: 39/40.
                     Currently running:
                       * fitlins_wf.loader
200928-16:35:01,541 nipype.workflow WARNING:
         [Node] Error on "fitlins_wf.loader" (/scratch/Fri/jsmentch/fitlins_wf/loader)
200928-16:35:03,537 nipype.workflow ERROR:
         Node loader failed to run on host node007.cm.cluster.
200928-16:35:03,538 nipype.workflow ERROR:
         Saving crash info to /scratch/Fri/jsmentch/crash-20200928-163503-jsmentch-loader-35421cea-9531-43db-b665-37d7ec945aa2.txt
Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py", line 148, in __init__
    mp_context=mp_context,
TypeError: __init__() got an unexpected keyword argument 'initializer'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 397, in run
    runtime = self._run_interface(runtime)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/fitlins/interfaces/bids.py", line 220, in _run_interface
    analysis.setup(drop_na=False, **selectors)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/bids/analysis/analysis.py", line 98, in setup
    step.setup(input_contrasts, drop_na=drop_na, **selectors)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/bids/analysis/analysis.py", line 256, in setup
    **kwargs)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/bids/layout/layout.py", line 960, in get_collections
    skip_empty=skip_empty, **kwargs)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/bids/variables/io.py", line 93, in load_variables
    dataset = _load_time_variables(layout, dataset, scope=scope, **_kwargs)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/bids/variables/io.py", line 186, in _load_time_variables
    raise ValueError("No functional images that match criteria found.")
ValueError: No functional images that match criteria found.

200928-16:35:03,545 nipype.workflow INFO:
         [MultiProc] Running 0 tasks, and 0 jobs ready. Free memory (GB): 226.54/226.54, Free processors: 40/40.
200928-16:35:05,536 nipype.workflow INFO:
         ***********************************
200928-16:35:05,537 nipype.workflow ERROR:
         could not run node: fitlins_wf.loader
200928-16:35:05,537 nipype.workflow INFO:
         crashfile: /scratch/Fri/jsmentch/crash-20200928-163503-jsmentch-loader-35421cea-9531-43db-b665-37d7ec945aa2.txt
200928-16:35:05,537 nipype.workflow INFO:
         ***********************************
Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/neuro/bin/fitlins", line 8, in <module>
    sys.exit(main())
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/fitlins/cli/run.py", line 271, in main
    sys.exit(run_fitlins(sys.argv[1:]))
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/fitlins/cli/run.py", line 263, in run_fitlins
    analysis.setup(**selectors)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/bids/analysis/analysis.py", line 98, in setup
    step.setup(input_contrasts, drop_na=drop_na, **selectors)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/bids/analysis/analysis.py", line 256, in setup
    **kwargs)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/bids/layout/layout.py", line 960, in get_collections
    skip_empty=skip_empty, **kwargs)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/bids/variables/io.py", line 93, in load_variables
    dataset = _load_time_variables(layout, dataset, scope=scope, **_kwargs)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/bids/variables/io.py", line 186, in _load_time_variables
    raise ValueError("No functional images that match criteria found.")
ValueError: No functional images that match criteria found.

jsmentch · October 1, 2020, 4:51pm

For anyone else that runs into this, this particular error was bypassed by adding: --desc-label '' to the fitlins command. It was not finding the cifti file because by default it was looking for a file with *desc-preproc_bold.nii.gz

and the file in question has a name without that e.g.:
sub-19_task-MerlinMovie_space-fsLR_den-91k_bold.dtseries.nii

jdknotts · May 22, 2023, 2:38pm

I’m running into this “PipelineDescription” error, and have tried modifying my ‘dataset_description.json’ file per this post (also tried the solution here), but can’t get around the error.

Here’s the code, where rawDir is the path to my rawdata directory (I have confirmed that running this without derivatives=True successfully generates a layout instance for the rawdata directory):

layout = bids.BIDSLayout(rawDir, derivatives=True)

And it generates the same error as mentioned above:

BIDSDerivativesValidationError: Every valid BIDS-derivatives dataset must have a PipelineDescription.Name field set inside 'dataset_description.json'. 
Example: {'PipelineDescription': {'Name': 'Example pipeline'}}

This is the current ‘dataset_description.json’ file in my derivatives directory:

{
    "Name": "fMRIPrep - fMRI PREProcessing workflow",
    "BIDSVersion": "1.4.0",
    "DatasetType": "derivative",
    "PipelineDescription":{
        "Name": "fMRIPrep_v20.2.6"
    },
    "HowToAcknowledge": "Please cite our paper (https://doi.org/10.1038/s41592-018-0235-4), and include the generated citation boilerplate within the Methods section of the text.",
    "License": "_insert license info_ "
}

And on the off chance that this has to do with the way my data are structured, here’s my layout:

├─ rawdata/
│ ├─derivatives/
│ │ ├─ fmriprep_v20.2.6/
│ │ │ ├─ fmriprep/
│ │ │ │ ├─ logs/
│ │ │ │ ├─ sub-S01/
│ │ │ │ ├─ dataset_description.json
│ │ │ │ ├─ desc-aparcaseg_dseg.tsv
│ │ │ │ ├─ desc-aseg_dseg.tsv
│ │ │ │ ├─ sub-S01.html
│ │ │ ├─ freesurfer/

Any help would be very appreciated! Thanks!

jdknotts · May 22, 2023, 8:23pm

Update: the following solved the issue. First, I updated pybids to 0.16.1 and got essentially the same error only swapping out “PipelineDescription” for “GeneratedBy” (as discussed above). Then I tried removing the “fmriprep_v20.2.6” folder from my original file structure so it now looks like this:

├─ rawdata/
│ ├─derivatives/
│ │ ├─ fmriprep/
│ │ │ ├─ logs/
│ │ │ ├─ sub-S01/
│ │ │ ├─ dataset_description.json
│ │ │ ├─ desc-aparcaseg_dseg.tsv
│ │ │ ├─ desc-aseg_dseg.tsv
│ │ │ ├─ sub-S01.html
│ │ ├─ freesurfer/

And now it’s working. So I’m not sure if the pybids update was necessary, as it seems to have been restructuring the data that solved the issue.