pyAFQ ValueError: No file found with these parameters:

Summary of what happened:

Dear Neurostars community,

I am trying to run tractography with pyAFQ on custom bundles defined with two ROIs. Here, MT and PT from the left hemisphere. I am following this tutorial, and trying to use this with my own data : Using Subject Space ROIs from Freesurfer — AFQ 2.0 documentation

Command used (and if a helper script was used, a link to the helper script or the command generated):

import os
import os.path as op
import plotly
from AFQ.api.group import GroupAFQ
import AFQ.data.fetch as afd
from AFQ.definitions.image import RoiImage
import AFQ.api.bundle_dict as abd
################## Run pyAFQ     ##################
bids_path = os.path.expanduser("~/Documents/research/ampb_mt_tractometry_analysis/tests/pyAFQ_tests/afq-functionalROI")#anat"

#Set tractography parameters
tracking_params = dict(n_seeds=10000,
                    random_seeds=True,
                    rng_seed=42,
                    seed_mask=RoiImage(use_endpoints=True))
#Define custom bundle dict
bundles = abd.BundleDict({
    "L_OR": {
        "start": {
            "scope": "functionalROIs",
            "suffix": "mask",
            "desc": "lhMT"
        },
        "end": {
            "scope": "functionalROIs",
            "suffix": "mask",
            "desc": "lhPT"
        },
        "cross_midline": False,
        "space": "subject"
    }})
#Initialize groupAFQ object
myafq = GroupAFQ(
    bids_path=bids_path,
    preproc_pipeline='qsiprep',
    tracking_params=tracking_params,
    bundle_info=bundles)

bundle_html = myafq.export("indiv_bundles_figures")
plotly.io.show(bundle_html["01"]["L_OR"])

Version:

Environment (Docker, Singularity / Apptainer, custom installation):

Python

Data formatted according to a validatable standard? Please provide the output of the validator:

I have tried making all folders BIDS compatible, though here is the bids validator output (one error, several warnings, though the error should not be a big deal since the tutorial dataset does not have the sub- folders either):

Relevant log outputs (up to 20 lines):

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File ~/Documents/research/ampb_mt_tractometry_analysis/tests/pyAFQ_tests/afq-functionalROI/code/2_groupAFQ.py:25
      9 bundles = abd.BundleDict({
     10     "L_OR": {
     11         "start": {
   (...)
     22         "space": "subject"
     23     }})
     24 #Initialize groupAFQ object
---> 25 myafq = GroupAFQ(
     26     bids_path=bids_path,
     27     preproc_pipeline='qsiprep',
     28     tracking_params=tracking_params,
     29     bundle_info=bundles)
     31 bundle_html = myafq.export("indiv_bundles_figures")
     32 plotly.io.show(bundle_html["01"]["L_OR"])
File /opt/anaconda3/envs/bids-env/lib/python3.9/site-packages/AFQ/api/group.py:377, in GroupAFQ.__init__(self, bids_path, bids_filters, preproc_pipeline, participant_labels, output_dir, parallel_params, bids_layout_kwargs, **kwargs)
    374 if "bundle_info" in this_kwargs and isinstance(
    375         this_kwargs["bundle_info"], abd.BundleDict):
    376     for b_name in this_kwargs["bundle_info"].bundle_names:
--> 377         this_kwargs["bundle_info"].apply_to_rois(
    378             b_name,
    379             this_kwargs["bundle_info"]._use_bids_info,
    380             bids_layout, dwi_data_file, subject, session,
    381             dry_run=False)
    383 self.valid_sub_list.append(subject)
    384 self.valid_ses_list.append(str(session))
File /opt/anaconda3/envs/bids-env/lib/python3.9/site-packages/AFQ/api/bundle_dict.py:979, in BundleDict.apply_to_rois(self, b_name, func, dry_run, apply_to_recobundles, *args, **kwargs)
    977 if roi_type in self._dict[b_name]:
    978     if roi_type in ["start", "end", "prob_map"]:
--> 979         return_vals[roi_type] = func(
    980             self._dict[b_name][roi_type], *args, **kwargs)
    981     else:
    982         changed_rois = []
File /opt/anaconda3/envs/bids-env/lib/python3.9/site-packages/AFQ/api/bundle_dict.py:827, in BundleDict._use_bids_info(self, roi_or_sl, bids_layout, bids_path, subject, session)
    825 if isinstance(roi_or_sl, dict):
    826     suffix = roi_or_sl.get("suffix", "dwi")
--> 827     roi_or_sl = find_file(
    828         bids_layout, bids_path,
    829         roi_or_sl,
    830         suffix,
    831         session, subject)
    832     return roi_or_sl
    833 else:
File /opt/anaconda3/envs/bids-env/lib/python3.9/site-packages/AFQ/definitions/utils.py:125, in find_file(bids_layout, path, filters, suffix, session, subject, extension, required)
    123 # Nothing is found
    124 if nearest is None:
--> 125     return _ff_helper(required, (
    126         "No file found with these parameters:\n"
    127         f"suffix: {suffix},\n"
    128         f"session (searched with and without): {session},\n"
    129         f"subject: {subject},\n"
    130         f"filters: {filters},\n"
    131         f"near path: {path},\n"))
    133 path_subject = bids_layout.parse_file_entities(path).get(
    134     "subject", None
    135 )
  136 file_subject = bids_layout.parse_file_entities(nearest).get(
    137     "subject", None
    138 )

File /opt/anaconda3/envs/bids-env/lib/python3.9/site-packages/AFQ/definitions/utils.py:85, in _ff_helper(required, err_msg)
     83 def _ff_helper(required, err_msg):
     84     if required:
---> 85         raise ValueError(err_msg)
     86     else:
     87         logger.warning(err_msg)

ValueError: No file found with these parameters:
suffix: mask,
session (searched with and without): 04,
subject: NSxLxPQx1973,
filters: {'scope': 'functionalROIs', 'suffix': 'mask', 'desc': 'lhMT', 'extension': '.nii.gz'},
near path: ~/Documents/research/ampb_mt_tractometry_analysis/tests/pyAFQ_tests/afq-functionalROI/derivatives/qsiprep/sub-NSxLxPQx1973/ses-04/dwi/sub-NSxLxPQx1973_ses-04_acq-HCPdir99_space-ACPC_desc-preproc_dwi.nii.gz,

Screenshots / relevant information:


This error seems to only occur when I use this bids filter structure:

bundles = abd.BundleDict({
    "L_OR": {
        "start": {
            "scope": "functionalROIs",
            "suffix": "mask",
            "desc": "lhMT"
        },
        "end": {
            "scope": "functionalROIs",
            "suffix": "mask",
            "desc": "lhPT"
        },
        "cross_midline": False,
        "space": "subject"
    }})

but not when I specify the actual ROI directory. I don’t want to use specific ROI directories, as I am running this code on a whole dataset.

Thanks,
Best,
-Loïc

Hi @ldaumail,

In the future, please organize your post with the Software Support post category template. You can see I edited in for you this time.

I am not sure we can effectively debug this without seeing relevant tree directory structures. Also what PyAFQ version are you using?

Best,
Steven

Hi @Steven,
Thanks for your reply.
Sure, my bad, will do.

pyAFQ version: 1.3.6. I saw pyAFQ 2.0 was released today (I tried it too, and am getting the same error), though have been trouble shooting on this error for a couple days already.

afq-functionalROI/
├── code/
│   ├── 1_bis_dcm2bids.sh
│   ├── 2_groupAFQ.py
│   └── utils/
│       ├── config.json
│       └── subjects.txt
├── dataset_description.json
└── derivatives/
    ├── afq/
    ├── functionalROIs/
    │   ├── dataset_description.json
    │   └── sub-NSxLxPQx1973/
    │       └── ses-04/
    │           └── anat/
    │               ├── sub-NSxLxPQx1973_ses-04_desc-freesurfer_T1w.nii.gz
    │               ├── sub-NSxLxPQx1973_ses-04_desc-lhMT_mask.nii.gz
    │               └── sub-NSxLxPQx1973_ses-04_desc-lhPT_mask.nii.gz
    └── qsiprep/
        ├── dataset_description.json
        ├── sub-NSxLxPQx1973/
        │   ├── anat/
        │   │   ├── sub-NSxLxPQx1973_from-ACPC_to-anat_mode-image_xfm.mat
        │   │   ├── sub-NSxLxPQx1973_from-ACPC_to-MNI152NLin2009cAsym_mode-image_xfm.h5
        │   │   ├── sub-NSxLxPQx1973_from-anat_to-ACPC_mode-image_xfm.mat
        │   │   ├── sub-NSxLxPQx1973_from-MNI152NLin2009cAsym_to-ACPC_mode-image_xfm.h5
        │   │   ├── sub-NSxLxPQx1973_space-ACPC_desc-aseg_dseg.nii.gz
        │   │   ├── sub-NSxLxPQx1973_space-ACPC_desc-brain_mask.nii.gz
        │   │   ├── sub-NSxLxPQx1973_space-ACPC_desc-preproc_T1w.json
        │   │   ├── sub-NSxLxPQx1973_space-ACPC_desc-preproc_T1w.nii.gz
        │   │   └── sub-NSxLxPQx1973_space-ACPC_dseg.nii.gz
        │   ├── figures/
        │   │   ├── sub-NSxLxPQx1973_about.html
        │   │   └── sub-NSxLxPQx1973_summary.html
        │   ├── log/
        │   │   ├── 20241220-183146_df8cad90-1ee6-4e4e-ba80-78d2f91e54b8/
        │   │   │   ├── crash-20241220-184059-ldaumail3-denoiser-c50e6f4a-90be-4d8e-8d84-d3f9f2033e6c.txt
        │   │   │   └── qsiprep.toml
        │   │   ├── 20241220-185120_4c071914-8c4d-475d-9893-91ec05be6c33/
        │   │   │   └── qsiprep.toml
        │   │   └── 20241220-200120_105caedc-37db-40e7-b82c-febf8b948f38/
        │   │       └── qsiprep.toml
        │   └── ses-04/
        │       ├── dwi/
        │       │   ├── sub-NSxLxPQx1973_ses-04_acq-HCPdir99_desc-confounds_timeseries.tsv
        │       │   ├── sub-NSxLxPQx1973_ses-04_acq-HCPdir99_space-ACPC_desc-brain_mask.nii.gz
        │       │   ├── sub-NSxLxPQx1973_ses-04_acq-HCPdir99_space-ACPC_desc-image_qc.csv
        │       │   ├── sub-NSxLxPQx1973_ses-04_acq-HCPdir99_space-ACPC_desc-preproc_dwi.b
        │       │   ├── sub-NSxLxPQx1973_ses-04_acq-HCPdir99_space-ACPC_desc-preproc_dwi.b_table.txt
        │       │   ├── sub-NSxLxPQx1973_ses-04_acq-HCPdir99_space-ACPC_desc-preproc_dwi.bval
        │       │   ├── sub-NSxLxPQx1973_ses-04_acq-HCPdir99_space-ACPC_desc-preproc_dwi.bvec
        │       │   ├── sub-NSxLxPQx1973_ses-04_acq-HCPdir99_space-ACPC_desc-preproc_dwi.json
        │       │   ├── sub-NSxLxPQx1973_ses-04_acq-HCPdir99_space-ACPC_desc-preproc_dwi.nii.gz
        │       │   ├── sub-NSxLxPQx1973_ses-04_acq-HCPdir99_space-ACPC_desc-slice_qc.json
        │       │   ├── sub-NSxLxPQx1973_ses-04_acq-HCPdir99_space-ACPC_dwiref.nii.gz
        │       │   ├── sub-NSxLxPQx1973_ses-04_acq-HCPdir99_space-ACPC_model-eddy_stat-cnr_dwimap.json
        │       │   └── sub-NSxLxPQx1973_ses-04_acq-HCPdir99_space-ACPC_model-eddy_stat-cnr_dwimap.nii.gz
        │       └── figures/
        │           ├── sub-NSxLxPQx1973_ses-04_acq-HCPdir99_desc-carpetplot_dwi.svg
        │           ├── sub-NSxLxPQx1973_ses-04_acq-HCPdir99_desc-coreg_dwi.svg
        │           ├── sub-NSxLxPQx1973_ses-04_acq-HCPdir99_desc-sdc_b0.svg
        │           ├── sub-NSxLxPQx1973_ses-04_acq-HCPdir99_desc-summary_dwi.html
        │           ├── sub-NSxLxPQx1973_ses-04_acq-HCPdir99_desc-topupsummary_dwi.html
        │           ├── sub-NSxLxPQx1973_ses-04_acq-HCPdir99_dir-AP_desc-denoising_dwi.svg
        │           └── sub-NSxLxPQx1973_ses-04_acq-HCPdir99_dir-PA_desc-denoising_dwi.svg
        └── sub-NSxLxPQx1973.html

I guess my issue comes down to: how does one create a BIDS compatible directory with those custom based ROIs? What tool can we use? I assume dcm2bids won’t work here given that my binary masks are already in .nii.gz format? What needs to be included in the dataset_description.json file of the folder that contains ROIs for all subjects?

Thanks

OK - on some further examination (and some back and forth communication with the OP) the cause of the original post here was that the derivative folder in which the ROIs were stored was called “functionalROIs”, but the BIDS dataset_description file had the following:

{"Name": "AMPB", "PipelineDescription": {"Name": "freesurfer"}, "GeneratedBy": [{"Name": "freesurfer"}], "BIDSVersion": "1.4.0"}

And pyBIDS follows the scope based on the contents of the dataset_description and not the folder name. So changing

"scope": "freesurfer",

resolved that issue.