Intro to datalad: searching through file JSONs/sidecars

1

Hi-

I am very new to datalad. I would like to search through available data repos and find data sets (specifically, phase/frequency dsets to use for B0 correction) with certain keys in their JSONs.

I have been going through the help/intros and trying various search features, but I would like to check that I am understanding the search correctly. My key of interest is EffectiveEchoSpacing, and so I searched as follows:

datalad search EffectiveEchoSpacing
search(ok): /data/PROJECTS/datasets.datalad.org/openfmri/ds000223 (dataset)

which returned 1 dataset. Basically, I was surprised that only 1 dataset would be returned. Is that correct? Are there other options/information I should be providing?

2 
$> datalad search --show-keys short | grep -A2 EffectiveEchoSpacing 
bids.EffectiveEchoSpacing
 in  53 datasets
 has 79 unique values: 0.93, 0.375, 0, 0.000460001, 0.000550002, 0.00066, 0.000520003, 0.000264999, 0 ....
[WARNING] In this search mode, the reported list of metadata keys may be incomplete

so should be present in 53 datasets at least BUT it has bids. prefix.

The tricky part here a bit (datalad search --help is indeed quite long by now to catch that) is that by default we search through dataset level metadata, not files. So try

datalad -c datalad.search.index-egrep-documenttype=all \
    search bids.EffectiveEchoSpacing:.*

:wink:
And to get a bit more informative output rendering try

datalad -c datalad.search.index-egrep-documenttype=all \
   -f '{parentds} {path}: {metadata[bids][EffectiveEchoSpacing]}' \
   search bids.EffectiveEchoSpacing:.*

PS see https://github.com/datalad/datalad/issues/3571 for a possible enhancement here TODO

3

@yarikoptic may I suggest that -c datalad.search.index-egrep-documenttype=all is aliased into a flag like -all? Can post the suggestion on github if it helps.