I have released a small tool for a problem I kept running into: there is no ground truth for most of what our pipelines report, so when a new version shifts the numbers there is no way to tell whether it got better.
bidsgate takes a BIDS dataset, injects a known change into real images, writes a BIDS derivative with the truth next to it, and scores whatever BIDS app you run on the result. Two injectors so far: lesions (ellipsoids of 30 to 1500 mm3, FLAIR-bright and T1w-dark relative to the subject’s own white matter, placed inside a brain mask with a volume gate) and global atrophy (a smooth radial contraction by a known factor). Scoring gives Dice, lesion-wise sensitivity by size, false positives beyond a 2 mm margin, volume ratio, and for atrophy the recovered fraction of the injected change. One HTML scorecard per run.
First public result: LST-AI v2.0.0rc1 on six OpenNeuro ds007908 healthy controls found 64 of 72 injected lesions (mean Dice 0.82). The misses were not about size: lesions of 26 to 30 mm3 were found in every subject, and seven of the eight misses were among the four lowest-placed lesions of their subject, at cerebellum and brainstem level. Whether that is the model or the injection is exactly the kind of question I would like the tool to make askable.
For morphometry tools (FreeSurfer, FastSurfer, SIENAX-style pipelines) one caveat: the skull contracts with the brain inside the falloff zone, so anything normalised to intracranial volume will see less change than was injected; raw volumes are the fair comparison.
pip install bidsgate
bidsgate inject-lesions /data/bids --out /data/derivatives/bidsgate-lesions
# run your segmenter
bidsgate score-lesions --truth ... --pred "..../{subject}/{base}_seg.nii.gz" --pipeline mytool
Nothing in it is evidence about any disease; it is a test of software. I would welcome feedback on what to inject next (regional atrophy with a mask, fMRI activation) and on the placement rules.