Negative chi-square values in `neurosynth_decode` forward inference

const · September 10, 2026, 1:43pm

Summary of what happened:

While using NeurosynthDecoder to characterize an ROI, I noticed that its forward test can produce negative chi-square statistics and NaN results. It appears to use the mean selected study count across terms as the expected count for every term using one_way(n_selected_term, n_term), although their total study counts differ. This expected count can exceed a term’s total count.
With the default BH correction enabled, these NaNs propagate to the corrected forward results for all terms, including those with otherwise finite values.

The example below reproduces this using official Neurosynth v7 data. Is this a limitation of the intended method, or an issue with how the expected counts are calculated? How should these terms be handled?

Command used (and if a helper script was used, a link to the helper script or the command generated):

from nimare.extract import fetch_neurosynth
from nimare.decode.discrete import NeurosynthDecoder

dataset = fetch_neurosynth(
    version="7", source="abstract", vocab="terms",
    type="tfidf", return_type="dataset",
)[0]

ids = dataset.get_studies_by_coordinate([[-36, 18, 0]], r=12)

decoder = NeurosynthDecoder(correction=None)
decoder.fit(dataset)
result = decoder.transform(ids=ids)

print("Selected studies:", len(ids))
print("Terms:", len(result))
print(result[["pForward", "zForward"]].isna().sum())
result[["pForward", "zForward"]]

Version:

v0.21.0

Environment (Docker, Singularity / Apptainer, custom installation):

Python 3.12

Relevant log outputs (up to 20 lines):

Selected studies: 3568
Terms: 3228
pForward    343
zForward    343
dtype: int64
/usr/local/lib/python3.13/dist-packages/nimare/transforms.py:1316: RuntimeWarning: invalid value encountered in sqrt
  return np.minimum(_log_erfc(np.sqrt(chi2_values / 2.0)), 0.0)

Screenshots / relevant information:

jdkent · September 11, 2026, 6:31am

Hi @const!

Thanks for bringing this to our attention, I replicated the NaN results. You’re right, it should be one_way(n_selected_term, n_selected), which says of all the studies that report a result within the ROI (n_selected), how many contain the term of interest within that ROI (n_selected_term). one_way(n_selected_term, n_terms) is mixing the two marginal probabilities and is honestly a bit of a nonsensical statistic. With the BH correction propogating nans, that is another separate issue.

I’m fixing the one_way test here:

github.com/neurostuff/NiMARE

[FIX] Correct the Neurosynth decoder's uniformity test (#1138)

main ← jdkent:fix/neurosynth-decode-uniformity-test

opened 02:29AM - 11 Sep 26 UTC

jdkent

+331 -9

## Summary Closes the NaN / negative chi-squared report at https://neurostars.o…rg/t/negative-chi-square-values-in-neurosynth-decode-forward-inference/36387 `neurosynth_decode` called `one_way(n_selected_term, n_term)`. `one_way(data, n)` takes `n` to be the number of trials each count in `data` is drawn from, and every other caller passes a scalar — `MKDAChi2` passes `n_selected` (`nimare/meta/cbma/mkda.py:516`, `:648`), as does Neurosynth's own `MetaAnalysis`. Passing the per-label database count made the expected count `mean(n_selected_term)` exceed `n` for any label rarer than average, so `n - expected` went negative, the statistic went negative, and `chi2_to_nlogp` returned NaN out of `sqrt`. Decoding a 12 mm sphere at `[-36, 18, 0]` against Neurosynth v7: **343 NaN labels of 3228**. With the default BH correction those NaNs then propagate through `nlogp_fdr`'s `minimum.accumulate` and erase **all 3228** corrected p-values. The bug dates to `745b26cd` (2018), the commit that added the discrete decoders — the same commit that moved `one_way` out of `mkda.py` into `stats.py`, introducing the correct scalar usage and this incorrect vector usage side by side. It was invisible until #1099: `special.chdtrc(1, negative)` returns `1.0`, so a negative statistic used to become a silent p-value of 1 instead of a NaN. ## What the fix rests on Four independent sources agree that `n` should be `n_selected`: | Source | Forward statistic | `n` | |---|---|---| | `docs/decoding.rst` step 10 | mean count across labels | "the mean number of studies selected across labels" | | tsalo, [NeuroStars 18148](https://neurostars.org/t/nimare-decoding-results/18148) | "the NeurosynthDecoder is just a reformulation of the MKDA Chi2 approach" | `MKDAChi2` passes `n_selected` | | upstream `neurosynth` `MetaAnalysis` | `one_way(n_selected_active_voxels, n_selected)` | scalar | | NiMARE `MKDAChi2` | `one_way(n_selected_active_voxels, n_selected)` | scalar | `neurosynth_decode` was the only caller passing a vector. Note the docs already specified the correct algorithm; the code just never matched them. ## Changes 1. **`one_way(n_selected_term, n_selected)`.** Since every `n_selected_term <= n_selected`, the expected count cannot exceed `n`, so `n - expected` can no longer go negative. NaN goes 343 → 0, and 3228 → 0 after BH. 2. **`one_way` returns zero where it has no variance to measure.** Caught in review. The argument fix guarantees `n - expected >= 0`, not `> 0`. The expected count is the mean, so for counts bounded by `n` it hits 0 only when every count is 0 and hits `n` only when every count is `n` — both zero the `expected * (n - expected)` term. The numerator is zero in those cases too, so the statistic came back as 0/0 and every label decoded to NaN. Reachable from `neurosynth_decode` when no retained label appears in any selected study, or every retained label appears in all of them. The statistic is now zero there, which is what the arithmetic is reaching for: no cell deviates from uniformity, so there is nothing to report. This sits in `one_way` rather than in the caller because that is where the division is, and because `MKDAChi2` shares it — a meta-analysis in which no voxel is active in any selected study previously produced NaN maps. 3. **New `min_studies` parameter**, mirroring the argument of the same name on upstream's `MetaAnalysis`: an integer is a count of studies, a float in (0, 1) a proportion of the database. Yarkoni et al. (2011) excluded voxels active in fewer than 3% of studies "to stabilize results and ensure all cells had sufficient observations for the parametric Chi-Square test"; NiMARE had no equivalent floor. Two deliberate choices worth review: - **Default is 1**, matching upstream. It only drops labels appearing in no studies, which currently produce a silent divide-by-zero in `probForward`. No existing result changes unless a user opts in. - **Filters labels before the test, not after.** Upstream masks output images post hoc, but the uniformity test's expected count is the mean of `n_selected_term` across labels, so a sub-threshold label perturbs every *other* label's statistic, not just its own. On the reporter's ROI, 2644 of 3228 labels sit below the 3% floor and the expected count moves from 81.2 to 262.6 — a 3.2× shift in the number every retained label is compared against. Masking afterward would leave that contamination in place. 4. **Documents what the Forward columns are.** They are Neurosynth's uniformity test, a name Neurosynth adopted after concluding that "forward inference" was misleading. Because the test compares each label against the average label and never looks at the unselected studies, its ranking tracks literature frequency rather than the selection. Measured across three unrelated ROIs on Neurosynth v7: | ROI | top labels (uniformity / Forward) | |---|---| | L anterior insula `[-36,18,0]` | task, magnetic, resonance, magnetic resonance, functional magnetic | | Occipital `[0,-88,4]` | magnetic, task, resonance, magnetic resonance, functional magnetic | | L motor `[-40,-22,54]` | task, magnetic, resonance, magnetic resonance, functional magnetic | 6 of the top 8 are identical across all three, and the ranking correlates 0.90–0.94 (Spearman) with raw label document frequency. This is the label-wise counterpart of the caveat Neurosynth gives for the voxel-wise map. The docstrings and `docs/decoding.rst` now say so and point users at the Reverse (association) columns or `CorrelationDecoder`. This also resolves the confusion raised on NeuroStars in 2021 (#436), where a user asked why NiMARE reports `probForward`/`probReverse` when Neurosynth had renamed the maps. ## Not in scope - `nlogp_fdr`'s NaN propagation is unreachable from `neurosynth_decode` after this fix but remains a latent defect in a shared utility — separate PR. - Renaming the `pForward`/`zForward` columns outright would be a breaking API change; this PR only documents them. - Replacing the uniformity test with a per-label base-rate test (BrainMap style, as `brainmap_decode` does) is a methods change, not a bug fix, and would need its own citation rather than `yarkoni2011large`. ## Tests Eight new tests across `test_decode_discrete.py` and `test_stats.py`: a rare-label regression test that would have caught the original bug, a BH-correction test, an equality check against the longhand two-cell chi-squared on `n_selected` trials, coverage of `min_studies` as both count and proportion plus its error path, both degenerate-variance cases for `one_way`, a pin that the ordinary `one_way` path is unmoved, and an end-to-end check that a degenerate selection decodes to p=1 / z=0. 75 passed across `test_stats`, `test_decode_discrete`, `test_correct`, and `test_meta_mkda`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) ## Summary by Sourcery Fix Neurosynth decoding uniformity statistics and improve the handling and documentation of sparse labels. New Features: - Add configurable minimum-study filtering for Neurosynth decoding, accepting absolute counts or proportions. Bug Fixes: - Correct the Neurosynth uniformity test to use the selected-study count, preventing negative or NaN chi-squared statistics and invalid corrected p-values. - Return zero for degenerate one-way chi-squared cases with no measurable variance. Enhancements: - Clarify that Neurosynth Forward results represent a uniformity test driven largely by label frequency and distinguish them from Reverse association results. Documentation: - Update decoding documentation to describe the Forward/uniformity and Reverse/association tests and their interpretation. Tests: - Add regression and boundary tests covering finite uniformity results, multiple-comparison correction, minimum-study filtering, degenerate selections, and reference statistic equivalence.

and the BH error here:

github.com/neurostuff/NiMARE

[FIX] NaN-safe FDR correction, and a one-tailed z for BrainMap forward inference (#1139)

main ← jdkent:fix/nan-safe-fdr-and-brainmap-forward-tail

opened 02:35AM - 11 Sep 26 UTC

jdkent

+186 -8

Two independent defects found while investigating the `neurosynth_decode` NaN re…port (https://neurostars.org/t/negative-chi-square-values-in-neurosynth-decode-forward-inference/36387, fixed in #1138). Neither is caused by that bug and neither is fixed by it, so they are split out here. ## 1. One NaN erases an entire FDR correction `nlogp_fdr` sorts with `argsort`, which places NaN last, then runs `minimum.accumulate` over the reversed array. `np.minimum` propagates NaN, so the NaN is the first value the running minimum sees and everything behind it becomes NaN: ```python >>> nlogp_fdr(np.array([-20.0, -10.0, -1.0, np.nan, -5.0])) array([nan, nan, nan, nan, nan]) ``` One unevaluated test discards the whole family. This is how 343 NaN labels became 3228 NaN labels in the linked report, but the defect is in a shared utility used by `FDRCorrector`, `MKDAChi2`, and both discrete decoders, so it outlives that particular caller. **Fix:** NaN entries are held out of the step-up procedure and returned as NaN, while still counting toward the number of tests. That is the more conservative of the two readings and the one R's `p.adjust` takes — it drops NAs from the procedure while leaving `n` at the full length of the input: ```r if(all(nna <- !is.na(p))) nna <- TRUE p <- p[nna]; lp <- length(p) BH = { i <- lp:1L; ...; pmin(1, cummin(n/i * p[o]))[ro] } # n is the full length ``` With no NaN present, `lp == n` and the arithmetic is identical to before. A test pins that path byte-for-byte so the NaN handling cannot perturb ordinary use, and another checks the NaN case against the `n/i` factor computed longhand. `nlogp_bonferroni` adds a scalar per entry and never compares across them, so it was already NaN-safe. It gains a test rather than a change. ## 2. BrainMap forward inference converted a one-sided p-value with a two-tailed rule `brainmap_decode` computes forward inference as the upper tail of a binomial: ```python nlogp_fi = binom.logsf(k=n_selected_term, n=n_term_foci, p=p_selected) sign_fi = np.sign(n_selected_term - np.mean(n_selected_term)) ... z_corr_fi = nlogp_to_z(nlogp_corr_fi, "two") * sign_fi ``` `logsf` is the upper tail alone, so a small p-value can only mean *more* selected studies carry the label than the selection rate predicts. Converting it with the two-tailed rule and then multiplying by a sign taken from the mean label count meant a label could be reported as significantly **depleted** on the strength of a p-value that only ever tested enrichment — and the sign came from a comparison against the average label, which is not the quantity the binomial tested either. **Fix:** `zForward` is now one-tailed and unsigned. `zReverse` comes from a two-way chi-squared and remains two-tailed and signed. **This changes `zForward` values from `brainmap_decode`.** `pForward`, `zReverse`, and all effect sizes are unaffected. The docstring now states that forward inference here cannot report depletion. ## 3. The upper tail excluded the observation itself Caught in review. `binom.logsf(k)` is `P(X > k)`, but the one-sided p-value for an observed count `k` is `P(X >= k)`, so the argument has to be `k - 1`. I had originally flagged this as out of scope; that was the wrong call, because it interacts directly with the change above — dropping the sign turns what used to surface as a large *negative* z into a large *positive* one. Two reachable cases: - **A label whose every focus falls inside the selection** gets `P(X > n) == 0` exactly, so `pForward` underflowed to `4.94e-324` and `zForward` came out as `inf`. The `n_selected_term < 5` guard does not catch this, since such a label can carry any number of studies. On a 40-study synthetic where all 8 carriers of a label are selected and `p_selected` is 0.5, `zForward` was `inf` and is now `2.66`, with `pForward` exactly `0.5 ** 8`. - **A label observed zero times** gets `P(X > 0)`, which is small whenever the label is rare, so it reads as enrichment on the strength of never having been observed (z of 3.09 at `p_selected = 0.001`). The `< 5` guard does mask this one in the current code, but it is the same defect. `logsf(-1)` is `log(1)`, so a zero count needs no special handling. **This changes every `pForward` from `brainmap_decode`**: the previous values were anti-conservative by one term of the pmf. ## Tests Six new tests: NaN non-propagation for both `bh` and `by`, agreement with the `p.adjust` factor under a NaN, an exact-equality pin on the no-NaN path, the all-NaN case, Bonferroni pass-through, and a check that BrainMap's `zForward` is non-negative and matches a one-tailed conversion. 118 passed across `test_stats`, `test_decode_discrete`, `test_transforms`, and `test_correct`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) ## Summary by Sourcery Make FDR correction robust to unevaluated tests and align BrainMap forward inference with its one-sided enrichment test. Bug Fixes: - Prevent NaN values from propagating through and invalidating an entire FDR correction while preserving conservative multiple-testing counts. - Correct BrainMap forward-inference p-values to use the inclusive binomial upper tail and report one-tailed, non-negative z-values limited to enrichment. Enhancements: - Document the distinct directional interpretation of BrainMap forward and reverse inference. Tests: - Add regression coverage for NaN-safe BH and BY corrections, unchanged no-NaN behavior, all-NaN inputs, and Bonferroni NaN pass-through. - Add BrainMap tests covering one-tailed unsigned z-values, inclusive upper-tail probabilities, saturated and unobserved labels.

I’m also adding a min_studies argument, so you can filter out low frequency terms.

Please let me know if those fix your issue!

James

const · September 11, 2026, 8:35am

Thanks James, that makes sense! Thanks for fixing this so quickly.