Project Summary: NiMARE is a Python library for coordinate- and image-based neuroimaging meta-analysis. It provides rigorously tested implementations of widely used methods such as ALE and MKDA, enabling reproducible synthesis of large neuroimaging literatures. NiMARE also serves as a long-term maintenance layer for research methods that are often released as one-off academic code, ensuring sustainability, documentation, and integration into a unified statistical framework.
This project has two complementary objectives:
- Improve contributor onboarding and development documentation, reducing friction for new developers.
- Enhance NiMARE’s statistical capabilities by implementing adaptive early stopping for permutation-based inference (mandatory), and, time permitting, adding Signed Difference Mapping (SDM) and publication bias diagnostics as stretch goals.
- These improvements will directly benefit both NiMARE users and downstream platforms such as Neurosynth Compose (2,700+ registered users), which relies on NiMARE for backend statistical computation.
Project Scope (175 Hours)
Weeks 1–2: Contributor onboarding improvements
Weeks 3–8: Algorithm development
Mandatory: Adaptive early stopping for permutation-based inference
Stretch goals: SDM implementation and publication bias diagnostics
Phase 1: Contributor Onboarding Improvements (Weeks 1–2)
Motivation
While NiMARE has an existing contributor guide, elements of the development environment setup and git workflow documentation are outdated. Although there are no known critical bottlenecks, ensuring that documentation reflects current practices reduces barriers for new contributors and minimizes avoidable friction.
Deliverables
Updated contributor guide covering:
Environment setup (Python versioning, dependency management, testing)
CI expectations
Code style and linting requirements
Clear documentation of recommended git workflow for PR submission.
Review and cleanup of outdated sections.
Verification that documentation reflects current repository structure and tooling.
Success Criteria
Documentation validated through a clean setup from scratch.
Clear, reproducible development environment instructions.
No outdated workflow steps.
All changes merged and published in documentation site.
Phase 2: Algorithm Development (Weeks 3–8)
- Adaptive Early Stopping for Permutation-Based Inference (Mandatory)
Motivation
Permutation-based Monte Carlo inference is central to ALE and MKDA workflows but can be computationally expensive. Currently, users often rely on fixed iteration counts (e.g., 10,000 permutations), which may be either insufficient or excessive.
This project will implement a fixed-width confidence interval stopping rule for permutation testing in both ALE and MKDA workflows. The objective is to:
Maintain strict type I error control.
Reduce unnecessary computational cost.
Dynamically determine when sufficient precision has been achieved.
Technical Plan
Implement fixed-width confidence interval stopping based on variance stabilization of p-value estimates.
Evaluate whether 10,000 iterations are sufficient or unnecessarily conservative.
Integrate stopping logic into permutation pipelines for ALE and MKDA.
Preserve statistical validity and type I error guarantees.
Validation
Benchmark against fixed 10,000-permutation runs.
Compare convergence behavior.
Confirm preservation of nominal type I error bounds using simulation.
Deliverables
Integrated early stopping implementation for ALE and MKDA.
Unit tests covering convergence logic.
≥85% test coverage maintained.
Benchmarking results documented.
- Signed Difference Mapping (SDM) (Stretch Goal)
Motivation
Signed Difference Mapping allows integration of coordinate- and image-based meta-analytic data within a unified framework. Implementing one SDM algorithm would broaden NiMARE’s methodological coverage and support mixed-modality analyses.
Technical Considerations
Requires extension of NiMARE’s Estimator base class.
API changes may be necessary to support SDM-specific data structures.
Must preserve backward compatibility where possible.
Validation
Compare output against existing SDM implementations using available validation datasets.
Confirm statistical consistency with reference results.
Deliverables
One SDM estimator implemented.
Necessary base class modifications.
Comprehensive unit tests.
Documentation describing usage and assumptions.
Risk Mitigation
Any API changes will be versioned carefully.
Backward compatibility impacts will be explicitly documented.
Core NiMARE workflows will be regression tested.
- Publication Bias Diagnostic (Stretch Goal)
-
Motivation- Neuroimaging meta-analyses may be susceptible to publication bias. A common diagnostic approach is estimating how many null studies would need to be added to render results non-significant (fail-safe N–style methods).
-
This project will implement a standalone diagnostic module to estimate how many null studies would be required to overturn significance.
-
Technical Challenges
Computational scaling if large numbers of simulated null studies are required.
Efficient simulation strategies may be necessary.
Deliverables
Standalone diagnostic module.
Simulation-based validation.
Unit tests with ≥85% coverage.
Documentation describing assumptions and limitations.
Validation & Testing Standards
-
All new code must meet NiMARE’s testing standards (≥85% coverage).
-
Existing workflows must pass regression tests.
-
Simulation-based validation datasets are available.
-
Continuous integration checks must pass for all PRs.
Risks and Mitigation
Risk
Mitigation Strategy
API instability due to SDM integration
Careful extension of Estimator base class; preserve backward compatibility where possible
Computational scaling for fail-safe diagnostic
Implement optimized simulation strategies; cap simulation growth
Statistical correctness of early stopping
Simulation studies to verify type I error control
The mandatory early stopping implementation is feasible within the timeline. Stretch goals will be pursued sequentially based on progress.
Impact: This project will:
-
Improve computational efficiency in permutation-based inference.
-
Expand NiMARE’s methodological coverage.
-
Strengthen reproducibility and sustainability of neuroimaging meta-analysis.
-
Improve developer onboarding, lowering contribution barriers.
-
Because NiMARE serves as the statistical backend for Neurosynth Compose (2,700+ registered users), improvements will directly affect real-world meta-analytic workflows.