GSOC 2026 Project #37 : ATHENA Research Center / EBRAINS - Automating Performance Verification for the EBRAINS Software Distribution

GSoC
1

The EBRAINS Software Distribution (ESD) is a curated collection of tools for computational neuroscience, supporting education, research, and simulation across heterogeneous hardware — including HPC, GPUs, and neuromorphic computing systems. Many ESD tools are developed by the research community and leverage diverse hardware architectures for performance and scalability.

This project aims to integrate, parameterize, and automate a comprehensive benchmarking suite for key neural simulators within ESD. The focus will be on established benchmarks for widely used simulators like Arbor, NEURON, and NEST, creating a unified framework for performance validation, regression testing, and hardware profiling across EBRAINS cloud and HPC environments.

Key Objectives:

  1. Integrate and Unify Simulator Benchmarks
  2. Develop a framework to incorporate existing benchmark suites into spack package descriptions for:
  • Arbor: HPC-focused simulator benchmark suite
  • NEURON: Widely-used classic simulator performance tests
  • NEST: Large-scale network simulation benchmarks

  1. Parameterize Benchmark Configurations

  2. Design a flexible configuration leveraging the related ESD spack package descriptions system allowing users and CI systems to easily adjust:

  • Model size and complexity
  • Hardware targets (CPU core counts, GPU usage, MPI configuration)
  • Output granularity (performance summaries, detailed profiling)

  1. Embed Micro-Benchmarks for System Profiling and Performance Verification

  2. Integrate low-level performance tests into the ESD to complement simulator benchmarks:

  • OSU Micro-Benchmarks — Network and MPI performance
  • nccl-test — Multi-GPU communication scaling

Technology Stack & Required Skills:

  • Primary (Must Have): Python and Shell Scripting (Bash)
  • Secondary (Good to Have): SLURM, Spack Package Manager, System Benchmarking
2 Likes
2

My name is Ojas Vatsyayan, and I am currently pursuing my undergraduate degree in Computer Science and Mathematics at the University of Illinois Urbana-Champaign. My interests focus on systems engineering, performance optimization, and building scalable data-driven pipelines.

Over the past year, I have been working on projects involving backend systems, large-scale data processing, and performance-focused engineering. I also worked as an Engineering Product Specialist Intern at Cisco, where I developed systems for analyzing large datasets and worked on improving throughput and latency in networking environments.

In addition, I have worked on projects involving LLM systems and evaluation pipelines, where I focused on experimentation, benchmarking, and analyzing system behavior under different configurations. This has strengthened my interest in building reproducible and scalable benchmarking frameworks.

Because of my strong interest in systems performance and benchmarking infrastructure, I am particularly interested in this project’s focus on integrating and automating simulator benchmarks across heterogeneous hardware environments. I am especially interested in the parameterization and automation aspects of the benchmarking workflow.

I am planning to begin by exploring one of the simulator benchmarks (likely Arbor) and understanding how it can be parameterized and integrated into a reproducible workflow.

I would be grateful for any guidance on:

• how I could begin contributing to the repository
• whether there are existing benchmark integrations or components that would be good entry points
• any beginner tasks or areas where additional automation would be helpful

I would also be happy to share my GitHub projects or discuss my prior work in more detail if helpful.

Thank you very much for your time and for the work you are doing in building infrastructure for computational neuroscience.

Kind regards,
Ojas Vatsyayan

3

Dear Mr. Vatsyayan,

Thank you for reaching out.

Our objective is to include a set of micro and scientific benchmarks within the ESD, which will then be run using CI/CB at different EuroHPC centers.

For further details, please refer to these resources:

Best
Krishna (and the other ESD people)

4

Hi Krishna,

Thank you for the detailed response and for sharing these resources - this helps clarify the direction a lot. The idea of integrating both micro-benchmarks and simulator-level benchmarks into the ESD and running them through CI/CB across EuroHPC systems is super interesting.

I’ll start going through the OSU benchmarks, the NEST documentation, and the Arbor benchmark repository to better understand how these are currently structured.

Looking forward to exploring this further.

Best,
Ojas

5

We are planning to obtain detail profiling of these benchmarks on the Volta, A100 and H100 GPUs.

6

Hello Krishna and the ESD team,

I have gone through the project description and the discussion thread carefully.

From my understanding, the core goal is to build a unified benchmarking pipeline that runs simulator benchmarks (NEST, NEURON, Arbor) with configurable parameters, collects performance metrics, and enables regression detection across different hardware environments.

To better understand how such a system could be structured, I built a small prototype:
https://github.com/vikask011/benchmark-pipeline-demo

I have a couple of questions that would help me understand the project better:

  1. Should the framework use a standardized interface across all simulators, or is per-simulator execution and output parsing expected?

  2. For Spack integration, would benchmarks hook into spack test, or is a separate workflow expected?

I am currently going through the OSU Micro-Benchmarks documentation, the NEST random balanced network benchmark, and the Arbor ring benchmark repository.

If you have any feedback on the approach I’ve taken in the prototype, I would greatly appreciate it.

Thank you for your time.

Best,
Vikas K

7

Dear Vikas,
Thank you for your interest.

Indeed, this one of the main goal. Please, keep in mind, HPC centers may have diverse hardware architectures.

At present, one-simulator at a time.

Separate workflow. Please have a look a tmulti-area-model-wf.

CI/CB pipelines would be executed using Jacamar on selected HPC systems.

Best,
Krishna

8

Dear Krishna,

Thank you for the clarifications.

I understand that the framework should support one simulator at a time, with benchmarks executed through a separate workflow, and that CI/CB pipelines will run via Jacamar on selected HPC systems. I will go through the multi-area-model-wf and Jacamar documentation to better understand the expected workflow design and integration.

If there are any additional resources, internal references, or examples you would recommend, I would be grateful to have them.

Thank you again for your time and guidance.

Best,
Vikas K