The EBRAINS Software Distribution (ESD) is a curated collection of tools for computational neuroscience, supporting education, research, and simulation across heterogeneous hardware — including HPC, GPUs, and neuromorphic computing systems. Many ESD tools are developed by the research community and leverage diverse hardware architectures for performance and scalability. Currently, the ESD covers ~70 top-level scientific tools and 800+ dependencies, and it is deployed as a “monolith”. This creates two major challenges:
Flexibility: Researchers might want to combine the ESD with other, possibly experimental, tools that are not yet in the official release. Conversely, smaller minimum-size software ecosystems can be beneficial for other applications, such as for GDPR-sensitive environments.
Service compatibility: The ESD contains “client” software that interacts with EBRAINS “web” services. The service-side environment must be strictly compatible with the client-side version, which can’t be guaranteed unless services are also part of the ESD.
This project focuses on adding modularity to the ESD’s spack buildtool-based build and deployment functionality to support more flexible, on-demand environments. You will develop spack-based tooling to generate custom environments, both subsets and supersets, that maintain consistency and alignment with the official ESD releases.
Another part of this project involves extending the ESD to support “services”. While most of the ESD comprises end-user tools, services often depend on a smaller or different set of dependencies. By leveraging the spack package manager used by the ESD, we can explicitly track distinct service-side dependencies and define compatibility constraints between client and server versions, influencing both the ESD client-side and the service-side build configurations.
Key Objectives:
Add support for creating ESD-derived software ecosystems:
Develop tools to extract partially overlapping subsets of the ESD while ensuring the “concretized” dependency tree remains compatible with the full official deployment.
Develop and integrate a format for ESD service packages based on the spack package format; i.e. recipes for backend services that define strict compatibility constraints with their client-side counterparts in the ESD.
Showcase a service build (e.g., for a Knowledge Graph or Neuromorphic access service) that adheres to specific client-server version range constraints and can be deployed as a lightweight, ESD-aligned container.
I’m Divyanshu, a 3rd-year CS student. I’m really interested in GSoC Project #38 and have been digging into the ebrains-spack-builds repository to understand the current architecture before writing my proposal.
While exploring the codebase, a few technical questions came up that I’d love to get your thoughts on to make sure I’m heading in the right direction:
I noticed the environment uses concretizer: unify: true, but there isn’t a committed spack.lock file. For the subset generator, do you envision us generating and committing a lockfile to act as the official baseline, or should subset validation just work against the abstract specs in spack.yaml?
The current operations scripts (like cleanup_script.py) interface directly with Spack’s internal Python API via spack python. For the new subset/superset tooling, should I stick to this pattern, or would this be a good time to build a standalone CLI using argparse?
I saw that py-ebrains-kg-core already handles some client-server version constraints using when= conditions on py-pydantic. Since that structure is already partially there, is the KG Core service your main target for the containerized showcase, or do you have another service in mind?
Are there any existing service-side package definitions floating around in the broader ESD infrastructure that I could look at as a reference for the new service package format?
I’m currently putting together a rough outline of my technical approach. Would any of you be open to taking a quick look at it before I finalize the proposal?
Thanks for your patience and I am sorry for the delay in getting back to you.
1. concretizer: unify: true and the absence of a spack.lock file:
You’ve hit on a key architectural point. We don’t strictly pin the environment because the EBRAINS Software Distribution (ESD) is deployed across diverse targets, including various HPC sites. Since these sites provide system libraries as “external packages,” our strategy for the ESD is to:
Provide a curated (pinned versioned and tested) set of top-level tools that remain consistent across all environments.
Ensure each top-level package is verified on every deployment target.
Perform inter-tool testing via dedicated “workflow” packages.
Utilize “benchmark” packages for performance verification, which is particularly critical in HPC contexts.
→ Together, this allows us to have the “same” user experience on all deployment targets.
2. Regarding the tooling pattern (Spack API vs. standalone CLI):
We have been developing a Python-based tool called “dedal” for managing Spack operations. There is an active merge request that addresses some of this, though it hasn’t been merged yet. I’ll provide more specific details on this tomorrow (once I’m back at my main workstation).
3. Regarding services (KG Core vs. other services):
While KG Core is a candidate, we’ve also discussed NEST Desktop and the BrainScaleS experiment service. I previously experimented with this (BrainScaleS) in an older MR (“quiggeldy”), but it’s still a work in progress. I’ll dig up those references for you when I’m back at the right machine.
Currently, there isn’t a standardized reference for the new service package format, but I will bring this up with the ESD working groups to see if anything is surfacing.
The Python-based tool for building (replacing the shellscript-based installer) is located here: dedal and merge request 671 will replace the shell script.
If you could take out some time and review it , that would really help me understand the project scope better and make my approach correct towars the architecture choices
)