Python package managers - pros and cons

[somewhat of a broad question to get a sense of where the community stands]
After using it for years, I just deleted anaconda and am trying to start fresh, and decide on a package manager. My needs aren’t too complicated and want to keep my environment as simple as possible (one environment ideally) to avoid the slow crawls with conflict resolution in conda etc, and hence would appreciate sharing experiences with different package managers such as pyenv, pipenv, poetry, pip, conda, homebrew etc.

Template for your comment:
pkg/env manager you use: <>
complexity of your needs:
how long have you been using:
pros and cons:
any new development I could adopt (as I have a “fresh” start now):

I mostly use macOS (all development and initial testing), and Linux (testing and data processing etc). I do not care much for Windows (i don’t support for most of libraries/tools).

thanks for taking the time to share your tips.

cc @yarikoptic @Chris_Rorden @effigies @adelavega

Pkg/env manager: Conda
Complexity of my needs: Fairly simple, can usually keep everything to one environment per project/analysis
How long I have used: ~5 years
Pros /new development
Installing Mamba makes Anaconda much more bearable; it is written in C so it goes much faster.
Cons: things may get a bit hairy when a dependency is only available via Pip. As long as I have Pip installed in each conda environment, I can avoid cross-environemnt conflicts though.

I am not sure how one could avoid conflicts with just one environment. I would just install a base environment in miniconda with nothing but mamba, and then have everything else in separate environments.

1 Like

Hey, Pradeep. My usage isn’t entirely stable, but I am using Python environments daily, and here are my current thoughts:

Environment manager/Python provider: miniconda (thinking of looking into micromamba)
Installer: pip/pipx
Project manager: hatch

Complexity of needs: I spend a lot of my day developing or running Python applications. I use DataLad/git-annex a lot. I don’t really compile C extensions.

How long: Been using Python pretty much daily for ~12 years. Been doing what I’m currently doing for a few months.

Pros/cons: Not sure how to answer this. I guess a “pro” is that once you get used to treating environments (and through conda, Python installations/versions) as disposable, most problems with broken environments go away with the environment. A “con” is that I still haven’t found a satisfying way to pin dependencies in a way that will make it easy to set up a Docker image, conda environment, or virtual environment from a single source after 3-4 years without something breaking and needing maintenance. Working on it.

New developments: I like hatch as a replacement for setuptools, and hatch-vcs as a replacement for versioneer. It also does a bunch of what tox does.

pkg/env manager you use: mamba from mambaforge (& pip, for packages that only exist on pypi or github). Whenever I start a new project, I spin up a new environment
complexity of your needs: Simple. I’m mostly doing one-off analyses/reports (e.g., publications) – not making frameworks, compiling things, or making complex packages
how long have you been using (this setup): 2 years
pros and cons:

any new development I could adopt:

@raamana I am not a Python expert. I would defer to the wisdon of @alexisthual and @jeromedockes @emdupre

1 Like

thanks everyone - appreciate your comments.

that’s very interesting Chris - can you elaborate more? how many environments do you have? what’s different among them? what makes you feel you need a new environment? I know specific differing version dependencies for different libraries are one reason, but I am curious to hear your thoughts.

thanks for sharing - its been useful to hear them, and outline the landscape and the surrounding complexity.

1 Like

looks like I also need to start adopting PEP 621 / pyproject.toml

there is a library to convert existing to a pyproject.toml:

although I’d have to keep for editable installs, as I use that quite often:

On this laptop?
$ conda env list
# conda environments:
base                     /home/chris/miniconda3
bids-tools               /home/chris/miniconda3/envs/bids-tools
default               *  /home/chris/miniconda3/envs/default
fmriprep-dev             /home/chris/miniconda3/envs/fmriprep-dev
looseversion-test        /home/chris/miniconda3/envs/looseversion-test
mamba                    /home/chris/miniconda3/envs/mamba
nibabel-38               /home/chris/miniconda3/envs/nibabel-38
nibabel-bugfix           /home/chris/miniconda3/envs/nibabel-bugfix
nibabel-dev              /home/chris/miniconda3/envs/nibabel-dev
nilearn                  /home/chris/miniconda3/envs/nilearn
nipype-dev               /home/chris/miniconda3/envs/nipype-dev
py2                      /home/chris/miniconda3/envs/py2
pybids-dev               /home/chris/miniconda3/envs/pybids-dev
smriprep-doc             /home/chris/miniconda3/envs/smriprep-doc
spec-schematools         /home/chris/miniconda3/envs/spec-schematools
templateflow-docs        /home/chris/miniconda3/envs/templateflow-docs
templateflow-docs2       /home/chris/miniconda3/envs/templateflow-docs2

$ ls .local/pipx/venvs 
black  codespell  datalad      flake8  magic-wormhole  pre-commit  scriv     tickers
blue   darker     datalad-osf  hatch   mypy            ruff        sdcflows

One nice thing about this approach is that once you get used to it, it’s trivial to get set up on a new system. Install miniconda, create a default environment, update your .bashrc, and go. Everything is disposable, so starting up is the same as starting over, and both are cheap.

For the conda environments, base is the default environment for managing conda. I leave it alone and only update it to upgrade conda. default is what I work in on a regular basis, and has generally useful utilities like ipython, pandas, nibabel, and pipx. All of the other ones are for working on specific projects or debugging an issue in a specific version of Python. They might be ones I use regularly or for half an hour, and all of them can be deleted without any anguish.

Note that the only things I install through conda are Python itself and git-annex. Everything else is installed through pip.

For the pipx envs, these are one-off environments for tools that I want generally available but don’t need to import, don’t want to worry about mutual incompatibilities in their dependency trees.

Generally if there’s a task to complete that can’t be done with my current default environment and my collection of pipx-installed utilities, I’ll use a new virtual environment.

Nope. Editable installs are standardized now. Use pip install -e . instead of python develop and you’re good. I would recommend hatch: Chris Markiewicz - Contemporary Python Packaging (2023)

1 Like

thanks - that stackoverflow reply is wrong then! I use pip install -e . also, and if that works with pyproject.toml, and I would move my libraries to TOML going forward.

thanks for elaborating - let me ask a slightly different question:if I didn’t have many version conflicts, can I just make do with pip alone without hatch, poetry or miniconda

trying to read docs on hatch and adopt it, and their docs pages are not geared towards “newbies” :slight_smile:

Pip is not everything. It’s a package installer, and just enough of an environment manager to let you know that your environment is broken. I think you need virtual environments, and I use miniconda because it combines environment management and installing any Python you want into it, and is easy to install in the home directory of a computer that I have no privileges on. If you only need virtual environments pip install virtualenv and you’ve got them.

The alternative to virtual environments is a cluttered base environment that eventually needs to be reinstalled. If you’re eventually going to throw it out anyway, might as well make throwing out and recovering part of your common practice.

You can absolutely do without hatch or poetry if you aren’t using them as development tools. And you can configure use hatchling as a build backend without using hatch as a frontend. I did for several months and am only just starting to explore hatch’s other features.

very useful post - will share it for folks starting to spring clean like me :slight_smile:

btw, hatch new --init will convert an existing automatically, so no need for other libraries.

regarding the suggestion of a move to /src layout, I am kinda of afraid of the pain it might induce, as resolving MRO issues with a lot of classes/OOP in python was not fun, when maintaining my various libraries :slight_smile:

@effigies , I am able to learn and work with pyproject.toml and hatchling, and I am moving most of libraries to that setup. hatch-vcs works for me for publishing etc, but I’ve trouble using it to set __version__ inside the library when they need to query it run time (looks like that is due to a bug in hatch yet to be fixed).

I see pybids still relies on versioneer here, so I am wondering if you were able to successfully use hatch-vcs for __version__ elsewhere?

Thanks for opening this topic which triggered many interesting comments.

Here’s my contribution:

pkg/env manager you use: pip and conda
complexity of your needs: quite modest. I work a lot with jupyter to develop the right pipeline to analyze data from various studies in the lab. Something I pack them up in a python script (à la BIDS) to ease looping on the whole dataset (for this I use PyCharm, by the way).
how long have you been using: about 5 years
pros and cons: most has already been said.

In practice I have one conda env per study/projet.

for the benefit of others reading, hatch-vcs docs seems to suggest version-file must be under [tool.hatch.version] in pyproject.toml but it doesn’t work. it needs to be under [], so it gets generated in build, and distributed in publish

hi Chris, how do you incorporate in github pages docs? as we remove it from the repo (although we include it for distribution), I can’t depend on it in the gh-pages branch for version info in the editable install locally (which is where I generate the docs to be uploaded)