[somewhat of a broad question to get a sense of where the community stands]
After using it for years, I just deleted anaconda and am trying to start fresh, and decide on a package manager. My needs aren’t too complicated and want to keep my environment as simple as possible (one environment ideally) to avoid the slow crawls with conflict resolution in conda etc, and hence would appreciate sharing experiences with different package managers such as pyenv, pipenv, poetry, pip, conda, homebrew etc.
Template for your comment:
pkg/env manager you use: <>
complexity of your needs:
how long have you been using:
pros and cons:
any new development I could adopt (as I have a “fresh” start now):
I mostly use macOS (all development and initial testing), and Linux (testing and data processing etc). I do not care much for Windows (i don’t support for most of libraries/tools).
Pkg/env manager: Conda Complexity of my needs: Fairly simple, can usually keep everything to one environment per project/analysis How long I have used: ~5 years Pros /new development
Installing Mamba makes Anaconda much more bearable; it is written in C so it goes much faster. Cons: things may get a bit hairy when a dependency is only available via Pip. As long as I have Pip installed in each conda environment, I can avoid cross-environemnt conflicts though.
I am not sure how one could avoid conflicts with just one environment. I would just install a base environment in miniconda with nothing but mamba, and then have everything else in separate environments.
Hey, Pradeep. My usage isn’t entirely stable, but I am using Python environments daily, and here are my current thoughts:
Environment manager/Python provider: miniconda (thinking of looking into micromamba)
Installer: pip/pipx
Project manager: hatch
Complexity of needs: I spend a lot of my day developing or running Python applications. I use DataLad/git-annex a lot. I don’t really compile C extensions.
How long: Been using Python pretty much daily for ~12 years. Been doing what I’m currently doing for a few months.
Pros/cons: Not sure how to answer this. I guess a “pro” is that once you get used to treating environments (and through conda, Python installations/versions) as disposable, most problems with broken environments go away with the environment. A “con” is that I still haven’t found a satisfying way to pin dependencies in a way that will make it easy to set up a Docker image, conda environment, or virtual environment from a single source after 3-4 years without something breaking and needing maintenance. Working on it.
New developments: I like hatch as a replacement for setuptools, and hatch-vcs as a replacement for versioneer. It also does a bunch of what tox does.
pkg/env manager you use: mamba from mambaforge (& pip, for packages that only exist on pypi or github). Whenever I start a new project, I spin up a new environment complexity of your needs: Simple. I’m mostly doing one-off analyses/reports (e.g., publications) – not making frameworks, compiling things, or making complex packages how long have you been using (this setup): 2 years pros and cons:
Some common neuroimaging tools are not available without pip
Building packages is tricky (I end up using pip and am looking forward to trying hatch more)
I find the lock file system finicky
any new development I could adopt:
This is only sort of related, but I liked this forward-looking discussion on python packaging in general: Python Packaging Strategy Discussion - Talk Python Live Stream - YouTube. Topics include the distinction between packages vs apps, whether package managers should bootstrap installation of python, and goals for hatch.
that’s very interesting Chris - can you elaborate more? how many environments do you have? what’s different among them? what makes you feel you need a new environment? I know specific differing version dependencies for different libraries are one reason, but I am curious to hear your thoughts.
One nice thing about this approach is that once you get used to it, it’s trivial to get set up on a new system. Install miniconda, create a default environment, update your .bashrc, and go. Everything is disposable, so starting up is the same as starting over, and both are cheap.
For the conda environments, base is the default environment for managing conda. I leave it alone and only update it to upgrade conda. default is what I work in on a regular basis, and has generally useful utilities like ipython, pandas, nibabel, and pipx. All of the other ones are for working on specific projects or debugging an issue in a specific version of Python. They might be ones I use regularly or for half an hour, and all of them can be deleted without any anguish.
Note that the only things I install through conda are Python itself and git-annex. Everything else is installed through pip.
For the pipx envs, these are one-off environments for tools that I want generally available but don’t need to import, don’t want to worry about mutual incompatibilities in their dependency trees.
Generally if there’s a task to complete that can’t be done with my current default environment and my collection of pipx-installed utilities, I’ll use a new virtual environment.
thanks - that stackoverflow reply is wrong then! I use pip install -e . also, and if that works with pyproject.toml, and I would move my libraries to TOML going forward.
thanks for elaborating - let me ask a slightly different question:if I didn’t have many version conflicts, can I just make do with pip alone without hatch, poetry or miniconda
trying to read docs on hatch and adopt it, and their docs pages are not geared towards “newbies”
Pip is not everything. It’s a package installer, and just enough of an environment manager to let you know that your environment is broken. I think you need virtual environments, and I use miniconda because it combines environment management and installing any Python you want into it, and is easy to install in the home directory of a computer that I have no privileges on. If you only need virtual environments pip install virtualenv and you’ve got them.
The alternative to virtual environments is a cluttered base environment that eventually needs to be reinstalled. If you’re eventually going to throw it out anyway, might as well make throwing out and recovering part of your common practice.
You can absolutely do without hatch or poetry if you aren’t using them as development tools. And you can configure use hatchling as a build backend without using hatch as a frontend. I did for several months and am only just starting to explore hatch’s other features.
regarding the suggestion of a move to /src layout, I am kinda of afraid of the pain it might induce, as resolving MRO issues with a lot of classes/OOP in python was not fun, when maintaining my various libraries
@effigies , I am able to learn and work with pyproject.toml and hatchling, and I am moving most of libraries to that setup. hatch-vcs works for me for publishing etc, but I’ve trouble using it to set __version__ inside the library when they need to query it run time (looks like that is due to a bug in hatch yet to be fixed).
I see pybids still relies on versioneerhere, so I am wondering if you were able to successfully use hatch-vcs for __version__ elsewhere?
Thanks for opening this topic which triggered many interesting comments.
Here’s my contribution:
pkg/env manager you use: pip and conda
complexity of your needs: quite modest. I work a lot with jupyter to develop the right pipeline to analyze data from various studies in the lab. Something I pack them up in a python script (à la BIDS) to ease looping on the whole dataset (for this I use PyCharm, by the way).
how long have you been using: about 5 years
pros and cons: most has already been said.
In practice I have one conda env per study/projet.
for the benefit of others reading, hatch-vcs docs seems to suggest version-file must be under [tool.hatch.version] in pyproject.toml but it doesn’t work. it needs to be under [tool.hatch.build.hooks.vcs], so it gets generated in build, and distributed in publish
hi Chris, how do you incorporate _version.py in github pages docs? as we remove it from the repo (although we include it for distribution), I can’t depend on it in the gh-pages branch for version info in the editable install locally (which is where I generate the docs to be uploaded)
