This article started as a random thought and then this post on Mastodon.
If you haven't read Hypermodern Python from 2020, it is a good companion to this article and apparently will be a released as a book in 2024.
Introduction
It feels like Python packaging has changed pretty substantially over the past few years. When I started creating Python packages the preferred approach was with setup.py and (what felt like) a myriad of other random files that were needed like MANIFEST.in, setup.cfg, etc. I basically found a guide once and then copied and pasted files around once I seemed to get something working.
I was an early adopter of pipenv with the primary appeal being lock files which promised consistent deployments. My early excitement got me burned multiple times and I looked around some more.
I remember Poetry being the cool, new upstart at the time. It completely eschewed setup.py, requirements.txt, and other disparate files -- it had one pyproject.toml file to configure a project, developer-friendly tooling to add dependencies, a lock file for consistent deployments, and an algorithm to make sure that a tree of dependencies was all compatible with each other (which was not possible with that version of pip at that time).
I was in love. There were definite rough patches and there were a few bugs that caught me here or there, but overall Poetry has been my go-to for a while.
However, in the past few years, Python packaging seems to be going through a new era of innovation. pip released a dependency resolver in 2020 that now complains when dependencies conflict. Also in 2020, PEP 621 and PEP 631 standardized pyproject.toml as the new normal for Python packages instead of setup.py. PEP 517 and PEP 660 created standards for Python build systems.
Based on those PEPs, Poetry isn't the only forward-thinking package manager anymore. pipenv is still around (and I presume more stable at this point!), but Hatch, PDM, and Rye all promise a pleasant developer experience when creating Python packages.
Benefits
Most of these package managers are of the "all in one" mentality and provide similar benefits: adding dependencies from the command-line, lock files, building, and publishing packages. If you want to do this manually Packaging Python Projects lays out the steps necessary. Reducing the number of tools required to create new Python packages means there is only one place to look for documentation and keep up to date.
Poetry
My first modern Python package manager. I would say the documentation is still the best available, although it might be too "designed" for some. It does a lot -- maybe too much? -- but, in my opinion, it pioneered a lot of features that are now expected in other Python package managers. I get the feeling it sometimes pushes forward without waiting for official PEPs, although I think that has changed over the past few years.
Occasionally lock files are not backwards-compatible which can be challenging when working in a team -- everyone needs to update their Poetry install at the same time.
Another annoying feature lacking in Poetry that is available in every other option is the ability to define "scripts". For example, poetry run dev as an alias for poetry run manage.py runserver 0:8000. That is not available in standard Poetry, although I use poethepoet (a Poetry plugin) to provide that functionality.
Hatch
Hatch never deviates from the Python PEP standards and brings a few innovative features to the table, including being able to group dependencies and scripts into custom environments. The hatch new [project] command is surprisingly opinionated and has settings for ruff (my new favorite Python tool), pytest, and coverage. I'm here for all of the tool choices, but I would be lying if I did not say it was a little surprising.
Under the PyPA GitHub organization which probably gives it more of a sheen of officialdom than other tools.
PDM
PDM for some reason feels like the sleeper option compared to the rest. I have not used it too much other than to set up a new project and play around with it a little bit. Dan Sloan chimed in on Mastodon pointed out that PDM follows all of the current PEP standards, has good support for cross-platform lock files, and also includes a useful pdm export command to output a standard requirements.txt file.
Rye
rye seems to have had a quick and meteoric rise. It was developed by the creator of Flask, Armin Ronacher, based on his opinionated approach to building Python packages. It reminds me of black a little bit in which a respected developer made an opinionated tool and it caught fire in the community. Rye also provides an overview of its philosophy here.
Rye has multiple cautions in the docs that it is experimental and not production-ready, although I did not see any glaring issues with my simple testing. Rye is a little different than the other options because it mostly consolidates a set of other tools (pip, pip-tools, etc.) into a particular workflow. Another distinguishing factor is that it handles multiple installing Python versions directly -- Poetry recommends using pyenv to deal with the same use case.
GitHub Stats
I do not believe GitHub stats are especially useful, but the number of stars do give some indication of the community engagement with a particular tool (current as of 2023-11-12).
Conclusions and takeaways
Personally I have multiple libraries using setup.py and Poetry, one using Hatch, and now one using Rye. The next one might use PDM. 😎 I like trying new tools to see how they work and to improve my workflow.
One could look at all of the options for Python packaging and be frustrated by the paradox of choice. I understand that viewpoint, but I also appreciate that all of these tools are making different trade-offs and pushing the state of the art forward in their own ways. Maybe at some point in the future there will be one "approved" Python package manager ala NPM or Cargo, but I am doubtful. There are pros and cons with multiple options, but I tend to think it's useful overall.
I personally have not used each package manager extensively. But, I have started a new project from scratch with each, did some simple tasks, and read through all of the documentation. Each project is relatively mature, so you can just choose any of the options and you would be fine. Standardizing on pyproject.toml means that the conversion from one package manager to another is usually just updating some TOML. So, choose the package manager that speaks to you (or try them all!) and improve your developer experience henceforth. 🛠️
Photo by Claudio Schwarz on Unsplash
Top comments (12)
Hey, nice post, thanks!
It would be nice to hear more criticism on poetry from you, really! Please describe what you mean by 'too designed'?
Regarding built-in scripts, I agree with you here. Yet, all the automations are easy to describe with good old makefiles. :D
My biggest criticism of
poetryis that it isn't completely PEP-compliant. For some people that's a big deal. It hasn't stopped me from using it in the past, though.Dealing with
Poetryandpyenvcan be annoying (hence whyrye's approach is appealing to me). I've run into a few problems that could only be solved by nuking the venv folder and re-installing everything from scratch which is not ideal.I have run into a few gnarly bugs over the years (one was github.com/python-poetry/poetry/is... which has now been fixed), but that doesn't really feel like a fair comparison -- I've used Poetry consistently for 3+ years, whereas I haven't used any other package manager nearly that much.
Yeah, those are exactly the same pains I assumed you would be talking about! :D It's so true, especially when it comes to poetry+pyenv and nuking .venv
Probably I should try rye for my side-projects. Thanks a lot!
Nice article here, Adam. I compared all the above tools and came back here to give my understanding and opinion.
Have used poetry and because it was the first tool to take the initiative and push the development in this structured direction, it deserves all the praise. Having said that, I want
pdmto grow much more than others because of the PEP standards and all the built-in functionalities. Haven't found Rye and Hatch to be so much mature though but they are definitely good.If poetry doesn't follow the PEP standards people would slowly but eventually migrate to one of the others fo sure.
Poetry is actively working to support PEP 621, see github.com/python-poetry/roadmap/i... and the linked PR. Handling a large installed base of current users makes it tricky I'm sure.
PDMis next on my list to try out on a real library to get a more in-depth understanding of how it works.I have danced with conda/mamba when I started my journey in DS. Then switched to pyenv and poetry after having issues with pipenv as I advanced in DS and CS.
I have tried hatch on the early days and not heard of rye until now 🙃. I will give it a try.
Today, I use pyenv + poetry, but also pixi when I want packaged from conda and other none Python.
Thank you for sharing comparison
Thanks! I thought about investigating conda as well, but for some reason that feels like it solves different problems than these 4. I'd love to hear your thoughts about it, though.
I would not bother with conda/mamba. Pixi, written in Rust, is what I would explore where I to want Anaconda Continuum packages 📦.
conda is like a merge of pyenv + poetry|pipenv into a single not so smart tool. Not so smart in a way you cannot share Python cross projects that use the same Python version.
The positive about it is that packages from official channels are reviewed before publication and thus not likely to be malicious.
Here is a list comparison from yet another package manager
pyflow.See also An Unbiased Evaluation of Environment and Packaging Tools
Nice overview! I hadn’t heard of several of these before
Are Python's toolings just too bad that we need this many package managers?
Well... that's not my view. :)
Python is 30+ years old and has grown organically over time (mostly through volunteers) and the number of users who use the language is massive. Trying to solve everyone's use cases is nigh impossible. Newer languages (Rust, Go, Node.JS) can look at pain points in Python and design tooling that solves those problems. And even with that advantage Node.JS has a few package managers (NPM, Yarn, pnpm, bun, and maybe more I don't even know about?). It just seems to happen at some point.
I realize it can be confusing for end-users to have too many options, but tools stagnating and not continuing to push the ecosystem forward seems worse to me. The newer PEPs were a welcome breath of fresh air that has reinvigorated Python tooling and I'm here for it.
Like I said in my conclusion, all of these tools are pretty stable at this point -- just pick one and stick with it.