Are you confused by the various names that float around the Python packaging ecosystem? Have you ever asked a colleague to help you with an installation issue, only for them to reply "use {other_tool} instead" and make the problem worse? Have you seen the infamous XKCD comic on Python environments but you're still wondering how to solve your mess?
This short blog post is not a guide that will help you troubleshoot everything, but instead a list of resources that I consider up to date, modern, informative, and free of "hot takes" or unnecessary hate towards maintainers.
1. "The Basics of Python Packaging in Early 2023"
https://drivendata.co/blog/python-packaging-2023
This blog post by Jay Qi, Lead Data Scientist at DrivenData, is an informative take on how to produce your own Python package, specifically writing your pyproject.toml (modern replacement of setup.py). It neatly explains all the concepts involved, including PEP 517 build backends, PEP 621 project metadata, and some extra stuff.
2. "An unbiased evaluation of environment management and packaging tools"
Anna-Lena Popkes, Senior Machine Learning Engineer at inovex, delivered this talk at PyConDE and EuroPython. She offers a neat categorization of the different aspects or facets of Python packaging:
The chart does not include all tools, but it's an excellent starting point.
One thing I'd change is recommending rtx over pyenv, which works in a very similar way but it's written in Rust (so it's super fast) and avoids the typical problems with shims.
3. The pyOpenSci Python packaging guide
https://www.pyopensci.org/python-package-guide/package-structure-code/intro.html
This long guide was written by Leah Wasser, Executive Director at pyOpenSci, with the help of many contributors (including myself). It goes into more depth on how to choose a specific build backend or workflow tool when developing and creating packages. Whether you're creating a complex Python package with compiled extensions or a plain, pure Python one, this guide will help you navigate the ecosystem quite effectively.
4. "Why not tell people to 'simply' use pyenv, poetry or anaconda"
https://www.bitecode.dev/p/why-not-tell-people-to-simply-use
Okay, this one is a bit more loaded and opinionated than the others. But I believe this blog post in the Bite code! is very necessary - it's often too tempting to tell someone to "just" install yet another tool that will fix their problems, but this creates a massive amount of collective pain, and Python packaging is particularly affected. Please refrain from doing that!
(Which reminds me of this fantastic piece by Ned Batchelder, "How to be helpful online" https://nedbatchelder.com/blog/202009/how_to_be_helpful_online.html ❤️)
More often than not, installing yet another tool won't fix the user original problem. So, instead, try to spend some time helping them debug their problem.
Would you like me to write a guide on how to debug Python installation issues? Leave a comment saying "yes" or, much better, a situation that has affected you recently, or even right now.
5. "Thoughts on Python packaging"
https://pradyunsg.me/blog/2023/01/21/thoughts-on-python-packaging/
If you've made it this far is because you're another Python packaging nerd, like myself. Congratulations! (Or... sorry?)
To end with a positive, I'd highly recommend you to check out this blog post by Pradyun Gedam, maintainer of pip and many other packaging projects, which offers a meditated perspective on where we stand now, how we got here, and where we go next.
Hope you liked this list of resources! If you'd like to read more about this, comment, recommend, follow, and spread the love.
Also, remember to always thank your open source maintainers, they will appreciate it 💖
Top comments (0)