Package management in python is evolving continuously. This evolution is strewn with initiatives of all kinds, each with their pitfalls. It's frustrating to see all these ways of doing things coexist, to have to question what we know about the subject every year. At the same time, it is the mark of an active community that is looking for new ways of doing things, an environment that evolves with the times. It reminds us of this long transition that took place over more than 10 years to go from python 2 to python 3. It was a hard choice but come with great benefits.
In 2019, I chose to use the pipenv + setuptools combination as a package manager. This combination has had served its time. I decided at the end of 2022 to migrate all the projects I work on to poetry.
I still have doubts while writing this article. The purpose of this article is to share my knowledge and my findings at the time of writing. I can be wrong or even make false assumptions. Share in comments if you have a different vision, and share in comments in you know interesting PEP (python enhancement proposal) on this topic not mentioned in this article.
What am I expecting from a package manager at the end of 2022?
A python project without a manifest is a car without air conditioning. After having tasted it, it is difficult to do without it at certain key moments.
A manifest contains the declaration of dependencies, the version number, the exposed python packages, the development dependencies and a whole bunch of other metadata... The manifest is mandatory to publish a library on pypi. It avoids a lot of headaches when the project grows, for example the hell of relative packages or dependencies that evolve and become incompatible with each other.
from .util import filter
from ..core import local
The only problem is that each package manager has its own manifest format. Here are the manifests that you may have come across and which are more or less exclusive: the requirement.txt, pyproject.toml, setup.cfg, setup.py and Pipfile
A package manager may transpile its manifest to other formats as needed. For example, I think that poetry transpiles the
pyproject.tomltosetup.pymanifest to build a wheel, the format expected by pypi to distribute a library, and publish it. It's transparent to us.
In 2022, a package manager is a tool that from a manifest will install dependencies, manage a virtual environment, lock dependencies to a given version, ... Things that at first are optional in python.
Here is a grid analysis that represents my expectations for a package manager in 2023 with a subjective evaluation of the benefits of each combination.
| Poetry | pipenv + setuptools | pipenv | setuptools | |
|---|---|---|---|---|
| standard manifest | 4 | 4 | 2 | 5 |
| manifest scaffholding | 5 | 1 | 3 | 0 |
| virtual environment (.venv layout) | 5 | 3 | 3 | 0 |
| dependencies locking | 5 | 3 | 3 | 0 |
| update locked dependencies | 5 | 2 | 2 | 0 |
| source code organization (src layout) | 4 | 4 | (?) | 4 |
| publication of library | 4 | 3 | 0 | 4 |
| version management | 3 | 4 | 3 | 4 |
| mono-repository | 1 | 1 | 0 | 0 |
projects that use setuptools : requests, django, flask, boto3...
projects that use poetry : ward, ... (if you have some with more than 500 stars, share them in comments)
projects that use pipenv : ... (if you have some with more than 500 stars, share them in comments)
setuptools is awesome, but it is doing only one things, but it is doing it almost perfectly. Unfortunately, it miss features I want to have in my project as managed virtual environment, locked dependencies, ... I won't talk about it in this article.
standard manifest
The pyproject.toml manifest has been official since PEP 621. It is still young. poetry uses this format with its own guidelines. setuptools is supported it for few months. The support is pretty good but there is some limitation. I tried this manifest two months ago and I fail to install a python application in editable mode with pyproject.toml manifest. I don't know if it's a limitation of pip or setuptools, but it's a problem for development.
pipenv uses the Pipfile format. He is the only one to use it. A Pipfile manifest can reference a setup.cfg or setup.py manifest. It is this feature that led me 2 years ago to choose pipenv + setuptools.
manifest scaffholding
I instantiate a manifest with every python project I start. With setuptools, I have to write this manifest by hand. The scaffholding brings together assistants who assist us in writing a manifesto. This is the case with poetry with the init command. pipenv also offers scaffholding but on the Pipfile manifest.
poetry init
The package manager can offer a more advanced scaffholding. poetry can declare new dependencies in the manifest with poetry add.
poetry add requests
poetry add --dev pytest
The scaffholding is essential for a newcomer who comes from nodejs or java. This is his first contact with the environment. This is the guarantee of creating a python project that can be used by other developers without reading 10 tutorials that contradict each other...
pyproject.toml manifest
[tool.poetry]
name = "myproject"
version = "1.0.0"
description = "this awesome project"
authors = ["Fabien Arcellier <>"]
readme = "README.md"
[tool.poetry.dependencies]
python = "^3.8"
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
virtual environment (.venv layout)
pipenv and poetry share the same DNA. The 2 tools bind a virtual environment to a python project. A virtual environment, independent of the python installed on the system, is automatically associated with the manifest and mounted as soon as the user issues a command in the package manager, for example poetry install.
On all the projects that I have initiated for 2 years, I use the .venv convention. the virtual environment is created at the root of the python project in .venv directory. poetry uses the default .venv layout. As soon as you create a project with poetry init, the associated virtual environment is automatically declared in the .venv folder next to pyproject.toml. This convention allows me to share run configurations in the IDE with other developers, whether on pycharm or vscode.
.
├── pyproject.toml
└── .venv
├── bin
├── .gitignore
├── lib
└── pyvenv.cfg
dependencies locking
locking dependencies ensures that all dependencies will be reinstalled with the same version regardless of which developer clones the project locally.
I happened to clone a repo several months after the last commit and noticed dependency problems because a 2nd or 3rd level library evolved and this evolution "broke" a level 1 library that uses it. It's a frustrating problem. I had the case this year with label-studio-converter one of the repo of the excellent label-studio
Dependency locking is provided by poetry and pipenv. poetry is faster and seems more robust.
pipenv fails to install dependencies on machine learning pocs that used pytorch or yolo.
poetry.lock
[[package]]
name = "altair"
version = "4.2.0"
description = "Altair: A declarative statistical visualization library for Python."
category = "main"
optional = false
python-versions = ">=3.7"
[package.dependencies]
entrypoints = "*"
jinja2 = "*"
jsonschema = ">=3.0"
numpy = "*"
pandas = ">=0.18"
toolz = "*"
Locking dependencies is a plus to guarantee the deployment on a PaaS hosting that deploys source code instead of docker image like heroku, .... It guarantees the installation in production of the same dependencies that we use locally in our developments.
update of one / all dependencies
Once the dependencies are locked, we are guaranteed to always install the same versions, even a year later. However, it may become required to update these dependencies. poetry does this with poetry update command. Note that this command only updates dependencies locked in the poetry.lock file and does not touch the pyproject.toml manifest.
poetry also allows you to update just one library rather than all libraries in the project at once, for example, just requests. It allows more flexibility in the update policy, especially on security alerts that can be raised by github for example. This is something I missed with pipenv+setuptools. I had to update all the locked dependencies at once.
organization of the source code with src layout
Relative imports are bad practice in python. This pattern feel wrong, you can import a module of one directory above but not more.
# to avoid
from .util import foo
# to do
from myproject.util import foo
The project manifest definitely solves this problem. The package manager installs our project in the virtual environment and allows us to use our modules like the other modules of the standard library from their full path.
For 2 years, I have the impression that the src layout takes precedence over other methods of organizing source code such as the flat layout. (see discussion src layout vs flat layout).
A project organized according to src layout implements the packages in a src folder. The layout looks like the following.
.
├── poetry.lock
├── pyproject.toml
├── README.md
├── src
│ ├── my_project
│ │ ├── __init.py__
│ │ └── util.py
└── tests
Package are declared in the manifest pyproject.toml with the directive packages.
pyproject.toml
[tool.poetry]
name = "myproject"
version = "1.0.0"
description = "this awesome project"
authors = ["Fabien Arcellier <>"]
readme = "README.md"
packages = [
{ include = "my_project", from = "src" }
]
[tool.poetry.dependencies]
python = "^3.8"
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
src layout seems to be supported by all package managers, including poetry. This layout is not linked to a PEP. I don't know yet if this trend will continue. It is similar to what is found in nodejs or java. I find that it facilitates the organization of projects in python, makes it easier to get started with the project. Today, I apply it in libraries, web projects like flask or streamlit but it's one of the trends that could fall. I kept the flat layout only with django. manager.py does not work easily with src layout.
sharing a library on pypi
I think that having the ability to share a native python library helps to democratize python and encourage sharing in the community. Everyone can share their work. The overall quality is not terrible, but this ocean of contribution pulls the ecosystem upwards and encourages its evolution.
poetry allows you to publish a library on pypi with the poetry publish command. For pipenv + setuptools , I had to install twine in the development dependencies and use it to orchestrate the release to pypi. Even if the quality is there, I had to find out how to organize my manifesto, how to use twine, how to industrialize it before being able to share my projects.
version management
The version management in poetry is poorer than with setuptools. You must write the version number in the manifest, not elsewhere. setuptools has more options to retrieve this version number, either from a file or from an attribute in a module. This evolution makes all the same sense because the manifest is the ideal place to record a version number.
I use the version number of a python package to automatically create a tag in git and trigger a pipeline in the CI/CD. Here is the snippet, compatible with poetry, that I use to retrieve the version number of a package and then create a corresponding git tag.
import fixtup
ROOT_DIR = os.path.realpath(os.path.join(__file__, "..", ".."))
VERSION = importlib.metadata.version(fixtup.__name__)
multiple python projects in a mono repository
In enterprise repos, I use a mono-repository architecture with multiple python projects in a single repo. Above, I only have partial answers. poetry does not offer a workflow for managing multiple python projects. I use either a homemade utility alfred-cli to manage the mono-repository, bash scripts, or simply a makefile. No solution fully meets my requirements.
At the moment, alfred-cli is an open source experimental project I have written but its documentation and design is too messy.
Moving to poetry
In 2022, with the standardization of the pyproject.toml manifest, poetry stands out as the most robust and easy-to-use package manager, even for a newcomer to the python universe. In all aspects, it is better than pipenv with a developer experience that I find more complete and enjoyable.
| Poetry | pipenv + setuptools | pipenv | setuptools | |
|---|---|---|---|---|
| standard manifest | 4 | 4 | 2 | 5 |
| manifest scaffholding | 5 | 1 | 3 | 0 |
| virtual environment (.venv layout) | 5 | 3 | 3 | 0 |
| dependencies locking | 5 | 3 | 3 | 0 |
| update locked dependencies | 5 | 2 | 2 | 0 |
| source code organization (src layout) | 4 | 4 | (?) | 4 |
| publication of library | 4 | 3 | 0 | 4 |
| version management | 3 | 4 | 3 | 4 |
| mono-repository | 1 | 1 | 0 | 0 |
poetry comes from the experience accumulated by decades of wandering around package management in python. I regret leaving setuptools. It is a robust and powerful utility which in its time also made a huge difference. Its development is active and it masterfully ensures compatibility between several generations of manifest.
I think this is the right direction to take for the next 1-2 years given the current ecosystem. I may have blind spots on this. Did you make the transition yourself ? Did you choose another package manager for your projects ? Share your experience in comments
- cover image by Eli Digital Creative from Pixabay
Top comments (8)
Hey, thanks for this! Do you have any advice on how to migrate project from
pipenvtopoetry?The migration is usually pretty smooth. On each project, I was able to organize it effortlessly. With team members, even members on windows, we were able to install
poetryeasily with pip.To migrate, I use this workflow:
pyproject.tomlfile generated bysetuptoolspyproject.tomlpyproject.tomlfrom the manifestsetup.cfgI have so far always regenerate the dependencies lock file. I didn't try to get the one from pipenv. If you have a way to do this, I would be interesting to know it :)
At the time of writing this article, I thought that the virtualenv was created by default in
.venv. This is not the case. I therefore systematically play this command to add this specification to the projects.Yo, didn't know about this command... THANKS A BUNCH!
Thanks for the great article!! As you already mention in comment, poetry doesn't create folder .venv. However after I tried to switch storing env in folder, I run this : poetry config virtualenvs.in-project true --local and run poetry install again. it doesn't create folder for me. I'm trying to figure it out.
I'm using Poetry (for the first time) with my Quickiebase project. I like Poetry so far, but would prefer it if the was only 1 package manager for Python: There should be one -- and preferably only one -- obvious way to do it.
It would be awesome. I dream of an official fusion between pip and setuptools with the features of poetry.
The dataclasses-json project has 1.2k starts and uses poetry.
Also, I think FastAPI uses poetry, but it also has requirements files there.
It's interesting. dataclasses-json is a good example.
FastAPI is using hatch. I don't know this tool chain. It is hosted by pypa. They move away one year ago from flit, another toolchain also hosted by pypa. I really don't know the benefits of this toolchain.
Fastapi, as a lot of library, does not have requirement to freeze dependency. It's something we need to ensure at the application level. For a library, it's something that raise issues. Overspecified dependency tree on a library make composition pretty hard with other libraries.