DEV Community: Frost Ming

A Review: Pipenv vs. Poetry vs. PDM

Frost Ming — Fri, 26 Mar 2021 13:05:54 +0000

Abstract

It is 2021 and we are all using or heard of package managers in Python, among which are Pipenv and Poetry. I also built a new package manager PDM to solve similar problems. There exist some comparisons between them around the community, but this article is not going to talk about the user interface or their versatility, it is going to focus on two important aspects: performance and correctness.

Setup

Pipenv: HEAD@275f7e151eb0aa17702215165a371df7da9ad476

Poetry: 1.1.5

PDM: HEAD@d72ba5d2ee0b0305f917d8739667ba78465c5cc8

Python version: 3.9.1

Performance

Dependency set(in poetry format):

[tool.poetry.dependencies]
python = "^3.9"
requests = { git = "https://github.com/psf/requests.git", tag = "v2.25.0" }
numpy = "^1.19"

[tool.poetry.dev-dependencies]
pytest = "^5.2"

This includes a Git dependency and a heavy binary dependency. In the following result, the cost times are measured in seconds.

Unless explicitly given, the measurements are done against the install command.

Result

	Pipenv	Poetry	PDM
Clean cache, no lockfile	98	150	58
With cache, no lockfile	117	66	28
Clean cache, reuse lockfile*	128	145	35
With cache, reuse lockfile**	145	50	16

*: Test with command poetry add click pipenv install --keep-outdated click pdm add click respectively.

**: Commands are the same as above.

Performance Review

Pipenv has a problematic cache system, which slows down the performance with the existence of caches
Poetry and PDM both benefit a lot from the caches, PDM takes even less time.
Pipenv uses a very different mechanism to reuse the lock file — it runs full locking first then modifies the content of the old lock file, while PDM can reuse the pinned versions in the lock file. Poetry improves a little with the lock file existing.

Correctness

The goal of these 3 package managers is to produce a reproducible environment and they all try their best to make it work on cross-platforms and python versions. Let's see how it turns out.

Python Compatibility

The project file is designed as following(Poetry):

[tool.poetry.dependencies]
python = "^3.6||^2.7"

And PDM:

[project]
requires-python = ">=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*"

They both reference to the same range of Python versions to support. Although Pipenv doesn't support a Python version range constraint, I will also post its results here as a reference.

Now let's add one dependency pytest from a clean project. This dependency is chosen purposely because it has a different version set to support Python 2 and Python 3.

Result of Poetry:

$ poetry add pytest
...
Using version ^6.2.2 for pytest

Updating dependencies
Resolving dependencies...

  SolverProblemError

  The current project's Python requirement (>=2.7,<3.0 || >=3.6,<4.0) is not compatible with some of the required packages Python requirement:
    - pytest requires Python >=3.6, so it will not be satisfied for Python >=2.7,<3.0

  Because no versions of pytest match >6.2.2,<7.0.0
   and pytest (6.2.2) requires Python >=3.6, pytest is forbidden.
  So, because foo depends on pytest (^6.2.2), version solving failed.

Lock failed as 6.2.2 was picked but it was not compatible with python 2.7. Anyhow Poetry shows a well human-readable error message telling people what happened and how to solve it.

Result of Pipenv

Depending on Python 3 or Python 2 is used to create the virtualenv(with --three/--two option), pytest is locked to 6.2.2 and 4.6.11 respectively. The lock file surely can't work on both Python 2 and Python 3 environment at the same time.

Result of PDM

$ pdm add pytest
Adding packages to default dependencies: pytest
✔ 🔒 Lock successful
...

  ✔ Install pytest 4.6.11 successful

...
🎉 All complete!

pytest is correctly resolved to 4.6.11 that supports all versions within the requires-python contraint.

Version Tree Searching

A dependency resolver should be able to search for other candidates when the current one causes version conflicts.

Let's take the example from Poetry's README:

Result of Pipenv

$ pipenv install oslo.utils==1.4.0
...
There are incompatible versions in the resolved dependencies:
  pbr!=0.7,<1.0,>=0.6 (from oslo.utils==1.4.0->-r C:\Users\FROSTM~1\AppData\Local\Temp\pipenvkbeeio2trequirements\pipenv-0zsj0laj-constraints.txt (line 3))
  pbr!=2.1.0,>=2.0.0 (from oslo.i18n==5.0.1->oslo.utils==1.4.0->-r C:\Users\FROSTM~1\AppData\Local\Temp\pipenvkbeeio2trequirements\pipenv-0zsj0laj-constraints.txt (line 3))

Unable to resolve* since Pipenv failed to search for lower versions of oslo.i18n to find one that is compatible with pbr<1.0

*: Be aware that Pipenv's strategy is "lock after install", so the incompatible package will be installed into the environment before the lock failure is reported.

Result of Poetry

As illustrated in the README, poetry successfully resolves with oslo.i18n==2.1.0 . It searches along the candidate list of oslo.i18n and discard those that bring conflicts.

Result of PDM

$ pdm add oslo.utils==1.4.0
...
  ✔ Install oslo.i18n 2.1.0 successful
...

Similarly, PDM also locks successfully with the same version of oslo.i18n

Environment Marker Propagation

To make a cross-platform project template, we often need to define some platform-specific dependencies and those also may have their subdependencies. These platform-specific dependencies may not be able to build successfully on the source system. We don't want these dependencies and subdependencies to be installed on the systems that do not match the requirements.

Let's start by adding a dependency gevent; os_name == "posix", which has several subdependencies. Commands are run from a Windows computer.

Result of Pipenv(Pipfile.lock)

{
    "_meta": {
        "hash": {
            "sha256": "9ce84144d1fc47c173581ae74c51ffbe28ac242b1fe0e1642915e803f15d3063"
        },
        "pipfile-spec": 6,
        "requires": {},
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "gevent": {
            "hashes": [
                ...
            ],
            "markers": "os_name == 'posix'",
            "version": "==21.1.2"
        }
    },
    "develop": {}
}

Only gevent is resolved with the marker in the lock file and Pipenv stops trying to find its children dependencies when the marker doesn't match the current system. This means if you use this Pipfile.lock to deploy on the target Linux server, some significant dependencies WILL NOT be installed!

Result of Poetry

gevent together with greenlet, cffi, pycparser, zope.event, zope.interface are pinned in the lock file and don't get installed to the environment, which is the expected behavior. But I didn't figure out how to add a requirement with markers from the CLI and I have to manually write it in pyproject.toml. Further, if I run poetry add pycparser after that, pycparser can be installed correctly.

Result of PDM

The same result as Poetry, except that in pdm.lock, children dependencies also have the marker os_name == 'nt' so that installers won't have to search the dependency tree to see whether a single package should be installed.

Conclusion

On the performance perspective, Pipenv doesn't play well due to its design choice that it integrates with other third-party tools and libraries instead of building its own. Pipenv can only wrap, combine, and do a little improvement on those upstream libraries. Moreover, Pipenv doesn't meet the goal of reproducible environment as well. It can produce a determinsitic installation setup on the source system but it is not a good idea to deploy to a different system without a careful check.

On contrast, Poetry and PDM are both doing great on performance and correctness, PDM is even better especially on the time cost and compatible dependency resolving. If you do not know this tool yet, start now.

You don't really need a virtualenv

Frost Ming — Fri, 22 Jan 2021 03:03:11 +0000

When you develop a Python project, you need to install the project's dependencies. For a long time, tutorials and articles have told you to use a virtual environment to isolate the project's dependencies. This way you don't contaminate the working set of other projects, or the global interpreter, to avoid possible version conflicts. We usually have to do these things:

$ python3 -m venv venv  # make a virtualenv named `venv`
$ . venv/bin/activate   # activate the virtualenv
(venv) $ pip install -r requirements.txt  # install the dependencies

There are also workflow tools that simplify this process, such as Pipenv and Poetry. They create virtual environments for you without perception and then install dependencies into them. They are used by a wide range of users. A virtual environment contains a Python interpreter and has the same directory structure as a normal installation so that it can be used as if it were a standalone Python installation.

The problems with virtual environments

Virtualenvs help us isolate project dependencies, but things get tricky when it comes to nested venvs: One installs the virtualenv manager(like Pipenv or Poetry) using a venv encapsulated Python, and creates more venvs using the tool which is based on an encapsulated Python. One day a minor release of Python is out and one has to check all those venvs and upgrade them if required before they can safely delete the out-dated Python version.

Another scenario is global tools. There are many tools that are not tied to any specific virtualenv and are supposed to work with each of them. Examples are profiling tools and third-party REPLs. We also wish them to be installed in their own isolated environments. It's impossible to make them work with virtualenv, even if you have activated the virtualenv of the target project you want to work on because the tool is lying in its own virtualenv and it can only see the libraries installed in it. So we have to install the tool for each project.

I've been maintaining the Pipenv project as a collaborator for the past two years and became a member of PyPA in early 2020. I am always thinking if the virtual environment is really a must-to-have for Python projects, just like npm, it doesn't need a cloned node binary, but just a node_modules directory that is unique for each project.

PEP 582 -- Python local packages directory

The solution has been existing for a long time. PEP 582 was originated in 2018 and is still a draft proposal till the time I wrote this article, but I found out it is exactly what node_modules are in Python.

Say you have a project with the following structure:

.
├── __pypackages__
│   └── 3.8
│       └── lib
└── my_script.py

As specified in the PEP 582, if you run python3.8 /path/to/my_script.py, __pypackages__/3.8/lib will be added to sys.path, and the libraries inside will become import-able in my_script.py.

Now let's review the two problems I mentioned in the last section and see how they change with the power of PEP 582. For the first problem, the main cause is that the virtual environment is bound to a cloned Python interpreter on which the subsequent library searching based. It takes advantage of Python's existing mechanisms without any other complex changes but makes the entire virtual environment to become unavailable when the Python interpreter is stale. With the local packages directory, you don't have a Python interpreter any more, the library path is directly appended to sys.path, so you can freely move and copy it.

For the second, once again, you just call the tool against the project you want to analyze, and the __pypackages__ sitting inside the project will be loaded automatically. This way you only need to keep one copy of the global tool and make it work with multiple projects.

PDM -- A new Python package manager and workflow tool

Starting from the PEP, I made PDM, a new Python package manager and workflow tool that leverages PEP 582 to get rid of virtualenv entirely. It installs dependencies into the local package directory __package__ and makes Python interpreters aware of it with a very simple setup. It is not only an implementation of PEP 582 but also the only package manager that supports PEP 621, a new metadata format based on pyproject.toml which becomes the standard recently. It is foreseen that pip will also gradually support this format. Besides, PDM uses the same dependency resolver as pip and has a full-featured plugin system, allowing for community-contributed plugins to enhance the functionalities.

In PDM, PEP 582 is not mandatory, you can also stick with virtualenv. PDM can detect existing venvs but not create new ones.

Another thing that is noteworthy is its dependency resolution mechanism -- it tries to lock versions that are compatible with the requires-python value of the project. Say your project requires Python 2.7 or 3.6 upper and you want to add pytest as a development dependency, in Pipenv(ver. 2020.11.15) you have to pin pytest = "<5" manually in Pipfile. And in Poetry(ver. 1.1.4) if you run poetry add -D pytest you will get:

The current project's Python requirement (>=2.7,<3.0 || >=3.6,<4.0) is not compatible with some of the required packages Python requirement:
    - pytest requires Python >=3.6, so it will not be satisfied for Python >=2.7,<3.0

Yes, it tells you to upgrade your Python requires version. However, in PDM, you can lock successfully:

❯ pdm add -d pytest
Adding packages to dev dependencies: pytest
✔ 🔒 Lock successful
Changes are written to pdm.lock.
Changes are written to pyproject.toml.
Synchronizing working set with lock file: 11 to add, 0 to update, 0 to remove

  ✔ Install atomicwrites 1.4.0 successful
  ✔ Install colorama 0.4.4 successful
  ✔ Install packaging 20.8 successful
  ✔ Install more-itertools 5.0.0 successful
  ✔ Install pyparsing 2.4.7 successful
  ✔ Install attrs 20.3.0 successful
  ✔ Install pluggy 0.13.1 successful
  ✔ Install py 1.10.0 successful
  ✔ Install pytest 4.6.11 successful
  ✔ Install six 1.15.0 successful
  ✔ Install wcwidth 0.2.5 successful

🎉 All complete!

As you can see, pytest is pinned to 4.* that is compatible with Python 2.7 and Python 3.6+.

For more features and demos, please refer to the PDM's website and GitHub repo.

Some random words

You may have seen this comic before, there are already so many package managers in Python's world, do we need a new one? No, I think. Gladly we have seen many improvements in the Python packaging ecosystem and the official installer pip, such as PEP 517/PEP 518, and the new dependency resolver, more to come in the future. But before the day comes, why not try something different from the traditional, why not make something new that makes at least myself happy. If you think so, PDM should be an option, hopefully.