DEV Community: Matt Davis

Pipenv: Named Categories

Matt Davis — Sun, 09 Oct 2022 15:15:50 +0000

Today I am happy to announce expanded support for "named package categories" in the release of pipenv==2022.10.9 and here I will explain what this feature is about and some ways you might find it helpful. You may also wish to reference the package categories documentation.

If you are already familiar with pipenv then you are probably aware that until now the tool only supported two package groups: packages and dev-packages. These mapped to the default and develop groups in the lock file.

What if you wanted to have other groups defined? For example: sometimes prerequisites are required to build other packages that are part of your default package group and these have to be installed ahead of time.

Let's say the setup.py being built has import six but six is not installed, and so pipenv gets errors trying to build and install the dependent package. The guidance prior was to pipenv run pip install six ahead of running pipenv sync so that it would be available. Now though it should be possible to define an arbitrary group of prerequisites, for example:

# install prerequisite package six
# note: single category syntax
pipenv install six --catetgories prereq

# install the perquisite group followed by the packages group
# note: multiple categories syntax
pipenv sync --categories="prereq packages"

Support was added for across the range of pipenv commands for the categories argument.

# lock and uninstall examples
pipenv lock --categories="prereq dev-packages"
pipenv uninstall six --categories prereq

Note: The packages/default specifiers are used to constrain all other categories just as they have done for dev-packages/develop category. However this is the only way constraints are applied. The presence of other named groups do not constraint each other, which means it is possible to define conflicting package versions across groups. This may be desired in some use cases where users only are installing groups specific to their system platform.

Some organizations may wish to explore using named package categories to specify platform specific dependencies in groups such as linux, windows or macos. Using platform groupings may be a useful mechanism to work around some limitations of the pip resolver with regards to resolving certain platform specific packages targeting an alternate platform than the one being locked.

This change was also made possible by the concurrent efforts of Oz Tiram who first converted pipenv to the library plette for its Pipfile/Pipfile.lock management. Since we have access to modify plette it became possible to release changes required to support the feature of named package categories.

This particular feature for arbitrarily named package groups was requested multiple times in the pipenv issues reports. I hope the users of pipenv find this feature helpful in project workflows. Have you imagined other use cases not described in this article? Feel free to drop a message in the comments or send us a note.

Pipenv Install Performance Improved

Matt Davis — Tue, 06 Sep 2022 20:03:05 +0000

It wasn't until I changed jobs in November of 2021 that I began using something other than pip to manage Python packages for my projects. I started using pipenv at work and felt that it had a solid base and should use it in my own projects to manage package hashes and provide a more secure install. I also quickly realized that there were areas of the pipenv codebase that needed improvement.

I found myself combing through the GitHub Issues, triaging reports and responding to other users. At the start of 2022 I joined forces with Oz Tiram--with the support of Frost Ming who was seeking pipenv maintainers in order to work more exclusively on pdm, a modern alternative python package manager.

We have been hard at work in our spare time improving pipenv and there have have been more than a few exciting releases this year. The latest and greatest pipenv==2022.9.4 has worked out the remaining edge cases in a performance optimization that was released the end of August 2022.

Benchmarks have been a tough question to answer in the python package manager space--there are many variables: host computer, internet connection, private package indexes, many dependencies, resolver, installer, oh my! Well fortunately at the end of July 2022 Lincoln Loop built an independent benchmark for python package managers which runs on GitHub actions every 6 hours and averages the results over the prior 4 runs for the primary python package managers.

Upon learning of this benchmark, we began exporting the results at major release points in order to have data to track how pipenv performance is overtime. The 2022.8.5 release had an average install benchmark time of just under 150 seconds, and perhaps the 2022.8.24 release was marginally faster than that, we had just fully dropped pip-shims after all. The reality was, pipenv install was still slow and so I spent a weekend looking over the install logic in more detail and refactoring how the batch install worked.

The fundamental problem was that pip was being invoked per individual dependency and on my 8 core SMT processor laptop it sounded like a plane taking off. I began to wonder about batching up the resolved dependencies and invoking pip a minimal number of times to accomplish the same install. It turned out that the prior level of parallelism used a lot of processing power without improving the wall clock time.

The other potential problem was doing a true batch install meant it was no longer possible to show an incremental progress bar, where the increment the completion of each individual package install. The compromise here was to read the stdout of the pip process in real-time for users that pass in the --verbose flag which allows observing the installation progress, though its not visually a progress bar.

My PR went through the peer review process and fortunately Oz is very supportive of making such changes for performance, and so the excitement began to grow as we got closer to releasing the enhancement. What would our benchmark tell us about the install performance after this released? Based on my best feeling, I was expecting it could be a 33% speed increase but it was anybody's guess. Then August 31st we got our first inclination that the improvement was closer to 50% for the case of installing the sentry requirements, whoa!

So how does pipenv==2022.9.4 compare to the other package managers for install speed? Here are the benchmarks at the time of writing:

For the latest benchmarks and complete stats for each operation: install, lock, update, add package, and tooling--visit the Python Package Manager Shootout.