If you are a Python developer, you’ve probably experienced that dependency hell where you can easily end up in with Python’s packaging. A tool called
pip has been around for quite some time, but its implementation is not that sufficient if you develop or maintain any larger Python-based application.
A project called pip-tools tries to address the lack of dependency management in pip. It uses two text files -
requirements.txt. The first file,
requirments.in managed by a user, states direct dependencies of an application with version range specifications. The latter one,
requirements.txt managed by
pip-tools, acts as a lock file stating all the packages in specific versions necessary to run an application (an analogy to
package-lock.json from the npm ecosystem).
pip-tools, you need to explicitly maintain a Python virtual environment (if you need one). Starting Python 3.3., you can issue the following command to create one and activate it:
python3 -m venv venv/ source venv/bin/active
After you have created the virtual environment, commands
pip-sync will become your friends.
pip-tools workflow as described in pip-tools package description available on PyPI
For dependencies needed to execute your test suite,
pip-tools introduced a convention using
dev-requirements.txt. The semantics can be analogically deduced.
You have probably seen
requirements.txt file used with
setup.py. Note the difference with
pip-tools that can be misleading for newcomers. If you want to publish a library, you don’t want to restrict all the versions by pinning transitive dependencies to specific versions as you don’t know how the resolved software stack will look like on an application side. The final resolution should always happen on the application level, not on the library level. You, as a library maintainer, just want to provide information about the compatibility of your library with dependencies used within the library.
Kenneth Reitz, the author of one of the most popular Python library — requests, introduced a project called Pipenv in January 2017. The project gained popularity and attention from the community very quickly.
Pipenv: Python Dev Workflow for Humans
The main aim of the project was to simplify dependency management and make it more user-friendly. It helped to maintain a virtual environment and manage dependencies with one single command, using newly introduced files
Pipfile (TOML) and
We, at Red Hat, adopted Pipenv and made it one of the options for dependency management during deployment in the OpenShift’s Source-To-Image build process (see for example Fedora-based Python container images or Thoth’s Python s2i container images). It gained popularity also in our AICoE and Thoth team where de-facto all the repositories with Python source code use Pipenv for dependency management.
Another option is to use Poetry. It looks like Poetry attracted the Python community, especially during the silent phase of Pipenv. Poetry uses Pipenv: Python Dev Workflow for Humans
a different type of lock format than Pipenv and is not compatible with Pipenv or pip at all.
Some of the requirements and ideas lead to introducing a new tool called micropipenv. No, there is no intention to introduce another
This lightweight implementation (one file, around 1200 LOC with comments) has one optional dependency and can install your requirements from files that are managed using Pipenv, Poetry, pip-tools, or simple
requirements.txt file as commonly used together with
setup.py. The only thing you need to do is to issue:
The tool will automatically detect what type of requirements file or lock file you use for managing your dependencies and performs the desired installation.
Moreover, the tool offers a simple conversion feature where any lock file can be transformed to
To produce a pip-tools style
requirements.txt you can simply perform the following commands, assuming you have Pipenv or Poetry lock files present in the directory:
micropipenv requirements --no-dev > requirements.txt micropipenv requirements --no-dev --only-direct > requirements.in micropipenv requirements --no-default > dev-requirements.txt micropipenv requirements --no-default --only-direct > dev-requirements.in
If you just want to install dependencies, you don’t need to install micropipenv at all. You can just simply download it and let it do its one time job:
curl https://raw.githubusercontent.com/thoth-station/micropipenv/master/micropipenv.py | python3 - install
See documentation and sources for more info.
One of the ongoing efforts in Red Hat’s Office of the CTO is a project called Thoth. The newly introduced tool in this article, micropipenv, was born in this project. Check one of our publicly available demos to see micropipenv in action (the micropipenv demo starts at 9:00):
You can follow our YouTube channel for more updates.
The main reason behind micropipenv was to reduce the maintenance cost of Pipenv during its silent phase. However, it turned out to be a good idea to have such a minimalistic tool for installing dependencies, especially when it comes to containerized applications. The main advantage of micropipenv turned out to be its size.
When deploying applications in containerized environments, it’s a really good idea to maintain a lock file for the application. As the lock file states the whole dependency stack already resolved, there is no reason why there needs to be shipped Poetry or Pipenv in the container image. A tool that just installs the dependencies from any lock file supplied seems to be like a minimalistic way to go to reduce container image size and software present in it (and thus shipped with the application).
A simple size comparison done a while back showed approximately 30.4MiB difference when Pipenv was not installed into the containerized environment in comparison to a single file approach using micropipenv.
micropipenv can read in
poetry.lockstating already resolved software stack and install it using pip.
The design of the CLI made micropipenv a straightforward tool to make a compatibility layer between all popular Python dependency management tools available out there in the open-source world.
Project Thoth is an application that aims to help Python developers. If you wish to be updated on any improvements and any progress we make in project Thoth, feel free to subscribe to our YouTube channel where we post updates as well as recordings from scrum demos. We also have a Twitter account.