In this article, we will take a look at a tool called "micropipenv". Its main goal is to serve as a common layer for installing Python dependencies as specified by pip, pip-tools, Pipenv or Poetry.
If you are a Python developer, you’ve probably experienced that dependency hell where you can easily end up in with Python’s packaging. A tool called pip has been around for quite some time, but its implementation is not that sufficient if you develop or maintain any larger Python-based application.
pip-tools
A project called pip-tools tries to address the lack of dependency management in pip. It uses two text files - requirements.in and requirements.txt. The first file, requirments.in managed by a user, states direct dependencies of an application with version range specifications. The latter one, requirements.txt managed by pip-tools, acts as a lock file stating all the packages in specific versions necessary to run an application (an analogy to npm-shrinkwrap.json or package-lock.json from the npm ecosystem).
To use pip-tools, you need to explicitly maintain a Python virtual environment (if you need one). Starting Python 3.3., you can issue the following command to create one and activate it:
python3 -m venv venv/
source venv/bin/active
After you have created the virtual environment, commands pip-compile and pip-sync will become your friends.
pip-tools workflow as described in pip-tools package description available on PyPI
For dependencies needed to execute your test suite, pip-tools introduced a convention using dev-requirements.in and dev-requirements.txt. The semantics can be analogically deduced.
A note to setup.py and requirments.txt
You have probably seen requirements.txt file used with setup.py. Note the difference with pip-tools that can be misleading for newcomers. If you want to publish a library, you don’t want to restrict all the versions by pinning transitive dependencies to specific versions as you don’t know how the resolved software stack will look like on an application side. The final resolution should always happen on the application level, not on the library level. You, as a library maintainer, just want to provide information about the compatibility of your library with dependencies used within the library.
Pipenv
Kenneth Reitz, the author of one of the most popular Python library — requests, introduced a project called Pipenv in January 2017. The project gained popularity and attention from the community very quickly.
Pipenv: Python Dev Workflow for Humans
The main aim of the project was to simplify dependency management and make it more user-friendly. It helped to maintain a virtual environment and manage dependencies with one single command, using newly introduced files Pipfile (TOML) and Pipfile.lock (JSON).
Pipenv in our team
We, at Red Hat, adopted Pipenv and made it one of the options for dependency management during deployment in the OpenShift’s Source-To-Image build process (see for example Fedora-based Python container images or Thoth’s Python s2i container images). It gained popularity also in our AICoE and Thoth team where de-facto all the repositories with Python source code use Pipenv for dependency management.
Poetry
Another option is to use Poetry. It looks like Poetry attracted the Python community, especially during the silent phase of Pipenv. Poetry uses Pipenv: Python Dev Workflow for Humans
a different type of lock format than Pipenv and is not compatible with Pipenv or pip at all.
micropipenv
Some of the requirements and ideas lead to introducing a new tool called micropipenv. No, there is no intention to introduce another pip, pip-tools, Poetry or Pipenv.
This lightweight implementation (one file, around 1200 LOC with comments) has one optional dependency and can install your requirements from files that are managed using Pipenv, Poetry, pip-tools, or simple requirements.txt file as commonly used together with setup.py. The only thing you need to do is to issue:
micropipenv install
The tool will automatically detect what type of requirements file or lock file you use for managing your dependencies and performs the desired installation.
Moreover, the tool offers a simple conversion feature where any lock file can be transformed to requirements.txt and/or requirements.in.
To produce a pip-tools style requirements.in and requirements.txt you can simply perform the following commands, assuming you have Pipenv or Poetry lock files present in the directory:
micropipenv requirements --no-dev > requirements.txt
micropipenv requirements --no-dev --only-direct > requirements.in
micropipenv requirements --no-default > dev-requirements.txt
micropipenv requirements --no-default --only-direct > dev-requirements.in
If you just want to install dependencies, you don’t need to install micropipenv at all. You can just simply download it and let it do its one time job:
curl https://raw.githubusercontent.com/thoth-station/micropipenv/master/micropipenv.py | python3 - install
See documentation and sources for more info.
Thoth
One of the ongoing efforts in Red Hat’s Office of the CTO is a project called Thoth. The newly introduced tool in this article, micropipenv, was born in this project. Check one of our publicly available demos to see micropipenv in action (the micropipenv demo starts at 9:00):
You can follow our YouTube channel for more updates.
Why micropipenv?
The main reason behind micropipenv was to reduce the maintenance cost of Pipenv during its silent phase. However, it turned out to be a good idea to have such a minimalistic tool for installing dependencies, especially when it comes to containerized applications. The main advantage of micropipenv turned out to be its size.
When deploying applications in containerized environments, it’s a really good idea to maintain a lock file for the application. As the lock file states the whole dependency stack already resolved, there is no reason why there needs to be shipped Poetry or Pipenv in the container image. A tool that just installs the dependencies from any lock file supplied seems to be like a minimalistic way to go to reduce container image size and software present in it (and thus shipped with the application).
A simple size comparison done a while back showed approximately 30.4MiB difference when Pipenv was not installed into the containerized environment in comparison to a single file approach using micropipenv.
micropipenv can read in
Pipfile.lock,requirements.txtorpoetry.lockstating already resolved software stack and install it using pip.
The design of the CLI made micropipenv a straightforward tool to make a compatibility layer between all popular Python dependency management tools available out there in the open-source world.
Project Thoth
Project Thoth is an application that aims to help Python developers. If you wish to be updated on any improvements and any progress we make in project Thoth, feel free to subscribe to our YouTube channel where we post updates as well as recordings from scrum demos. We also have a Twitter account.
Top comments (0)