Deploying Python Software to Production (5 Part Series)
Easily deploy any Python application in form of an ‘omnibus’ Debian packaxge, i.e. one that contains all the application's dependencies, just like in a Java WAR. A basic understanding of Debian packaging, the Linux command prompt, and Python tooling is assumed.
In this article, I'll show how to use
dh-virtualenv to create self-contained Debian packages to deploy a Python application. The resulting package is very similar to a executable JAR that you can start via
java -jar, in that it contains all the moving parts except Python itself, without influencing or being influenced by version requirements of other applications. This also frees you from being restricted to the dependencies and their versions found on your target platforms, and makes porting to several different target environments easier.
The advantage of using a Debian package for deployment as opposed to the native Python tool chain is that you are less dependent on typical development tools and services, i.e. to deploy to QA or production environments you need neither Internet access nor any compiler suites (for extension packages). To achieve the same with direct use of
pip, you'd need to have an in-house PyPI repository accessible from production networks, and also release any extension packages as wheels pre-built for the target platform. Removing and updating an application is also much easier with Debian packages.
dh-virtualenv, you just have to extend your existing application project with a
debian subdirectory – project meta-data like
pip requirements and so on will be leveraged to build the final package, i.e. common tasks are delegated to the standard Python eco-system.
Note that just like with any other form of omnibus packaging, you take over the responsibility to release security updates of the contained dependencies in a timely manner.
dh-virtualenv is a debhelper plugin that extends the normal Debian tool chain for package building with the ability to create a Python virtualenv (an isolated Python environment), and then wrap that into the final Debian package.
Depending on the details of the application, you often also have to provide some kind of configuration of the software itself, and possibly some means to run it as a service. This can be done in several ways:
- add a
debian/«pkg».installdescriptor to add configuration files to the Debian package.
- provide a Puppet recipe or Ansible playbook that deploys the package and integrates it into the system.
- embed (default) configuration into the application's Python package (via the
All those can be combined, e.g. provide defaults via Python package data, and then add external configuration that only provides values specific to the concrete host installation.
A real-world example is the devpi supervisor ERB template that serves both the purpose of passing configuration to the application process (via command line options), and also starting and controlling that process (i.e. handle demonization and automatic startup on boot).
If this is your first time to build a Debian package, you also need to add the basic tools for that:
sudo apt-get install build-essential debhelper devscripts equivs
Finally, to take advantage of the available template for easily adding an inital
debian directory, install the cookiecutter tool. Note that you can opt to build packages in a Docker container instead, with only Docker as a requirement on your build host.
To add the necessary
debian directory with minimal effort, you can use the dh-virtualenv-mold cookiecutter. The following commands basically repeat what the integration test script of that project does, namely instantiate a Python project and then add debianization on top of it.
To provide common defaults to
cookiecutter, it makes sense to have a
~/.cookiecutterrc file similar to the one I use.
Let's first create a sample project:
mkdir -p ~/tmp/dh-venv-blog cd ~/tmp/dh-venv-blog cookiecutter --no-input \ "https://github.com/borntyping/cookiecutter-pypackage-minimal.git" cd cookiecutter_pypackage_minimal/ python3 setup.py build
You can of course also use one of your own, then just check that out instead. Next, we add the
cookiecutter --no-input \ "https://github.com/Springerle/dh-virtualenv-mold.git" dch -r "" # insert proper date & distro
--no-input causes the template's defaults to be accepted – it avoids answering all the template's prompts. After all, this is just a demo not requiring sensible inputs. Take the time to have a look at what's in the
You're now able to build the package and if that succeeds, print the contained meta data:
dpkg-buildpackage -uc -us -b dpkg-deb -I ../pyvenv-foobar_*.deb
The last command should show you something like this:
Package: pyvenv-foobar Version: 0.1.0 Architecture: amd64 Maintainer: Jürgen Hermann <firstname.lastname@example.org> Installed-Size: 12877 Pre-Depends: dpkg (>= 1.16.1), python3, python3-venv Section: contrib/python Priority: extra Homepage: https://github.com/jschmoe/foobar Description: A Python package and its dependencies packaged up as DEB in an isolated virtualenv.
Finally, install the new package via
dpkg -i, or upload it to a repository and use it from there with
These are examples of
dh-virtualenv packaging for non-trivial applications:
The last two show how to integrate Python web applications into
systemd, instead of using
supervisor like the
devpi example. The
Level up every day