DEV Community: Cristian Medina

12 Trending Alternatives to Distribute Python Applications in 2020

Cristian Medina — Tue, 10 Dec 2019 04:00:00 +0000

One of the more prevalent topics in the Python ecosystem of 2019 was that of packaging and distribution. As the year comes to an end, I wanted to put together a summary of the many paths we currently have available to distribute apps built with Python. Though some of these also apply to any language.

Whether delivering an executable, a virtual environment, your packaged code, or a full application, the following list includes both standard systems and some up-and-comers to keep in mind as we enter 2020.

Applications

To distribute an application, you need more than just a library to pip-install from PyPI. You require a fool-proof way of installing it and its dependencies in all supported operating systems. These dependencies can include other non-Python packages as well as external resources like binaries or images.

Below are some options to help install and distribute code across platforms.

Docker

Docker uses base functionality in an operating system to isolate a process in such a way that it's unaware of the rest of the system. It can run the kernel process of a different OS inside your host, much like virtualization, but without using the virtual hardware.

Its file system is layered, giving it a minimal footprint that only incorporates the files needed to run, instead of the typical virtual disk that also includes the free space in a virtual machine along with it.

With a minimally packaged root file system, it's not uncommon to see the image for an entire OS only occupy tens of MB, instead of the GB needed for a virtual machine.

You can distribute these containers in a public registry like DockerHub or a private one inside your org. Users install the Docker daemon on their own compute, then use it to pull your image and run it locally.

Since Docker images are nothing more than a root file system, it's also possible to distribute them as a file, using Docker to import it.

To build an image, you can start from a rootfs, or add on top of an existing registry image. Most operating system vendors maintain official stripped-down images in DockerHub, which are usually tiny.

Other organizations also make official images for new builds of their applications, like the Python image built on top of Debian. Postgress, MySQL, Redis, Nginx, and many other standard services do the same.

Docker now runs on Linux, OSX, and Windows. This cross-platform support means that any image you build has a wide distribution with minimal complexity. You control not only the application but also the environment it runs on, making compatibility much less of an issue.

However, complexity can exist when configuring the network or persistent storage. The typical application doesn't have to deal with anything more than port forwarding, but it's sometimes hard to visualize how the abstraction layers work. It helps to budget for better documentation around it.

With Docker, you can control the Python distribution, the supporting OS packages - like C libraries needed for individual modules - and your virtual environment.

Since it takes milliseconds for a container to start, it's perfectly acceptable, even encouraged, to run a fresh container every time you wish to execute your app.

Some people even use Docker as a virtual environment replacement. Since it starts up quickly and can offer an interactive shell, it's not a bad idea to make a new container whenever you need to work on a particular project.

Once you have a running container configured with the basics, you can save the image for reuse or even export it to a file.

For example, the following command starts a new Python container, mounts the current working directory as /work, and drops you into a bash shell. Changes made inside the container directory reflect in the base directory.

docker run -it --name python-work -v `pwd`:/work python:3-slim bash

Once inside the container, you can install whatever Apt or PyPI packages are needed.

Exiting the shell returns you to the host. At which point, running this next command saves any changes as a new Docker image that you can reuse later or push to DockerHub for distribution:

docker commit python-work my-python-image

It's possible to share the new image in a private Docker Registry internal to your organization, or run the following to export the container as a file that anyone can download and import into their local Docker environment:

docker export python-work -o my-python-work.dock

However, the real workflow for distribution is to create a Dockerfile that anyone can use to create the image themselves. In other words, you give out a small text file with instructions for the Docker daemon, instead of a copy of the entire filesystem.

Anyone could clone your repo with that file and run this command to create a local version of the image:

docker build -t my-python-image .

A typical Dockerfile looks like below, please look through their documentation for more info:

FROM Python:3-slim

COPY path/to/your/app /work

WORKDIR /work
RUN pip install -r requirements.txt

More details

Virtual Machines and Vagrant

The next step up from containers is to distribute a full virtual machine. This type of system has been around for a while since virtualization became widespread and hardware supported.

Delivering a "virtual appliance" is attractive because you have close to total control of the environment in which your app runs. Everything is configurable: the operating system, its packages, disks, and networking, even the amount of free space.

The drawback of using VMs is the size of your distribution, typically in the gigs. Plus, you'll have to work out a mechanism for getting the image to your customers. Things like Amazon S3 or Digital Ocean Spaces are a great place to start.

In the beginning, only server hardware supported running virtual machines, but these days every processor has the capability, and all major operating systems support it. There are also free applications to help you manage and configure VMs like Oracle's VirtualBox.

Vagrant is another system for configuring and running VMs on top of managers like VirtualBox. It functions a lot like Docker in that you specify everything the VM needs in a file, and it takes care of building and running it for you.

Similar to the Dockerfile in the previous section, Hashicorp's Vagrant uses a Vagrantfile with instructions on how to start and configure a virtual machine.

Just like a Docker image provides the base file system to run a container, a Vagrant box provides the basis for a virtual machine.

The example Vagrantfile below does something similar to the Dockerfile in the previous section:

Vagrant.configure("2") do |config|
 config.vm.box = "bento/debian-10.2"
 config.vm.synced_folder "./", "/work"
 config.vm.provision "shell",
 inline: "pip install -r /work/requirements.txt
 keep_color: true
end

Cloning your repository with this file and running these commands starts the VM and gets you into its shell:

vagrant up
vagrant ssh

Vagrant does help the distribution problem by providing a similar experience to DockerHub with the Vagrant Box catalog.

With it, you can get your base images or upload new ones to share with others. You can even point a Vagrantfile to internal URLs to download a box.

More details

PyInstaller

We've written about this module before. PyInstaller takes care of bundling all resources required to run your application, including the Python distribution. Crawling your code, it figures out which Python dependencies to package, while still allowing you to specify additional assets to include with the bundle.

The result is an installable application for either Windows, Linux, or OSX. During execution, it unpacks it onto a folder, along with the bundled interpreter, and runs your entrypoint script.

It's flexible enough to give you control over the Python distribution and the execution environment. I've even successfully used it to bundle browsers with my Python packages.

But using it does bring a complication. When extracting itself, it changes the base directory from which your application runs. Meaning, any code dependent on __file__ to determine the current execution path now needs to use the internal environment that PyInstaller configures.

Distributing your bundle is entirely up to you. Most people choose object stores and CDNs for this type of setup.

A point to keep in mind when distributing this way is to check whether your code needs to validate the OS environment it's running on.

In other words, if you need a specific apt package installed, unlike using Docker or Vagrant, there's no guarantee that the package is already there at the time of execution.

More details

Briefcase

Briefcase is an up-and-comer in this category. It's a part of the Beeware project that aims to enable the packaging of Python applications for distribution to all operating systems and devices, including Android and iOS.

It's in a similar arena as PyInstaller, meaning it can bundle your module along with its dependencies into an installable application. But it also adds support for mobile devices and TVs with AppleTV or tvOS.

Unfortunately, documentation is still a little lax and mostly in the form of examples. However, the project shows promise and is still under active development. The popular editor Mu uses it for packaging.

You can submit applications built with Briefcase to the Android and Apple App Stores for distribution.

More details

Virtual Environments

Sometimes, it's possible to assume that your users have a standard operating system. Maybe they're all running from a stock image built by the IT department inside your company. Maybe your app is just simple enough to not worry about OS or interpreter complexities.

If all you need to think about is your Python code and its dependent libraries, this category is for you.

Pex

Built by the folks at Twitter, Pex is a way of distributing an entire virtual environment along with your Python application. Designed to use a pre-installed Python interpreter, it leverages the standard built for Python Zip Applications outlined in PEP-441.

Since Python 2.6, the interpreter has the ability to execute directories or zip-format archives as scripts.

Pex builds on top of that, simplifying distribution to the act of copying a single file. These files can work across platforms (OSX, Linux, Windows), as well as different interpreters. Though there are some limitations when using modules with C bindings.

After installing Pex in your base system, you can use it to produce a single file containing your entire Python environment.

Pass that file to your coworker's computer, and you'll be able to execute it there without installing anything other than the base Python interpreter.

You can even run a file in interpreter mode, such that it opens a Python REPL with all the necessary modules in your environment available for import.

Freezing your virtual environment into a .pex file is easy enough:

pex -r requirements.txt -o virtualenv.pex

Then you can execute that file to open a REPL with your environment:

./virtualenv.pex

You can also specify entrypoints when creating the file so that it executes a specific function in your module, behaving as if you run the python command directly.

Here's a great 15 min video with an example that bundles a simple Flask app.

There's no system to distribute Pex files for you, so just like the previous items, you're stuck with public object stores and CDNs.

More details

Shiv

Similar to Pex, the folks at LinkedIn built a different module called Shiv. Their main reason for building something different was to try and tackle a problem in the start time of Pex executables. Given the complexity of their repositories and dependencies, they wanted to handle the environment setup differently.

Instead of bundling wheels along with the application, Shiv includes an entire site-packages directory as installed by pip. It makes everything work right out of the box and almost twice as fast as Pex.

Here's an example of how to produce a .pyz file with Shiv that does something similar to the Pex section:

shiv --compressed -o virtualenv.pyz -r requirements.txt

Which you can then execute directly with:

./virtualenv.pyz

It's important to note that packaging libraries with OS dependencies is not cross-compatible between platforms. As mentioned in the Pex section, it's mainly an issue for modules that depend on lower-level C libraries. You'll have to produce different files for each platform.

Again, there's no system built for you to distribute these files, so you'll have to rely on AWS, DO, CDNs, or other artifact stores like JFrog's Artifactory or Sonatype's Nexus.

More details

Pipx

While not a method to build applications or distribute them, Pipx offers a different way to install them. It works with your OS to isolate the virtual environments and its dependencies, closer to what a system like Homebrew does for OSX.

Pipx provides an easy way to install packages into isolated environments and expose their command line entrypoints globally. It also provides a mechanism to list, upgrade, and uninstall those packages without getting into the virtual environment details.

A good example is the use of a linter. Let's say you work on multiple python applications, each of which has a separate virtualenv, and you wish to perform the same linting operations across all of them using flake8.

Instead of installing the flake8 module into every virtualenv, you can use Pipx to install a system-wide flake8 command that's available inside each of those environments but runs in its own entirely separate and isolated environment.

More details

Single-file Executables

Sometimes you want to give your customers an executable that doesn't require pre-installed libraries to run, as you would with Docker or Pex.

The mechanisms described here help you accomplish that. And just like the previous category, they all require some form of object or artifact store to help with distribution.

PyInstaller

While we already discussed PyInstaller, it's worth another mention in this category, because this is one of its primary functions.

It can produce a single-file executable of your entire application with all its dependencies bundled. You can make one for each operating system, and it runs just like any other native application.

PyOxidizer

One of the newest additions to the packaging and distribution arena, PyOxidizer, is very promising. It takes advantage of the packaging tools created for the Rust programming language.

Much like PyInstaller, you have complete control of everything you want to bundle into it, but it also allows you to execute code much like Pex or Shiv. Meaning you can create your package so that it runs as a REPL with all dependencies pre-installed for you.

Distributing an entire Python environment that includes the REPL makes for some exciting applications, especially with research teams or scientific computing that require several packages to do data exploration.

One advantage over PyInstaller is that instead of extracting out to the file system, it extracts itself into memory, making the start-up time of your actual Python application considerably faster.

This feature comes with a similar drawback to PyInstaller. You have to adjust any internal references to __file__ or similar operations, so they rely on the environment configured by PyOxidizer at runtime.

More details available in their official documentation, but we also wrote more about it in this article.

Nuitka

Beyond executing your Python code with a bundled interpreter, you also have the option of compiling your code down to C. This comes with several advantages, including the possibility of faster execution given that the C compiler can perform optimizations that the interpreter cannot.

Nuitka is a system built for compiling Python code to C. While the concept is similar to the more widely known Cython, it's not a separate language. And it's capable of doing things that Cythong can't, like crawling your dependencies and compiling everything down to one binary.

The resulting executable runs as is much faster, in native code, and never needing extraction.

Compilation can get complicated, especially when considering platform complexities. But if you budget the time for it, you'll be able to reap the benefits. I've done it successfully several times before.

More details

App Store Experiences

Other systems for distributing applications we use very successfully almost every day: app stores. This software exists to install and maintain other applications.

Just like the Apple App Store or the Google Play Store, there are similar mechanisms available in Linux that enable simple integrations.

Snapcraft

Snapcraft provides the standard app store experience. You can publish your application to their store, where users can discover and install it.

The installation is isolated to avoid conflicts with other applications, and it works across Linux distributions, including library dependencies.

Once installed, the store automatically keeps your application at the latest stable version and provides a mechanism to revert to previous states while preserving data.

Ubuntu manages the store, so after bundling your application (or snap as they call it), you'll have to publish the snap to the store with a registered Ubuntu One account.

More details

Flatpak

Another very similar concept to Snapcraft is Flatpak.

It also presents a store-like experience with FlatHub.org using container technologies to provide isolation to your application. Still, you can also host your own private hub, or distribute bundles in a single file.

Flatpak bundles are can also make use of some desktop integration capabilities. These provide information like locality detection, the ability to access resources external to the app (much like your phone asks for permissions to open files or URLs), notifications, window decorations, and others.

More details and the instructions to create your first Flatpak are available here.

Summarizing

We have a full, feature-rich ecosystem of application packaging and distribution mechanisms. Most of these are not specific to the Python language but easily integrate with it.

While it seems like many options, hopefully, the categorization applied here should help you pick what best suits your needs, given the pieces you can control.

Learn More!

Subscribe to the tryexceptpass.org mailing list for more content about Python, Docker, open source, and our experiences with enterprise software engineering.

Unconventional Secure and Asynchronous RESTful APIs using SSH

Cristian Medina — Fri, 15 Nov 2019 04:00:00 +0000

Some time ago, in a desperate search for asynchronicity, I came across a Python package that changed the way I look at remote interfaces: AsyncSSH.

Reading through their documentation and example code, you’ll find an interesting assortment of use cases. All of which take advantage of the authentication and encryption capabilities of SSH, while using Python’s asyncio to handle asynchronous communications.

Thinking about various applications I’ve developed over the years, many included functions that could benefit from decoupling into separate services. But at times, I would avoid it due to security implications.

I wanted to build informative dashboards that optimize maintenance tasks. But they bypassed business logic, so I wouldn’t dare expose them over the same interfaces. I even looked at using HTTPS client certs, but support from REST frameworks seemed limited.

I realized that asyncssh could provide the extra security I was looking for over a well known key-based system. And in my never-ending quest to find what makes things tick, I decided to take a stab at writing a REST-ish service over SSH.

A great way to familiarize myself with the library and the protocol, it helped me learn more about building asynchronous apps, creating a small framework called korv.

Read On...

Painless Status Reporting in GitHub Pull Requests - Designing CI/CD Systems

Cristian Medina — Mon, 14 Oct 2019 04:00:00 +0000

Continuing the build service discussion from the Designing CI/CD Systems series, we’re now at a good point to look at reporting status as code passes through the system.

At the very minimum, you want to communicate build results to our users, but it’s worth examining other steps in the process that also provide useful information.

The code for reporting status isn’t a major feat. However, using it to enforce build workflows can get complicated when implemented from scratch.

Since the pipeline covered so far in earlier articles already integrates with GitHub, it’s much easier to simplify things by taking advantage of GitHub’s features. Specifically, we can use the Status API to convey information directly into pull requests, and use repository settings to gate merges based on those statuses.

Reporting status to GitHub

We had a brief discussion of this API in a previous article about integrating pytest results with GitHub. It also covered GitHub Apps and how to authenticate them into the REST API. Today we’ll discuss more details about the Status API itself, keeping in mind a pre-existing App.

Reporting pull request status is a simple HTTP POST request to the status endpoint of the relevant PR. You can find that URL as part of the webhook event details related to the PR.

Read On...

Command Execution Tricks with Subprocess - Designing CI/CD Systems

Cristian Medina — Mon, 23 Sep 2019 04:00:00 +0000

The most crucial step in any continuous integration process is the one that executes build instructions and tests their output. There’s an infinite number of ways to implement this step ranging from a simple shell script to a complex task system.

Keeping with the principles of simplicity and practicality, today we’ll look at continuing the series on Designing CI/CD Systems with our implementation of the execution script.

Previous chapters in the series already established the build directives to implement. They covered the format and location of the build specification file. As well as the docker environment in which it runs and its limitations.

Execution using subprocess

Most directives supplied in the YAML spec file are lists of shell commands. So let's look at how Python's subprocess module helps us in this situation.

We need to execute a command, wait for it to complete, check the exit code, and print any output that goes to stdout or stderr. We have a choice between call(), check_call(), check_output(), and run(), all of which are wrappers around a lower-level popen() function that can provide more granular process control.

This run() function is a more recent addition from Python 3.5. It provides the necessary execute, block, and check behavior we're looking for, raising a CalledProcessError exception whenever it finds a failure.

Also of note, the shlex module is a complimentary library that provides some utilities to aid you in making subprocess calls. It provides a split() function that's smart enough to properly format a list given a command-line string. As well as quote() to help escape shell commands and avoid shell injection vulnerabilities.

Security considerations

Thinking about this for a minute, realize that you're writing an execution system that runs command-line instructions as written by a third party. It has significant security implications and is the primary reason why most online build services do not let you get down into this level of detail.

So what can we do to mitigate the risks?

Read On ...

Designing Continuous Build Systems: Docker Swarm Orchestration

Cristian Medina — Fri, 23 Aug 2019 04:00:00 +0000

Building code is sometimes as simple as executing a script. But a full-featured build system requires a lot more supporting infrastructure to handle multiple build requests at the same time, manage compute resources, distribute artifacts, etc.

After our last chapter discussing build events, this next iteration in the continuous builds series covers how to spin-up a container inside Docker Swarm to run a build and test it.

What is Docker Swarm

When running the Docker engine in Swarm mode your effectively creating a cluster. Docker will manage a number of compute nodes and their resources, scheduling work across them that runs inside containers.

It handles scaling across nodes while maintaining overall cluster state, such that you can adjust how many worker containers are running in the cluster, automatically failover when nodes go offline, etc.

It also builds the networking hooks necessary so that containers can communicate with each other across multiple host nodes. It does load balancing, rolling updates, and a number of other functions you would expect from cluster technologies.

Cluster setup

When compute hosts form part of a Docker Swarm, they can either run in manager mode or as regular worker nodes. The nodes are able to host containers as directed by the managers.

One Swarm can have multiple managers, and the managers themselves can also host containers. Their job is to track the state of the cluster and spin-up containers across nodes as needed. This allows for redundancy across the cluster such that you can loose one or more managers or nodes and keep basic operations running. More details on this are available in the official Docker Swarm documentation.

To create a swarm you need to run the following command on your first manager (which also serves as your first node).

docker swarm init

The previous command will tell you what to run in each of your nodes in order to join that swarm. It usually looks like this:

docker swarm join --token SOME-TOKEN SOME_IP:SOME_PORT

The docker daemon listens on a unix socket located at /var/run/docker.sock by default. This is great for local access, but if you need remote access, you'll have to enable tcp sockets explicitly.

Enabling remote access

Docker Swarm provides a good API for managing services, but for our particular use case, we need a feature that's not available at the swarm-level and requires individual access to the nodes. Part of the reason is because we're using Swarm for a special case that it wasn't built for: running a one-off short-lived container - more on this later.

You'll have to enable remote access on your node daemons in order to connect to them directly. Doing so seems to vary a bit between Linux versions, distributions and the location of docker config files. However, the main objective is the same: you must add a -H tcp://IP_ADDRESS:2375 option into the daemon service execution, where IP_ADDRESS is the interface it listens on.

You'll find that most examples set it to 0.0.0.0 so that anyone can connect to it, but I would recommend limitting it to a specific address for better security - more below.

I was on an Ubuntu image that used the file in /lib/systemd/system/docker.service to define the daemon options. You just find the line that starts with ExecStart=... or has a -H in it, and add the extra -H option mentioned previously.

Don't forget you have to reload daemon configs and restart docker after making the change:

sudo systemctl daemon-reload
sudo service docker restart

I've seen other distributions that track these settings under /etc/default/docker, and yet another that uses a file in /etc/systemd/system/docker.service.d/. You should google for docker daemon enable tcp or docker daemon enable remote api paired with your OS flavor to be sure.

Security implications

Given the nature of what you can do with Docker, it's important to point out that enabling TCP sockets for remote access is a very serious security risk. It basically opens your system to remote code execution because anyone could connect to that socket and start or stop containers, view logs, modify network resources, etc.

To mitigate this, you'll want to enable the use of certificate validation along with TCP sockets. This makes the daemon validate that the HTTPS certificate used by potential clients is signed by a pre-defined certificate authority (CA).

You create the certificate authority and sign any client certs before distributing them to the compute systems that will perform the orchestration - usually your build service.

Steps on how to generate the certificates, perform the signing and enable the verification options are available in the Docker documentaion for protecting the daemon.

Services, Tasks and Containers

When running in swarm mode, Docker terminology changes a little. You're no longer concerned only with containers and images, but also with tasks and services.

A service defines all the pieces that makeup an application running in the cluster. These pieces are tasks, and each task is a definition for a container.

For example, if you have Application ABC that runs a Flask API that you wish to load balance across two nodes, you define one ABC service with two tasks. The swarm takes care of keeping them running in two nodes (even if there are more nodes in the cluster) and also configures the network so that the service is available over the same port regardless of the node your connected to.

The number of tasks to run are part of a replication strategy that the swarm uses to determine how many copies of the tasks to keep running in the cluster. Not only can you set it to a specific number, but you can also configure it in a global mode, that runs a copy of the tasks in every node of the swarm.

These concepts are simple in principle, but can get tricky when you're trying to do more complicated things later. So I recommend you check out the service documentation for more information on how it all works.

Using this terminology to describe our use case: for every new build request, you'll run a new service in the swarm that contains one instance of a task and one container that performs the build. The swarm scheduler will take care of provisioning it in whatever node is available. The container should delete itself ones the work completes.

Alternatives

As mentioned earlier, Docker Swarm and its concepts are meant for maintaining long-running replicated services inside a cluster. But we have the very specific case of single-copy ephimeral services executed for every build.

We don't care about high availability or load balancing features, we want it for its container scheduling capabilities.

While simple to do, since it wasn't built for this, it can feel like we're forcing things together. So another option is to build our own scheduler (or use an existing one) and have it execute work inside containers.

This isn't hard with existing task systems similar to Celery or Dramatiq that use work queues like RabbitMQ to distribute container management tasks.

Along the same lines you can reuse distributed compute systems for the same goal. I've deployed Dask successfully for this purpose. It even simplified some workflows and enabled others that wouldn't be possible otherwise.

I know I've said this many times before, but I'll say it again. Like most choices in software engineering (and in life), you're always exchanging one set of problems for another. In this case, you exchange workflow complexity for infrastructure and maintenance complexity because you now have to keep worker threads running and listening to queues, as well as handle updates to those threads as your software evolves. This is completely abstracted for you by using a Docker Swarm.

Orchestration with Python

The official Python library maintained by the Docker folks is the docker module. It wraps all the main constructs and is straightforward to use. I've been leveraging it for a while now.

The library interacts with the docker daemon REST API. The daemon's command interfaces use HTTP verbs on resource URLs to transfer JSON data. For example: listing containers is a GET to /containers/json, creating a volume is a POST to /volumes/create, etc.

If you're interested, visit the Docker API Reference for more details.

APIClient vs DockerClient

The docker module itself exposes two "layers" of communication with the daemon. They manifest as different client classes: APIClient and DockerClient. The former is a lower-level wrapper around the interface endpoints directly, while the latter is an object-oriented layer of abstraction on top of that client.

For our purpose today, we'll be able to stick with instances of DockerClient to perform all operations. Going to the lower-level is rare, but sometimes required. Both interfaces are well documented in the link shared earlier.

Creating a client is very simple. After installing the module with pip install docker, you can import the client class and instantiate it without any parameters. By default it will connect to the unix socket mentioned earlier.

from docker import DockerClient
dock = DockerClient()

The DockerClient class follows a general "client.resource.command" architecture that makes it intuitive to use. For example: you can list containers with client.containers.list(), or view image details with client.images.get('python:3-slim').

Each resource object has methods for generic actions like list(), create() or get(), as well as those specific to the resource itself like exec_run() for containers.

All attributes are available with the .attrs property in the form of a dictionary, and the reload() method fetches the latest information on a resource and refreshes the instance.

To accomplish our goals, you'll need to create a service for each build, find the node where its task executes, make some changes to the container inside that task and start the container.

The execution script

Configuring a swarm service that executes a build is only half the battle. The other half is writing the code that follows the directives we defined in an earlier chapter to build, test, record results and distribute artifacts. This is what I call the execution script.

We'll discuss specifics on how the script itself works in future articles. For now it's sufficient to know that it's the command that each build container runs whenever it starts. This is relevant because it brings up another issue we have to handle: the execution script is written in Python, but the build container is not required to have Python installed.

I've designed and implemented continuous integration systems that require Python to execute, as well as those that don't. One adds complexity and constraints to the developers using the system, the other to the maintainers of the system.

If it's a known fact that you'll only ever build Python code, then this doesn't really matter because the docker images used in the code repositories being built already have Python installed.

If this is not the case, then you may see a substantial increase in build times, complexitiy and maybe even a limitation in supported images because the developers have to install Python into the container as part of their build.

My choice these days is to package the execution script such that it's runnable inside any docker image. I documented my attempts in a previous article about packaging Python modules as executables, where I concluded on using PyInstaller to do the job. Details on how to produce a package are included there.

Building and Executing Tests

With all the ingredients at hand, it's time to dive into the code that makes a new service and runs the build. As defined in earlier chapters of this series, the following steps assume that:

The repository being built has a YAML file in its root directory with the build directives.
This configuration file contains an image directive defining the Docker image to use with the build.
There's an environment directive as well where the user can set environment variables.
The webhook functions handling build events have retrieved build configuration info and stored it in a dictionary called config and provide pull request info in a pr dict.

Creating the service

When making the service, you need to consider what to name it, and how to find it when searching the swarm.

Service names must be unique and in order to simplify infrastructure maintenance, they should be descriptive. I prefer to use a suffix of -{repo_owner}-{repo_name}-{pr_number}-{timestamp}. There's also string size limits in these names, so careful not to get too creative.

Because you don't want duplicate builds of the same pull request, you also need the ability to programmatically search the swarm for a running build. In other words, if I'm executing a build for a given pull request, there's no point in allowing the build to complete if I committed new code in that same PR. Not only are you wasting resources, but even if the build finishes successfully, it's useless because it's already out of date.

To handle this situation, I use labels. A service can have one or multiple labels with metadata about what it is and what it's doing. The Docker API also provides methods for filtering based on this metadata.

The code that follows leverages those functions to determine whether a build is already running and stops it before creating a new service.

import logging
import docker

...

def execute(pr, action=None, docker_node_port=None):
    ...

    # Get environment variables defined in the config
    environment = config['environment'] if 'environment' in config and isinstance(config['environment'], dict) else {}

    # Get network ports definition from the config
    ports = docker.types.EndpointSpec(ports=config['ports']) if 'ports' in config and isinstance(config['ports'], dict) else None

    environment.update({
        'FORGE_INSTALL_ID': forge.install_id,
        'FORGE_ACTION': 'execute' if action is None else action,
        'FORGE_PULL_REQUEST': pr['number'],
        'FORGE_OWNER': owner,
        'FORGE_REPO': repo,
        'FORGE_SHA': sha,
        'FORGE_STATUS_URL': pr['statuses_url'],
        'FORGE_COMMIT_COUNT': str(pr['commits'])
    })
    logging.debug(f'Container environment\n{environment}')

    # Connect to the Docker daemon
    dock = docker.DockerClient()

    # Stop any builds already running on the same pr
    for service in dock.services.list(filters={'label': [f"forge.repo={owner}/{repo}", f"forge.pull_request={pr['number']}"]}):
        logging.info(f"Found service {service.name} already running for this PR")

        # Remove the service
        service.remove()

    # Create the execution service
    service_name = f"forge-{owner}-{repo}-{pr['number']}-{datetime.now().strftime('%Y%m%dT%H%M%S')}"
    logging.info(f"Creating execution service {service_name}...")

    service = dock.services.create(
        config['image'],
        command=f"/forgexec",
        name=service_name,
        env=[f'{k}={v}' for k, v in environment.items()],
        restart_policy=docker.types.RestartPolicy('none'),
        labels={
            'forge.repo': f'{owner}/{repo}',
            'forge.pull_request': str(pr['number']),
        }
    )

Note that before creating the service, we're not only grabbing the environment variables defined in the build config, but also adding extras that describe the action we're taking. They pass relevant information about the repository and pull request being built onto the execution script.

As you can see, we're using .services.list() to get a list of services currently running in the swarm filtered with the labels we described earlier. If a service exists, calling service.remove() will also get rid of its containers.

Creating the service is a call to .services.create() where we pass:

The container image that the service is based on - defined in our build config.
A command to execute when it starts, which is the name of our execution script - forgexec in this case.
The service name.
Environment variables are defined as a list of strings formatted as NAME=VALUE, so we convert them from our environment dict using a list comprehension.
The restart policy is the term that Docker uses to define what to do with containers in the event of a host restart. In our case, we don't want them to automatically come online, so we set it to none.
The labels with the metadata we described earlier.

Getting task information

Once the swarm creates the service, it takes a few seconds before it initializes its task and provisions the node and container that runs it. So we wait until its available by checking for the total number of tasks assigned to the service:

    # Wait for service, task and container to initialize
    while 1:
        if len(service.tasks()) > 0:
            task = service.tasks()[0]
            if 'ContainerStatus' in task['Status'] and 'ContainerID' in task['Status']['ContainerStatus']:
                break
        time.sleep(1)

There's only one task in a build service, so we can always assume it's the first one in the list. Each task is a dictionary with attributes describing the container that runs it.

There are two delays here: one before the task is assigned and one before a container spins up for the task. So it's necessary to wait until container information is available within the task details before continuing.

Copying the execution script into the container

As discussed earlier, you need to copy our packaged execution script into each build container in order for it to start - the equivalent of a docker cp.

Copying data into a container requires the files to be tar'd and compressed, so at the very beginning of our event server script we create a tar.gz file using the tarfile Python module:

if __name__ == '__main__':
    # Setup the execution script tarfile that copies into containers
    with tarfile.open('forgexec.tar.gz', 'w:gz') as tar:
        tar.add('forgexec.dist/forgexec', 'forgexec')

This means we have a forgexec.tar.gz file available to transfer with the container.put_archive() function that the docker module provides. Do this every time the webhook event server starts and override any existing file to make sure that you're not using stale code.

Transferring files into a container requires us to connect to the swarm node directly. There's no interface at the docker service level to help us do that. This is why we had to enable remote access earlier.

First we get information about the docker node from the task and then we make a new DockerClient instance that connects to the node:

    node = dock.nodes.get(task['NodeID'])
    nodeclient = docker.DockerClient(f"{node.attrs['Description']['Hostname']}:{docker_node_port}")

This time, the instantiation uses the tcp port in which the nodes are listening (configured during cluster setup) and the hostname of the node. Depending on your network and DNS setup, you may want to use the socket module to help with domain name resolution. Something like socket.gethostbyname(node.attrs['Description']['Hostname']) might be good enough.

At this point we can directly access the container, copy the file into it and start it:

    # Get container object
    container = nodeclient.containers.get(task['Status']['ContainerStatus']['ContainerID'])

    # Copy the file
    with open('forgexec.tar.gz', 'rb') as f:
        container.put_archive(path='/', data=f.read())

    # Start the container
    container.start()

Putting it together

Here's our new execute() function merged with code from the webhook event handling chapter:

def execute(pr, action=None, docker_node_port=None):
    """Kick off .forge.yml test actions inside a docker container"""

    logging.info(f"Attempting to run {'' if action is None else action} tests for PR #{pr['number']}")

    owner = pr['head']['repo']['owner']['login']
    repo = pr['head']['repo']['name']
    sha = pr['head']['sha']

    # Select the forge for this user
    forge = forges[owner]

    # Get build info
    config = get_build_config(owner, repo, sha)

    if config is None or config.get('image') is None or config.get('execute') is None:
        logging.info('Unable to find or parse the .forge.yml configuration')
        return

    # Get environment variables defined in the config
    environment = config['environment'] if 'environment' in config and isinstance(config['environment'], dict) else {}

    # Get network ports definition from the config
    ports = docker.types.EndpointSpec(ports=config['ports']) if 'ports' in config and isinstance(config['ports'], dict) else None

    environment.update({
        'FORGE_INSTALL_ID': forge.install_id,
        'FORGE_ACTION': 'execute' if action is None else action,
        'FORGE_PULL_REQUEST': pr['number'],
        'FORGE_OWNER': owner,
        'FORGE_REPO': repo,
        'FORGE_SHA': sha,
        'FORGE_STATUS_URL': pr['statuses_url'],
        'FORGE_COMMIT_COUNT': str(pr['commits'])
    })
    logging.debug(f'Container environment\n{environment}')

    # Connect to the Docker daemon
    dock = docker.DockerClient(docker_host)

    # Stop any builds already running on the same pr
    for service in dock.services.list(filters={'label': [f"forge.repo={owner}/{repo}", f"forge.pull_request={pr['number']}"]}):
        logging.info(f"Found service {service.name} already running for this PR")

        # Remove the service
        service.remove()

    # Create the execution service
    service_name = f"forge-{owner}-{repo}-{pr['number']}-{datetime.now().strftime('%Y%m%dT%H%M%S')}"
    logging.info(f"Creating execution service {service_name}...")

    service = dock.services.create(
        config['image'],
        command=f"/forgexec",
        name=service_name,
        env=[f'{k}={v}' for k, v in environment.items()],
        restart_policy=docker.types.RestartPolicy('none'),
        # mounts=[],
        labels={
            'forge.repo': f'{owner}/{repo}',
            'forge.pull_request': str(pr['number']),
        }
    )

    # Wait for service, task and container to initialize
    while 1:
        if len(service.tasks()) > 0:
            task = service.tasks()[0]
            if 'ContainerStatus' in task['Status'] and 'ContainerID' in task['Status']['ContainerStatus']:
                break
        time.sleep(1)

    node = dock.nodes.get(task['NodeID'])
    nodeclient = docker.DockerClient(f"{node.attrs['Description']['Hostname']}:{docker_node_port}")
    container = nodeclient.containers.get(task['Status']['ContainerStatus']['ContainerID'])

    with open('forgexec.tar.gz', 'rb') as f:
        container.put_archive(path='/', data=f.read())

    container.start()

What's next?

This article gave you the details needed to use Docker Swarm for provisioning compute that builds code inside a cluster. Along with the previous chapters on handling repository events and defining the directives required to execute a build, you're ready to move on to the next piece that covers the execution script itself. You'll need to configure the build environment, run commands and execute tests for the different stages in the pipeline.

Learn More!

Subscribe to the tryexceptpass.org mailing list for more content about Python, Docker, open source, and our experiences with enterprise software engineering.

Designing Continuous Build Systems: Handling Webhooks with Sanic

Cristian Medina — Sat, 10 Aug 2019 04:00:00 +0000

After covering how to design a build pipeline and define build directives in the continuous builds series, it’s time to look at handling events from a code repository.

As internet standards evolved over the years, the HTTP protocol has become more prevalent. It’s easier to route, simpler to implement and even more reliable. This ubiquity makes it easier for applications that traverse or live on the public internet to communicate with each other. As a result of this, the idea of webhooks came to be as an “event-over-http” mechanism.

With GitHub as the repository management platform, we have the advantage of using their webhook system to communicate user actions over the internet and into our build pipeline.

Read On ...

4 Attempts at Packaging Python as an Executable

Cristian Medina — Sun, 28 Jul 2019 04:00:00 +0000

A few years back I researched how to create a single-file executable of a Python application. Back then, the goal was to make a desktop interface that included other files and binaries in one bundle. Using PyInstaller I built a single binary file that could execute across platforms and looked just like any other application.

Fast forward until today and I have a similar need, but a different use case. I want to run Python code inside a Docker container, but the container image cannot require a Python installation.

Instead of blindly repeating what I tried last time, I decided to investigate more alternatives and discuss them here.

Read On ...

Designing Continuous Build Systems - Parsing the Specification

Cristian Medina — Mon, 15 Jul 2019 04:00:00 +0000

Every code repository is different. The execution environment, the framework, the deliverables, or even the linters, all need some sort of customization. Creating a flexible build system requires a mechanism that specifies the steps to follow at different stages of a pipeline.As the next chapter in the series Designing Continuous Build Systems, this article examines which instructions you’ll want to convey to your custom system and how to parse them.

Read On ...

PyCaribbean 2018 - Sofi Unity3d Lightning Talk

Cristian Medina — Fri, 05 Jul 2019 04:40:03 +0000

PyCaribbean 2018 lightning talk where I demo sofi-unity3d from a live Python interpreter. The talk starts at 18:43 and shows how to spawn, manipulate and animate objects in 3d space using Unity3D.

PyCaribbean 2018 - Practicality Beats Purity

Cristian Medina — Fri, 05 Jul 2019 04:28:32 +0000

Running Enterprise Builds in the Cloud

Cristian Medina — Fri, 28 Jun 2019 04:00:00 +0000

There are many solutions to building code, some of them are available as cloud services, others run on your own infrastructure, on private clouds or all of the above. They make it easy to create custom pipelines, as well as simple testing and packaging solutions. Some even offer open source feature-limited “community editions” to download and run on-premises for free.

Following is my experience on the important aspects to consider when deciding on a build system for your organization that depends on cloud services. The original intent was to include a detailed review of various online solutions, but I decided to leave that for another time. Instead, I’m speaking more from an enterprise viewpoint, which is closer to reality in a large organization than the usual how-to’s. We’re here to discuss the implications and the practicality of doing so.

Read on ...

Designing Continuous Build Systems

Cristian Medina — Wed, 24 Apr 2019 04:00:00 +0000

Continuous integration and delivery is finally becoming a common goal for teams of all sizes. After building a couple of these systems at small and medium scales, I wanted to write down ideas, design choices and lessons learned. This post is the first in a series that explores the design of a custom build system created around common development workflows, using off-the-shelf components where possible. You’ll get an understanding of the basic components, how they interact, and maybe an open source project with example code from which to start your own.

Read on ...