Create a PyPI (pip) package, test it and publish it using Github Actions (PART 2)

Welcome back! Let's use Github Actions to deploy your shining new package to PyPI!

Basic setup

We don't have a git repository yet, so create one:

git init

Next, let's create a Github Repository

Once you're done with that, let's link our git repository to our Github repository:

git remote add origin "https://github.com/USERNAME/REPOSITORY.git"

Next, we'll commit and push our changes:

git add .
git commit -m "First Commit!"
git push origin master

You should see your code live at your Github Repository.

Setting up your first action

A Github Action is a YAML file. It can be edited in any code editor, but I recommend Github's own Editor, because it provides intellisense when you press Ctrl+Space.

To create an action, head over to the Actions tab. I'll create an action to automatically test and publish our package to TestPyPI whenever we've pushed our code to the master branch.

We can choose an action template, or we can start from scratch. I'm going to choose the latter, so click the set up a workflow yourself link.

This will create a new file that you can name whatever. I'll call it test-and-upload-to-testpypi.yml. Make sure that it has the extension yaml or yml, otherwise Github won't run it.

This should be the starter code you get:

# This is a basic workflow to help you get started with Actions

name: CI

# Controls when the action will run. 
on:
  # Triggers the workflow on push or pull request events but only for the master branch
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "build"
  build:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest

    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/checkout@v2

      # Runs a single command using the runners shell
      - name: Run a one-line script
        run: echo Hello, world!

      # Runs a set of commands using the runners shell
      - name: Run a multi-line script
        run: |
          echo Add other actions to build,
          echo test, and deploy your project.

Anatomy of a Github Action

Let's go through this file step-by-step.

name: It is the name of your Action. It will appear in the Actions tab under that name

on: Controls what events trigger your action. Currently, the action runs when there's a pull request or a push to the master branch, but there are other events too, like creation of releases, issues, adding collaborators, etc. The workflow-dispatch event will add a button in the Actions tab that'll allow you to run this action.

jobs: Defines the jobs in the action. Here, we have a single job called build.

Inside the build job:

runs-on: The OS to run on. Supports ubuntu-latest, windows-latest and macos-latest.

steps: The steps in your job.

Inside steps

uses: Another Github Action to use (this is optional)

name: Name of the step. Will appear in the Actions tab.

run: The command to run.

The pipeline (|) that you see after the run command in some steps is the YAML multi-line string operator. It allows a string to span mutliple lines, each newline indicating a new command.

Special actions

You can see that our action uses version v2 of actions/checkout. This is an action made by someone else and we're using it. It checks-out our code and copies it to the machine, so we can use it. There are several other actions like these available in the Github MarketPlace

Creating our action

We will have 5 main steps in our action:

• Checkout our code
• Install python
• Install our dependencies
• Test using unittest
• Publish to testpypi

For the first two steps, we have actions that will do it for us, namely action/checkout@v2 and actions/setup-python@v2.

I want the action to run when we've pushed our code to the master branch, and I also want to have a button in the Actions tab to manually test & upload the project for me.

Here's the final code for our action:

# This is a basic workflow to help you get started with Actions

name: Test & Upload to TestPyPI

# Controls when the action will run. 
on:
  # Triggers the workflow on push to the master branch
  push:
    branches: [ master ]

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "build"
  build:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest

    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/checkout@v2

      # Sets up python3
      - uses: actions/setup-python@v2
        with:
          python-version: 3.8 

      # Installs and upgrades pip, installs other dependencies and installs the package from setup.py
      - name: "Installs and upgrades pip, installs other dependencies and installs the package from setup.py"
        run: |
          # Upgrade pip
          python3 -m pip install --upgrade pip
          # Install build deps
          python3 -m pip install setuptools wheel twine
          # If requirements.txt exists, install from it
          if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
          # Install the package from setup.py
          python3 setup.py install

      # Tests with unittest
      - name: Test with unittest
        run: |
          cd tests
          python3 -m unittest discover
          cd ..

      # Upload to TestPyPI
      - name: Build and Upload to TestPyPI
        run: |
          python3 setup.py sdist bdist_wheel
          python3 -m twine upload dist/*
        env:
          TWINE_USERNAME: __token__
          TWINE_PASSWORD: ${{ secrets.TWINE_TEST_TOKEN }}
          TWINE_REPOSITORY: testpypi

What are secrets?

Secrets are Github's way of passing environment variables to our action. In this case, we used the secret TWINE_TEST_TOKEN to hold our API key.

Creating a secret

Go to the settings tab and select secrets. Click Create Secret on the top right and put TWINE_TEST_TOKEN for the name and paste your API key for the value.

Now, save the file and watch your action work from the Actions tab!

In our actions tab, we can see that this action is running:

After fixing a small bug with our tests, you can see that our action has failed, indicated by the red X icon

This is because whenever we upload a package to PyPI (or TestPyPI), we cannot upload the package of the same version twice, so let's up the version in setup.py

And our action has successfully executed!

Uploading to PyPI

We did TestPyPI. What about the real python package index? I'm not going to upload the package everytime we push master, since there could be some changes not ready for production. Instead, I'm going to use the releases feature of Github.

Create a new action by clicking New workflow in the Actions tab and following the same steps as before. This time, I'm naming my action upload-to-pip.yml

Here's my action:

# This is a basic workflow to help you get started with Actions

name: Upload to PIP

# Controls when the action will run. 
on:
  # Triggers the workflow when a release is created
  release: 
    types: [created]
  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "upload"
  upload:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest

    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/checkout@v2

      # Sets up python
      - uses: actions/setup-python@v2
        with:
          python-version: 3.8

      # Install dependencies
      - name: "Installs dependencies"
        run: |
          python3 -m pip install --upgrade pip
          python3 -m pip install setuptools wheel twine

      # Build and upload to PyPI
      - name: "Builds and uploads to PyPI"
        run: |
          python3 setup.py sdist bdist_wheel
          python3 -m twine upload dist/*
        env:
          TWINE_USERNAME: __token__
          TWINE_PASSWORD: ${{ secrets.TWINE_TOKEN }}

Adding another secret

Let's add another secret, this time called TWINE_TOKEN, and give it the value of your PyPI API token.

Testing our Action

Let's create a new release by going to the releases section (to do that, click New Release on the right side of the main code tab).

I suggest naming the tag with the version in setup.py prefixed with v.

And our project has successfully been uploaded to PyPI!

Congratulations 🥳! You just learnt the basics of Github Actions, PyPI projects and DevOps! Give yourself a pat on your back!

Stuck?
Github repo
PyPI
Github Actions Quickstart
Uploading a package on PyPI

If you enjoyed this post, and would like to see more, let me know in the comments, and also, follow me for more content like this 😀

Comments

[ヘンタイちゃん]

I would love to have read this six month ago, took me like a solid week to figure everything out. Great writeup!

[arnu515]

Thank you!