Armand Sauzay

Posted on May 16, 2023

How to properly setup your Python project

#python #softwareengineering #devops #cli

Industry best practices to kickstart your python project.

Photo by David Clode on Unsplash

As you start working on your python project, you will likely need to set it up
in a consistent and collaboration-friendly way. In this article, I'll describe a
setup that works great for my projects. It includes many industry best practices
and aims at explaining how to install python, run automated checks in a GitOps
fashion and structure your project.

This article covers quite a few topics. To ease the reading, I've split each topic into two parts:

📚 indicates the theory part
🛠️ indicates the practical part (i.e. the commands you need to run).

Sometimes you will also see a 💡 section, indicating a tip or a trick.

Install a Python version manager
Choose an environment manager (poetry)
Alternatively, use Docker as a Dev Environment instead
Add Some Code
Write Some Tests
Lint your code
Automate checks on local with pre-commit
Automate checks on remote with GitHub Actions
Automate your release with GitHub Actions
Enjoy the benefits of your new code practices

1. Install a Python version manager

📚 The first thing you will need to do is to install a python version manager. A
python version manager will allow you to install multiple versions of python on
your machine and switch between them easily.

To illustrate this, let's say you have a project that requires python 3.6 and
another project that requires python 3.7. If you only have python 3.7 installed
on your machine, you will have to uninstall it and install python 3.6 every time
you want to work on the first project... This is where a python version manager
comes in handy.

You can use pyenv for this and follow the
installation instructions.

🛠️ At the time of writing this document, you can install pyenv by running the
following:

curl https://pyenv.run | bash

If you want more control over the installation, on mac for instance you can run:

Use brew to install pyenv
```
brew install pyenv
```

Depending on the shell you use, add pyenv to your PATH.

💡 If you're not sure which shell you're using, you can run the following command:

echo $SHELL

For bash:

echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
echo 'command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(pyenv init -)"' >> ~/.bashrc

For zsh:

echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc
echo 'command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc
echo 'eval "$(pyenv init -)"' >> ~/.zshrc

2. Choose an environment manager (poetry)

📚 Now that you have a python version manager, you will need to install an
environment manager. An environment manager will allow you to create isolated
environments for your projects. This is useful when you work on multiple
projects that require different versions of the same package. And it is a great
practice overall to make sure anyone can run your project.

To illustrate this, let's say you have a project that requires pandas and
matplotlib to do some data analysis and another project that requires
tensorflow to do some machine learning. It is considered a good practice to
create a separate environment for each project. This way, you can install
only the packages you need for each project and avoid any conflicts between
packages.

There are many environment managers (pipenv, conda, virtualenv, python
built-in venv, etc...). I personally use poetry.

In general, your environment manager will be installed with your python version.
For instance, if you use pyenv, you can install poetry by running the following command:

pyenv install 3.10.6
pyenv global 3.10.6
pip install poetry

In this case, poetry will live in your python environment. This means that
you can have a different version of poetry for each python version you have
installed on your machine.

🛠️ Pip installing poetry in your python environment with pip is one way to do it but you can also follow the steps below:

Use the curl command to install poetry

curl -sSL https://install.python-poetry.org | python3 -

Add poetry to your PATH. On MacOS, poetry is added at ~/Library/Application Support/pypoetry/venv/bin/poetry. So you can add it to your PATH by adding the
following line to your ~/.bashrc or ~/.zshrc file:
```
export PATH="$HOME/Library/Application Support/pypoetry/venv/bin:$PATH"
```
If you are using a different shell, you can find the path to poetry by running the following command:
```
poetry config --list --local | grep virtualenvs.path
```
And then add it to your PATH.

3. Alternatively, use Docker as a Dev Environment instead

If you don't want to install a python version manager and an environment
manager, or want to abstract this for your team, you can use Docker instead. I
won't go into too many details here but if you're interested, VSCode has a great
integration with
Docker

4. Add Some Code

📚 Now that you have a python version manager and an environment manager, you
can start working on your project. Let's write some simple functions to get
started. For illustration purposes we'll create a simple CLI that prints "Hello
<name>" when you run it and give how many repositories <name> has on GitHub.

💡 By default, poetry wants you to create a package with a folder named as your
git repository (with the only difference that it wants underscores instead of dashes). So if your git
repository is called python-project-template, poetry will create a folder
called python_project_template.

💡 You can find the code for this section at armand-sauzay/hello-world-cli

In this case let's call our project hello-world. So we'll create a folder
called hello_world and add a file called __init__.py to it.

🛠️ Use gh to create a new repository called hello-world and clone it to your
machine:

gh repo create <your-github-username>/hello-world-cli --template armand-sauzay/python-template --public

To initialize your project, run the following command (it comes from the template above):

./script/bootstrap

Add the requests package to your project:

poetry add requests

So far, we ran the following commands:

pyenv install 3.10.6 # install python 3.10.6
pyenv global 3.10.6 # (optional) set python 3.10.6 as the default python version
pip install poetry # install poetry as a python package in your python 3.10.6 environment
poetry add requests # add the requests package to your project

In hello_world_cli folder a file called main.py and add the following code to it:

import argparse
import requests


def main(argv=None) -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--name", help="The name to greet and count repos for", default="world"
    )
    args = parser.parse_args(argv)
    if args.name == "":
        print("Username cannot be empty")
        return 1

    print(f"Hello {args.name}!")

    repos = requests.get(f"https://api.github.com/users/{args.name}/repos")
    if repos.status_code != 200:
        print(f"Failed to fetch repos for {args.name}")
        return 1
    repos = repos.json()
    print(f"You have {len(repos)} repos.")  # type: ignore
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

There are a few tricks here that are worth mentioning:

We use argparse to parse the arguments passed to our CLI. This is a built-in python package that allows you to parse arguments passed to your CLI.
We use argv to pass the arguments to our main function. This is useful for testing purposes. We can pass a list of arguments to our main function and test it without having to run the CLI.
We use SystemExit to exit our program. This is a built-in python exception that allows you to exit your program with a specific exit code (0 for success, 1 for failure).

5. Write Some Tests

📚 Now that you have a few basic methods, you can start writing some tests. Tests make sure your code behaves as expected. There
are many types of tests (unit tests, integration tests, end-to-end tests, etc...)
but in this article, we will focus on unit tests. Unit tests are tests that
check the smallest unit of your code (i.e. a function or a method).

🛠️ Create a file called test_main.py and add the following code to it:

from hello_world_cli.cli import main

def test_main(capsys):
    assert main(['--name', 'test']) == 0
    out, err = capsys.readouterr()
    # test is a user which does not have any contributions since 2010. 
    # Hopefully this will not change in the future and the test will not break.
    assert out == 'Hello test!\nYou have 5 repos.\n'
    assert err == ''

def test_main_empty_name(capsys):
    assert main(['--name', '']) == 1
    out, err = capsys.readouterr()
    assert out == 'Username cannot be empty\n'
    assert err == ''

We use capsys to capture the output of our CLI. This is a pytest built-in that allows you to capture the output of your CLI.

6. Lint your code

📚 Now that you have some tests, you can start linting your code. Linting is the
process of checking your code for potential errors. There are many linters out
there (pylint, flake8, black, etc...). I personally use flake8, black, isort and
mypy. Ruff is also becoming quite popular and replaces flake8 and isort.

🛠️ To install flake8, black, isort and mypy, you can run the following command:

poetry add --dev flake8 black isort mypy

Then you can use mypy to check your code:

poetry run mypy hello_world_cli/cli.py

You can use flake8 to check your code:

poetry run flake8 hello_world_cli/cli.py

You can use black to format your code:

poetry run black hello_world_cli/cli.py

But you would not really want to run these commands manually every time you
change your code. This is where pre-commit comes in handy, to both make sure you
don't commit wrongly formatted code and to not have to run these commands
manually.

Also, do note that some of these linters have small incompatible differences.
For example, black will complain if you have a line longer than 88 characters,
but flake8 will complain if you have a line longer than 79 characters. You need
to configure them to work correctly together:

you should create a .flake8 file with the following content:

[flake8]
max-line-length = 88
extend-ignore = E203

you should create a .isort.cfg file with the following content:

[settings]
profile = black

you can read more on the black documentation

7. Automate checks on local with pre-commit

📚 Now that you have some tests and some linters, you can start automating your
checks. The goal of automating your checks is to make sure your code is always
in a good state. There are many tools out there that can help you automate your
checks (pre-commit, tox, etc...). I personally use pre-commit.

🛠️ To install pre-commit, you can run the following command:

brew install pre-commit

Then you can run pre-commit

pre-commit run --all-files

You can see on the image above that pre-commit is running the checks that we defined in the .pre-commit-config.yaml file.
Note that this file is coming from the template above in step 4.

8. Automate checks on remote with GitHub Actions

📚 Now that you have some tests and some linters, you can start automating your
checks on the server side. The goal of automating your checks on the server side
is to make sure your code is always in a good state. There are many tools out
there that can help you automate your checks (GitHub Actions, Travis CI, Circle
CI, etc...). I personally use GitHub Actions.

🛠️ To set up GitHub Actions, you can create a file called .github/workflows/ci.yaml and add the following code to it:

name: CI

on:
  workflow_dispatch:
  push:
jobs:
  lint:
    name: Lint
    runs-on: ubuntu-latest
    steps:
      - uses: armand-sauzay/actions-python/lint@v1
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}

  test:
    name: Test
    runs-on: ubuntu-latest
    steps:
      - uses: armand-sauzay/actions-python/test@v1
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          test-flags: --version

Note that this file should already be created if you used the template mentionned above (see gh command in step 4.

Explanation:

If you're not familiar with the syntax of GitHub Actions, you can read the documentation.
Here we call two jobs: lint and test. Each job runs on ubuntu-latest (a Linux machine).
The lint job uses the armand-sauzay/actions-python/lint action to run pre-commit.
The test job uses the armand-sauzay/actions-python/test action to run pytest. Here we pass a flag to pytest to just have a dummy test. You can remove this flag to run your actual tests.

9. Automate your release with GitHub Actions

📚 We have briefly talked about linting - using black, flake8, isort and mypy - and testing - using pytest. Release is an equally important concept that allows you to give versions to

🛠️ To set up GitHub Actions, you can create a file called .github/workflows/release.yaml and add the following code to it:

name: Release

on:
  push:
    branches:
      - main
  workflow_dispatch: # enable manual release

jobs:
  lint:
    name: Lint
    runs-on: ubuntu-latest
    steps:
      - uses: armand-sauzay/actions-python/lint@v1
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}

  test:
    name: Test
    runs-on: ubuntu-latest
    steps:
      - uses: armand-sauzay/actions-python/test@v1
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          test-flags: --version

  release:
    name: Release
    needs: [lint, test]
    runs-on: ubuntu-latest
    outputs:
      new-release-published: ${{ steps.release.outputs.new-release-published }}
      new-release-version: ${{ steps.release.outputs.new-release-version }}
    steps:
      - uses: armand-sauzay/actions-python/release@v1
        id: release
        with:
          github-token: ${{ secrets.ADMIN_TOKEN || secrets.GITHUB_TOKEN }}

release

A few things worth noting:

this workflow will trigger on push on the main branch and on manual trigger (workflow_dispatch)
it will run the lint and test jobs
it will then run the release job which is semantically versioning your code and creating a release on GitHub. In case you don't know what semantic versioning is, you can read the documentation.
In order to follow semantic versioning, you can follow conventional commits (this is why the commits show up as fix: ... or feat: ... in the release notes). You can read more on conventional commits.
Sometimes you might have a protected branch, for which the default GITHUB_TOKEN wouldn't have enough permissions to create a release. In that case, you can create a personal access token called ADMIN_TOKEN and give it the right permissions. You can then use this token in the workflow.