DEV Community: Scott Houseman

Creating a great Python DevX

Scott Houseman — Fri, 08 Sep 2023 21:38:54 +0000

Create an enjoyable and meaningful Python Developer eXperience.

I have recently created a trivial open source Python package, named tally-counter.
The objective here was to play around with some tooling that could make shipping Python projects easier and better, rather than the product itself.

"The journey is the thing."

-- Homer

Let me take you through it.

Tooling

Important
The scope of this post is not to give the reader instruction on how to configure and implement these tools. Some tips are given, and excellent documentation is available at the links provided.

Nox

Nox is a command-line tool that automates testing in multiple Python environments.

Why use this? An open source Python package should support all current Python versions (currently >= 3.9, <= 3.11). Nox can run unit tests and linting fixes or checks in all of these Python versions in a single execution.

Here is my noxfile.py (kind of cool that the configuration language is Python):

"""Nox configuration."""
import enum
import os

import nox

# Set to True if Nox is running in CI (GitHub Actions)
CI = os.environ.get("CI") is not None

# Supported Python versions
PYTHON_VERSIONS = ["3.8", "3.9", "3.10", "3.11"]


class Tag(str, enum.Enum):
    """Define acceptable tag values."""

    TEST = "test"
    LINT = "lint"


@nox.session(python=PYTHON_VERSIONS, tags=[Tag.TEST])
def pytest(session):
    """Run all unit tests."""
    ...  # Snipped


@nox.session(python=PYTHON_VERSIONS, tags=[Tag.TEST])
def doctest(session):
    """Run doc tests."""
    ...  # Snipped


@nox.session(python=PYTHON_VERSIONS, tags=[Tag.LINT])
def black(session):
    """Run the black formatter."""
    ...  # Snipped


@nox.session(python=PYTHON_VERSIONS, tags=[Tag.LINT])
def ruff(session):
    """Run the ruff linter."""
    ...  # Snipped


@nox.session(python=PYTHON_VERSIONS, tags=[Tag.LINT])
def mypy(session):
    """Run the mypy type checker."""
    ...  # Snipped

View source

Note
Specifying session tags=[...] parameters enables grouping of sessions by purpose.

Ruff

Ruff is an extremely fast Python linter, written in Rust.

Why use this? Linting is a great way to detect buggy or poorly-implemented (smelly) code.
Ruff is able to lint code to a number of well-defined style rules.

Here is an extract of rules to follow from the pyproject.toml file:

[tool.ruff]
select = [
  "E",  # pycodestyle Error
  "F",  # Pyflakes
  "B",  # flake8-bugbear
  "W",  # pycodestyle Warning
  "I",  # isort
  "N",  # pep8-naming
  "D",  # pydocstyle
  "PL", # Pylint
]
ignore = [
  "D107", # Missing docstring in `__init__`
  "D203", # 1 blank line required before class docstring
  "D212", # Multi-line docstring summary should start at the first line
]

View source

Ruff editor integrations are available.

Mypy

Mypy is a static type checker for Python.

Why use this? Checked Python type annotations make code safer and easier to read and understand.

A NyPy Visual Studio Code extension is available.

Black

Black is the uncompromising Python code formatter.

Why use this? When code is shared in a team or community environment, it makes sense to have this code conform to a standard format. This removes ambiguity and allows contributors to focus on implementation. Formatted code is also less prone to raising linting issues.

Black editor integrations are available.

pre-commit

pre-commit runs a number of checks before git commits may succeed.

Why use this? Running pre-commit checks may prevent committing incomplete or faulty code by detecting issues before the commit may succeed. Some of these are just downright useful:

configuration file (*.yaml, *.toml) formatting
newlines at the end of file
merge conflict markers
unit test function names
committing to main branch
etc., etc., etc.

make

I have also created a Makefile that contains some helpful shortcuts for often-used instructions.

❯ make help
help                 Show this help message
install              Install dependencies
lint                 Run linting in all supported Python versions
test                 Run unit tests in all supported Python versions
update               Update dependencies

The lint and test targets use Nox to run these jobs on all supported Python versions (>= 3.8, <= 3.11).
This does take a bit longer, and requires that the supported Python versions are installed on the developer's machine (on that note, do have a look at pyenv).

You can thus specify which Python version(s) should be used by Nox like this:

make lint PYTHON_VERSIONS="3.8"

Continuous Integration and Delivery

The project makes use of GitHub Actions to facilitate Ci/CD workflow.

When it comes to release workflows, there is no one size fits all solution. Requirements will vary based on the project's maturity and the number of contributors. And requirements will change as the project grows.

For this project, I have decided on the typical Git Feature Branch Workflow.

Create a new feature branch off main
Branch further off - and merge back into - this feature branch as required. The Build action runs for each push:
- If the current branch is off main, then
  - check for a semantic version increment (or "bump")
- Run all linting checks
- Run all unit tests
When ready, create a pull request to merge the feature branch to main
Once the pull request is approved, the branch should be squashed and merged to main
- The Build action runs
- if the branch is main, then
  - Create a new release named for the semantic version
  - Create a Python package build
  - Publish this version to PyPI

Build

The Build Action runs these steps:

check that the branch's semantic version has been bumped, in pyproject.toml
run linting checks on Ubuntu, MacOS and Windows operating systems, for all supported Python versions
run unit test on Ubuntu, MacOS and Windows operating systems, for all supported Python versions
if the branch is main, create a new GitHub release, named for the semantic version

Semantic version check

This was a bit of a hack, but it works quite well. At the moment, I cannot think of a better way.

Thepyproject.toml project.version attribute is our single source of truth for the package sematic version.

[project]
name = "tally-counter"
version = "0.0.9"

To create a mechanism of retrieving this, the package __init__.py contains this code to set a __version__ variable:

"""Tally Counter."""
import importlib.metadata

# ... snipped ...
__version__ = importlib.metadata.version("tally_counter")

This string values can then be accessed by:

>>> import tally_counter
>>> tally_counter.__version__
'0.0.9'

The build CI can then map this output to a BRANCH_VERSION environment variable:

# ... snipped ..
jobs:
  # Get the HEAD branch semantic version
  head-branch-version:
    runs-on: ubuntu-latest
    # Map outputs to environment variables
    outputs:
      HEAD_VERSION: ${{ steps.set-head-version.outputs.HEAD_VERSION }}
      HEAD_OFF_MAIN: ${{ steps.set-head-off-main.outputs.HEAD_OFF_MAIN }}
    steps:
      # ... snipped ..
      - name: Pip install this package
        run: |
          python -m pip install --upgrade pip
          pip install .
      - name: Set HEAD_VERSION
        id: set-head-version
        run: echo "HEAD_VERSION=$(python -c 'import tally_counter; print(tally_counter.__version__)')" >> "$GITHUB_OUTPUT"
      - name: Set HEAD_OFF_MAIN
        id: set-head-off-main
        if: ${{ github.ref != 'refs/heads/main' }} # Skip if "main"
        run: |
          git fetch origin main:main
          if git merge-base --is-ancestor main HEAD; then
            echo "HEAD_OFF_MAIN=1" >> "$GITHUB_OUTPUT"
          else
            echo "HEAD_OFF_MAIN=0" >> "$GITHUB_OUTPUT"
          fi
# ... snipped ..

And then checkout main, and do the same again

# ... snipped ..
jobs:
  # ... snipped ..
  version-check:
    runs-on: ubuntu-latest
    needs: head-branch-version
    # Run a semantic version check if this push is to a main descendant
    if: ${{ needs.head-branch-version.outputs.HEAD_OFF_MAIN == '1' }}
    steps:
      # ... snipped ..
      - name: Checkout main branch
        run: |
          git fetch origin main:main
          git checkout main
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install semver
          pip install .
      - name: Set MAIN_VERSION
        id: set-main-version
        run: echo "MAIN_VERSION=$(python -c 'import tally_counter; print(tally_counter.__version__)')" >> "$GITHUB_OUTPUT"
      - name: Compare HEAD_VERSION > MAIN_VERSION
        env:
          MAIN_VERSION: ${{ steps.set-main-version.outputs.MAIN_VERSION }}
          HEAD_VERSION: ${{ needs.head-branch-version.outputs.HEAD_VERSION }}
        run: python -c "import semver; assert semver.compare('${{ env.HEAD_VERSION }}', '${{ env.MAIN_VERSION }}') > 0, 'Version not bumped'"
# ... snipped ..

Once all checks have passed, a release is built.

# ... snipped ..
  # Create a new release if branch is "main"
  release:
    needs: [head-branch-version, build]
    runs-on: ubuntu-latest
    if: ${{ github.ref == 'refs/heads/main' }} # Only create a release if the push branch is "main"
    steps:
      - name: Create Release
        uses: actions/create-release@v1
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          HEAD_VERSION: ${{ needs.head-branch-version.outputs.HEAD_VERSION }}
        with:
          tag_name: "v${{ env.HEAD_VERSION }}"
          release_name: "v${{ env.HEAD_VERSION }}"
          draft: false
          prerelease: false

Publish

A second action will publish the new version to PyPI

Conclusion

This was just a quick and very high-level overview of the process. It can almost certainly be improved upon.
Have a look at the https://github.com/houseman/tally-counter repository to gain further insight on how it all fits together.

Cheers!

A Better Way To Store Record Status In A Relational Database

Scott Houseman — Mon, 14 Aug 2023 21:39:10 +0000

Definition

A status field is a relational database column or relationship (foreign key) in a table that indicates the current state or condition of each record in that table.

The value of this field is expected to change throughout the lifetime of any given table record.

Use Case

For this use case example, within a contrived e-commerce system, a catalogue products table might have status values such as:


`in stock`	In warehouse stock. May be purchased and shipped.
`on order`	On back order. May be purchased and shipped.
`unavailable`	Unavailable for purchase. May be viewed in customer order history.
`deleted`	Should not be viewable to customers.

Various database structures may be used to store this status. We'll explore these below.

A naive design: Use a varchar field

The most naive database design would simply store this status field as a varchar type:



/* Postgres */
DROP    TABLE IF EXISTS products CASCADE;
CREATE  TABLE           products (
    product_id  SERIAL        PRIMARY KEY,
    title       VARCHAR(200)  NOT NULL,
    sku         VARCHAR(50)   NOT NULL UNIQUE,
    status      VARCHAR(20)   NOT NULL
);
DROP    INDEX IF EXISTS idx_product_status;
CREATE  INDEX           idx_product_status
ON                      products(status);

Adding some sample data



INSERT  INTO products
        (title,                           sku,              status)
VALUES  ('EcoBoost Portable Charger',     'SKU-ECBW-1234',  'in stock'),
        ('AquaPure Water Filter',         'SKU-AQPF-5678',  'on order'),
        ('SolarGlow Garden Lights',       'SKU-SGLA-9101',  'unavailable'),
        ('FitFlex Yoga Mat',              'SKU-FFYM-1121',  'deleted'),
        ('BreezeAir Conditioner',         'SKU-BACA-3141',  'in stock'),
        ('CrispSound Bluetooth Speaker',  'SKU-CSBS-5161',  'on order'),
        ('SmoothBlend Juicer',            'SKU-SBJG-7181',  'unavailable'),
        ('QuickCook Microwave Oven',      'SKU-QCMO-9201',  'deleted'),
        ('UltraView Binoculars',          'SKU-UVBK-1221',  'in stock'),
        ('ProFit Running Shoes',          'SKU-PFRS-3241',  'in stock');

will give us a data set that looks like this:



SELECT  product_id,
        title,
        sku,
        status
FROM    products;
 product_id |            title             |      sku      |   status
------------+------------------------------+---------------+-------------
          1 | EcoBoost Portable Charger    | SKU-ECBW-1234 | in stock
          2 | AquaPure Water Filter        | SKU-AQPF-5678 | on order
          3 | SolarGlow Garden Lights      | SKU-SGLA-9101 | unavailable
          4 | FitFlex Yoga Mat             | SKU-FFYM-1121 | deleted
          5 | BreezeAir Conditioner        | SKU-BACA-3141 | in stock
          6 | CrispSound Bluetooth Speaker | SKU-CSBS-5161 | on order
          7 | SmoothBlend Juicer           | SKU-SBJG-7181 | unavailable
          8 | QuickCook Microwave Oven     | SKU-QCMO-9201 | deleted
          9 | UltraView Binoculars         | SKU-UVBK-1221 | in stock
         10 | ProFit Running Shoes         | SKU-PFRS-3241 | in stock
(10 rows)

The problem here is that the status varchar type is unconstrained; the value could be any string and this could lead to data inconsistencies.

An improved design: use an enum field

This improved design makes use of an ENUM type to define a restricted set of valid status values:



/* Postgres */
DROP    TABLE  IF EXISTS   product_status CASCADE;
DROP    TYPE   IF EXISTS   product_status CASCADE;
CREATE  TYPE               product_status AS ENUM (
    'in stock',
    'on order',
    'unavailable',
    'deleted'
);

DROP    TABLE IF EXISTS products  CASCADE;
CREATE  TABLE           products  (
    product_id  SERIAL          PRIMARY KEY,
    title       VARCHAR(200)    NOT NULL,
    sku         VARCHAR(50)     NOT NULL UNIQUE,
    status      product_status  NOT NULL          -- **Note** type here
);
DROP    INDEX IF EXISTS idx_product_status;
CREATE  INDEX           idx_product_status
ON                      products(status);

This limits the possible value of status to one of the defined string values; 'in stock', 'on order', 'unavailable' or 'deleted'.

There are several benefits of using an enum type over a varchar:

Data Integrity: ensure that the value is always within a specific set of values. This is not possible with varchar (unless you add a CHECK constraint).
Performance: enum values are stored as integers, making comparing and filtering data more efficient..
Indexing: enum types can be more efficient than indexing varchar columns, which can lead to faster search and retrieval of data.
Storage: enum values are stored as integers, which can be more space-efficient than varchar.

However, adding new values to an enum type requires database schema changes, which may be a heavy operation if your database is large.

Metadata

Metadata is data that provides information about other data.

These enum status values have the following metadata with regards to the relevant products table record:

Value	In stock	Buyable	Active
`in stock`	Yes	Yes	Yes
`on order`	No	Yes	Yes
`unavailable`	No	No	Yes
`deleted`	No	No	No

This metadata now need to be implemented through some business logic in code.

Something like:



# status.py
from __future__ import annotations

from dataclasses import dataclass


@dataclass
class ProductStatus:
    """A data model for product status"""

    is_in_stock: bool
    is_buyable: bool
    is_active: bool

    @classmethod
    def create(cls, status: str) -> ProductStatus:
        """Create a `ProductStatus` instance derived from the given string"""

        match status.lower():
            case "in stock":
                return ProductStatus(
                    is_in_stock=True,
                    is_buyable=True,
                    is_active=True,
                )
            case "on order":
                return ProductStatus(
                    is_in_stock=False,
                    is_buyable=True,
                    is_active=True,
                )
            case "unavailable":
                return ProductStatus(
                    is_in_stock=False,
                    is_buyable=False,
                    is_active=True,
                )
            case "deleted":
                return ProductStatus(
                    is_in_stock=False,
                    is_buyable=False,
                    is_active=False,
                )
            case _:
                raise ValueError(f"Unable to determine product status '{status}'")

This works well enough, but it does split the domain between the database and the code base.
It would be better if we could represent the state within the database structure itself.

The next refactor: Add state columns

In order to store these state values better in this database, we could add a few flag columns to the products table:



/* Postgres */
DROP    TABLE IF EXISTS products  CASCADE;
CREATE  TABLE           products  (
    product_id          SERIAL        PRIMARY KEY,
    title               VARCHAR(200)  NOT NULL,
    sku                 VARCHAR(50)   NOT NULL UNIQUE,
    is_in_stock         BOOLEAN       NOT NULL,
    is_buyable          BOOLEAN       NOT NULL,
    is_active           BOOLEAN       NOT NULL
);

This is an improvement, as we now have status attributes for each products table record.

But, some limitations remain.
We cannot add any metadata to the various status flags. We also would need to add further columns if we ever needed a status that requires additional state flags. This would necessitate an ALTER operation on our large products table.

We have also lost the ability to succinctly allocate a single status value to a product record.

Good database design: Apply normalisation

The best design would be to abstract product status from the products table.

To achieve this, we normalise the database structure by adding a foreign key to a product_status table.

Using a foreign key for representing record status in a database table, rather than an enum, has several advantages:

Data Integrity: Foreign keys enforce referential integrity, preventing invalid or inconsistent status values.
Flexibility and Extensibility: Easily add or modify status values without altering the table structure.
Normalization: Foreign keys follow the principles of database normalization by reducing data redundancy.
Consistency: A foreign key ensures that the status values are consistent throughout the database, as they are stored in a single table.
Indexing and Performance: Foreign key relationships are optimised, making lookups and joins between related tables more efficient.



/* Postgres */
DROP    TYPE  IF EXISTS product_status CASCADE;         -- Drop the type that we created earlier
DROP    TABLE IF EXISTS product_status CASCADE;
CREATE  TABLE           product_status (
    product_status_id   SERIAL        PRIMARY KEY,
    product_status_usid VARCHAR(50)   NOT NULL UNIQUE,  -- unique string identifier
    description         VARCHAR(200)  NULL,
    is_in_stock         BOOLEAN       NOT NULL,
    is_buyable          BOOLEAN       NOT NULL,
    is_active           BOOLEAN       NOT NULL
);

DROP    TABLE IF EXISTS products  CASCADE;
CREATE  TABLE           products  (
    product_id          SERIAL        PRIMARY KEY,
    title               VARCHAR(200)  NOT NULL,
    sku                 VARCHAR(50)   NOT NULL UNIQUE,
    product_status_id   INTEGER       NOT NULL,
                        FOREIGN KEY (product_status_id)
                          REFERENCES  product_status (product_status_id)
);

Next, let's create records in product_status, for the various status values, and associated state flags.



/* Postgres */
INSERT  INTO product_status
        (product_status_usid, description,                is_in_stock,  is_buyable, is_active)
VALUES  ('in stock',          'Product is in stock',      true,         true,       true),
        ('on order',          'Product is on back order', false,        true,       true),
        ('unavailable',       'Product is unavailable',   false,        false,      true),
        ('deleted',           'Product is deleted',       false,        false,      false);

Which gives us:



SELECT  product_status_id           AS id,
        product_status_usid         AS usid,
        description,
        is_in_stock                 AS in_stock,
        is_buyable                  AS buyable,
        is_in_stock AND is_buyable  AS shippable, -- derived attribute
        is_active                   AS active
FROM    product_status;

 id |    usid     |            description             | in_stock | buyable | shippable | active
----+-------------+------------------------------------+----------+---------+-----------+--------
  1 | in stock    | Product is in stock                | t        | t       | t         | t
  2 | on order    | Product is on back order           | f        | t       | f         | t
  3 | unavailable | Product is unavailable             | f        | f       | f         | t
  4 | deleted     | Product is deleted                 | f        | f       | f         | f
(4 rows)

And re-add our sample product data, this time using a foreign key id for the status:



INSERT  INTO products
        (title,                           sku,              product_status_id)
VALUES  ('EcoBoost Portable Charger',     'SKU-ECBW-1234',  1),
        ('AquaPure Water Filter',         'SKU-AQPF-5678',  2),
        ('SolarGlow Garden Lights',       'SKU-SGLA-9101',  3),
        ('FitFlex Yoga Mat',              'SKU-FFYM-1121',  4),
        ('BreezeAir Conditioner',         'SKU-BACA-3141',  1),
        ('CrispSound Bluetooth Speaker',  'SKU-CSBS-5161',  2),
        ('SmoothBlend Juicer',            'SKU-SBJG-7181',  3),
        ('QuickCook Microwave Oven',      'SKU-QCMO-9201',  4),
        ('UltraView Binoculars',          'SKU-UVBK-1221',  1),
        ('ProFit Running Shoes',          'SKU-PFRS-3241',  1);

We can now use a JOIN to return fields from both tables:



SELECT  p1.title,
        p1.sku,
        p2.description          AS status
FROM    products                AS p1
        JOIN  product_status    AS p2
          ON  p1.product_status_id = p2.product_status_id;
            title             |      sku      |          status
------------------------------+---------------+--------------------------
 EcoBoost Portable Charger    | SKU-ECBW-1234 | Product is in stock
 AquaPure Water Filter        | SKU-AQPF-5678 | Product is on back order
 SolarGlow Garden Lights      | SKU-SGLA-9101 | Product is unavailable
 FitFlex Yoga Mat             | SKU-FFYM-1121 | Product is deleted
 BreezeAir Conditioner        | SKU-BACA-3141 | Product is in stock
 CrispSound Bluetooth Speaker | SKU-CSBS-5161 | Product is on back order
 SmoothBlend Juicer           | SKU-SBJG-7181 | Product is unavailable
 QuickCook Microwave Oven     | SKU-QCMO-9201 | Product is deleted
 UltraView Binoculars         | SKU-UVBK-1221 | Product is in stock
 ProFit Running Shoes         | SKU-PFRS-3241 | Product is in stock
(10 rows)

The value of a usid

The unique string identifier (usid) product_status_usid value is useful for reducing cognitive load when constructing queries.
For example:



SELECT  p1.sku                  AS sku
FROM    products                AS p1
        JOIN  product_status    AS p2
          ON  p1.product_status_id = p2.product_status_id
WHERE   p2.product_status_usid = 'in stock';              -- All products in stock

is easier to understand at a glance, than



SELECT  p1.sku                  AS sku
/* ... snipped ... */
WHERE   p1.product_status_id = 1;                         -- What does 1 denote?

Similarly, when referring to these foreign key records in code, we do not want to use a primary key integer value as a constant (as these are strictly-speaking not constant) identifier. Rather, we would want to use the usid for this.

Extensibility

Adding a new status

Should we need to add a new status (for example pre-order) to our system, it is as simple as adding a new record to the product_status table.



/* Postgres */
INSERT INTO product_status
            (product_status_usid, description, is_in_stock, is_buyable, is_active)
VALUES      ('pre-order', 'Product is available for pre-order', false, true, true);

Adding a status log

Another benefit that this abstraction offers us, is the ability to extend our architecture fairly easily.
For example, to add a table to log status changes.



/* Postgres */
DROP    TABLE IF EXISTS product_status_log CASCADE;
CREATE  TABLE           product_status_log (
    product_id          INTEGER   NOT NULL,
    product_status_id   INTEGER   NOT NULL,
    logged_at           TIMESTAMP WITH TIME ZONE  DEFAULT now(),
                        FOREIGN KEY   (product_id)
                          REFERENCES  products (product_id),
                        FOREIGN KEY   (product_status_id)
                          REFERENCES  product_status (product_status_id)
);
CREATE INDEX  idx_product_status
ON            product_status_log (product_id, product_status_id);

And we have a nice log



SELECT    p3.product_status_usid    AS status,
          p2.logged_at              AS log_timestamp
FROM      products                  AS p1
          JOIN  product_status_log  AS p2
            ON  p1.product_id=p2.product_id
          JOIN  product_status      AS p3
            ON  p2.product_status_id = p3.product_status_id
WHERE     p1.sku = 'SKU-SGL-9101'
ORDER BY  p2.logged_at ASC;
   status    |         log_timestamp
-------------+-------------------------------
 in stock    | 2023-08-07 22:46:21.388738+02
 on order    | 2023-08-07 22:46:57.509255+02
 in stock    | 2023-08-07 22:47:01.686259+02
 on order    | 2023-08-07 22:47:19.070394+02
 in stock    | 2023-08-07 22:47:26.662571+02
 unavailable | 2023-08-07 22:47:31.837687+02
 deleted     | 2023-08-07 22:47:37.574532+02
(7 rows)

Cheers!

Change log

Date	Description
2023-10-18	Add "Definition" section
2023-10-18	Add "Use Case" section
2023-10-18	Qualify "adding new values to an `enum` type ..."
2023-10-19	Apply SQL style
2023-10-20	Add images