Automating Your Python Dev Tools With Tox

• Tooling Centralization
  • Tox Configuration
  • Tox Organization
  • Tox Factors
• Conclusion

So far the code for our beginner's project has become fairly stable and there are linting and testing tools to give confidence in basic functionality. It's almost at the level of being ready to release to the public. Before we do this, however, we need to fix one underlying issue to future project development: running all our tools manually. In this article we'll be looking at tox to centralize our tooling runs.

Tooling Centralization

So right now we have the following functionality as part of our python project:

• flake8: generalized linting
• pylint: more in depth linting
• pytest: test suite and test coverage
• sphinx: document generation

This is also the order we want to execute them as well. Running each of these manually every time isn't very efficient. To deal with this issue we'll be using the python tool tox. As tox is a development tool we'll need to install it as a traditional dev dependency with pdm. Along with that, there's actually a python package to make working with tox easier in a pdm project. I'll go ahead and install both:

> pdm add -dG dev tox tox-pdm

Tox Configuration

Now tox handles its configuration via an ini file called tox.ini. Let's go ahead and build this up slowly for each of the tools mentioned above. To start out create the tox.ini file in your project's toplevel directory. Then add the following:

[tox]
env_list = py311

This is declaring an environment that will be run under python 3.11. It's used on the backend for setting up a virtual environment. Next we'll setup the tests:

[tox]
env_list = py311

[testenv]
groups = dev
commands =
    flake8
    pylint --recursive=y .
    pytest --cov=my_pdm_project

We'll get to the docs in a later part as that requires an additional change. As far as the layout goes testenv is the default settings that tox uses for all environments. In this case the default is to install dev group dependencies and run the listed commands. env_list is something we'll look at in a bit but lets us diversify what we want to run. The current setting simply indicates that we wish to run the commands using python 3.11, shorthanded to py311.

Now before actually running anything tox will create a .tox folder when executed that acts much like our .venv folder. So this means we'll need to update flake8 and pylint to ignore it. Open up .flake8 in your project root directory and change the content to:

[flake8]
max-line-length = 99
exclude =
  .venv/*
  .tox/*
  docs/*

Then open up pyproject.toml to update this section to add the ignore as well:

[tool.pylint.MASTER]
ignore-paths = [ "^.venv/.*$", "^.tox/.*$", "^docs/*" ]

Note that I've also added docs/ because sphinx from our last article also uses python which we don't have much control over. Now for the moment of truth:

> pdm run tox
py311: commands[0]> flake8
py311: commands[1]> pylint --recursive=y .

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

py311: commands[2]> pytest --cov=my_pdm_project
<snip>
py311: OK (6.58=setup[3.38]+cmd[0.55,2.11,0.55] seconds)
congratulations :) (6.66 seconds)

I've focused the output but you can see tox is running all the normal commands for us. All of this is condensed into a single easy to run command!

Tox Organization

Now while everything is centralized, there's a few issues that can come up in larger projects:

• Sometimes you just want to run only linting or only tests
• Linting doesn't require the package to be installed, which tox will do every time

Fortunately for us there's a way to deal with it. tox supports the concept of multiple environments. That's why the first item is env_list. It also supports installation groups that we've defined in pyproject.toml (in this case the dev group with all of our dev tools). Up until now I've recommended installing everything that's not for the base code to be installed in the dev group for simplicity purposes. Now that we've come further in the project it's time to organize these for making our tox runs cleaner. Right now everything looks like this (save the versions of packages might be different):

[tool.pdm.dev-dependencies]
dev = [
    "flake8>=6.1.0",
    "pylint>=3.0.1",
    "pytest>=7.4.2",
    "pytest-cov>=4.1.0",
    "requests-mock>=1.11.0",
    "sphinx>=7.2.6",
    "tox>=4.11.4",
    "tox-pdm>=0.7.0",
]

tox and tox-pdm are fine as-is in the dev group. The rest we'll break up into linting, testing, and doc. Simply add each group as group_name = [] with the respective version entries inside. Let's do linting first:

[tool.pdm.dev-dependencies]
dev = [
    "pytest>=7.4.2",
    "pytest-cov>=4.1.0",
    "requests-mock>=1.11.0",
    "sphinx>=7.2.6",
    "tox>=4.11.4",
    "tox-pdm>=0.7.0",
]
lint = [
    "flake8>=6.1.0",
    "pylint>=3.0.1",
]

So here pylint and flake8 are now in a dedicate lint group. Now we'll do the same for testing and docs:

[tool.pdm.dev-dependencies]
dev = [
    "tox>=4.11.4",
    "tox-pdm>=0.7.0",
]
lint = [
    "flake8>=6.1.0",
    "pylint>=3.0.1",
]
testing = [
    "pytest>=7.4.2",
    "pytest-cov>=4.1.0",
    "requests-mock>=1.11.0",
]
docs = [
    "sphinx>=7.2.6",
]

As tox references the pdm.lock file, it will need to have all the groups refreshed that is listed within it. This can be done with:

> pdm lock -G:all

Now in tox.ini it's time to break everything up. We'll get everything we have currently updated first:

[tox]
env_list = lint, test

[testenv:lint]
groups = testing, lint
commands =
  flake8
  pylint --recursive=y .

[testenv:test]
groups = testing
commands =
  pytest --cov=my_pdm_project

The major difference here is that we have a testenv:lint and testenv:test. This is known as a "named environment". It's useful for cases of breaking out specific functionality. env_list will run in the order provided with lint first followed by test. For linting the reason why the testing packages is required is because pylint is running against our tests and as such needs to be able to resolve the modules we're using. Let's try this out with a quick run:

> pdm run tox
lint: install_deps> pdm sync --no-self --group testing --group lint
lint: commands[0]> flake8
lint: commands[1]> pylint --recursive=y .

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

lint: OK ✔ in 6.22 seconds
test: install_deps> pdm sync --no-self --group testing
test: commands[0]> pytest --cov=my_pdm_project
================================================= test session starts =================================================
tests\test_mymath.py .........                                                                                   [100%]

---------- coverage: platform win32, python 3.11.5-final-0 -----------
Name                                                     Stmts   Miss  Cover
----------------------------------------------------------------------------
.tox\test\Lib\site-packages\my_pdm_project\__init__.py       0      0   100%
.tox\test\Lib\site-packages\my_pdm_project\mymath.py        25      0   100%
----------------------------------------------------------------------------
TOTAL                                                       25      0   100%


================================================== 9 passed in 0.17s ==================================================
  lint: OK (6.22=setup[3.17]+cmd[0.55,2.50] seconds)
  test: OK (3.31=setup[2.75]+cmd[0.56] seconds)
  congratulations :) (9.61 seconds)

Again I've reduced the output to showcase the important parts. Since everything is broken up into separate parts, we can even choose to run only linting for example:

> pdm run tox -e lint
lint: install_deps> pdm sync --no-self --group testing --group lint
lint: commands[0]> flake8
lint: commands[1]> pylint --recursive=y .

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

  lint: OK (4.53=setup[0.86]+cmd[0.56,3.11] seconds)
  congratulations :) (4.62 seconds)

This is extremely useful for working on different parts of the development phase where you've passed linting but are trying to just get the tests to pass. Then once the tests are fixed you can run the whole suite to make sure everything as a whole looks fine. For the documentation generation it's slightly similar except we need to enter the docs directory first:

[tox]
env_list = lint, test, docs

[testenv:lint]
groups = testing, lint
commands =
  flake8
  pylint --recursive=y .

[testenv:test]
groups = testing
commands =
  pytest --cov=my_pdm_project

[testenv:docs]
groups = docs
changedir = docs
commands = sphinx-build source/ build/

Sphinx itself is installed thanks to be part of the docs group we setup in pyproject.toml. Running again we can see docs are now being generated:

docs: commands[0] C:\Users\johnsmith\my-pdm-project\docs> sphinx-build source/ build/
Running Sphinx v7.2.6
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
writing output...
building [html]: targets for 2 source files that are out of date
updating environment: 0 added, 0 changed, 0 removed
reading sources...
looking for now-outdated files... none found
preparing documents... done
copying assets... copying static files... done
copying extra files... done
done
writing output... [100%] mymath
generating indices... genindex py-modindex done
writing additional pages... search done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded.

The HTML pages are in build

Another issue here is that while sphinx and pytest need our package installed to work properly, linting is just working against the code itself. This means installation of our package just adds unnecessary time. We can skip the process by using skip_install = true in our lint section:

[testenv:lint]
groups = testing, lint
skip_install = true
commands =
  flake8
  pylint --recursive=y .

Running tox again shows it's no longer installing our package for the linting session:

> pdm run tox
lint: install_deps> pdm sync --no-self --group testing --group lint
lint: commands[0]> flake8
lint: commands[1]> pylint --recursive=y .

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

lint: OK ✔ in 4.59 seconds

Now unlike the previous definition where we had py311 there's nothing here indicating which version of python is used. In this case tox simply uses the one from the current environment (python 3.11 in this case, which is what we created the pdm virtual environment with). It turns out you can be explicit with python versions by utilizing an interesting feature called factors.

Tox Factors

This is somewhat of a fancy way of saying "names separated by hyphens", with a twist when it comes to python versions. I'm going to go ahead and make an isolated test case to showcase this:

[tox]
env_list = py311, py3.12

[testenv]
commands = python --version

Now to go ahead and run this:

> pdm run tox
py311: commands[0]> python --version
Python 3.11.5
py3.12: commands[0]> python --version
Python 3.12.0

So what's happening is that the format pyMajorMinor and pyMajor.Minor are special cases called "default factors" and they map to specific python versions. In this case py311 maps to python 3.11 and py3.12 maps to python 3.12. The tox documentation has the full details on special python interpreter factors. I will note that this only works if both python 3.11 and python 3.12 are installed on your system. Multiple python versions is an interesting topic, though might be somewhat more of an intermediate discussion. I've written about here if you're interested. This isn't just for python versions though, you can even get a bit more advanced by testing against multiple environment combinations. Let's say for example you made a webapp that you want to ensure works against database A and database B:

[tox]
env_list = py311-databaseA, py3.12-databaseB

[testenv]
commands = 
  python --version
  databaseA: python -c 'print("uses databaseA")'
  databaseB: python -c 'print("uses databaseB")'

So let's see what happens:

> pdm run tox
py311-databaseA: commands[0]> python --version
Python 3.11.5
py311-databaseA: commands[1]> python -c "print(\"uses databaseA\")"
uses databaseA
py311-databaseA: OK ✔ in 3.47 seconds

py3.12-databaseB: commands[0]> python --version
Python 3.12.0
py3.12-databaseB: commands[1]> python -c "print(\"uses databaseB\")"
uses databaseB

This is because what's actually happening is:

1. py311-databaseA = factor py311 + factor databaseA
2. py3.12-databaseB = factor py3.12 + factor databaseB

[testenv]
commands = 
  python --version
  databaseA: python -c 'print("uses databaseA")'
  databaseB: python -c 'print("uses databaseB")'

So here we can target specific factors, even if all are not included, to take on specific commands or install specific package dependencies. This makes tox an incredibly powerful tool. Even so our current needs are simple so leaving out the python version factors is fine as we're only testing the version set by pdm. Once you get farther in your python development career you'll be able to better understand how powerful the more advanced usages are.

Conclusion

My original intention was to combine this with uploading your package via twine. Then when I started writing it came to realization that the amount of explanation required would have made it a bit too verbose for my liking. I'd also like to note that I'm currently open for work if you like what you see. To not sound too spammy I'll just say check my dev.to profile for more information. In the next section we'll see the final installment of this series where we upload our python code for others to use.