Looking Beyond Nox

mwchase — Fri, 09 Apr 2021 00:50:33 +0000

I've gone through what I assume is a fairly typical journey between different ways to run tasks related to a Python project: from hand-rolled shell and Python scripts, to tox, to Nox. To be honest, I'm not sure how many people have switched from tox to Nox, but now that I've used Nox, I don't ever want to edit a tox.ini file again.

All that said, I think I'm starting to push past what is reasonable to do with Nox.

In one of my private hobby projects, I have a single host repo that has (currently) five package directories nested inside it. I've got a noxfile set up that will:

Prepare for the coverage runs by deleting the previous coverage data.
Run syntax checks, including isort through flake8, and generate a report for each sub-project.
Run Mypy against each project, generating coverage (in html) and error (in junit xml) reports.
Run pytest against each project, generating a junit xml report.
Run pytest, under coverage, and generate a coverage data file for each project, in addition to a separate junit xml report.
Run pytest, under pyinstrument, and generate a webpage that presents performance data, in addition to a separate junit xml report.
Merge the junit results from each kind of pytest run, creating three merged junit files, each containing data from all of the sub-projects.
Convert the various junit xml reports to html via junit2html
Merge the per-project coverage files into one
(Currently disabled) winnow down the coverage report to ignore certain coverage contexts that I have decided "shouldn't count". (This is implemented through a package I wrote that, while public, is probably not ready for general use, even among people who want what it provides.)
Generate an HTML report of the coverage.
Report if the coverage metric is below a defined threshold.
Build executable python archives for each endpoint defined by a project, using shiv.

This is... a lot, but my hobby code tends not to do much I/O, so, just for the sake of my own curiosity, let's do a time nox. Running that while writing this post and having YouTube playing in the background gets me an execution time of... not instant... Long enough that I should probably tab away from this text box...

________________________________________________________
Executed in  295.79 secs    fish           external
   usr time  251.25 secs  494.00 micros  251.25 secs
   sys time   29.97 secs  154.00 micros   29.97 secs

Oof. So, you can see why I'd be interested in shaving time off of this.

There are currently GitHub issues against Nox for adding parallelism, but I'm not sure if they're going anywhere, and they're prioritizing stuff like "having legible input when you're running multiple commands at once", which is a non-issue for me; due to the large volume of data that Nox was already generating, I rewrote my sessions to direct all relevant output into webpages. Now, I just need to read the summary at the end to see which commands had a non-zero exit code, and I can open my web browser to see the data produced by subsequent commands acting on the output of the failed command.

(Remember that last sentence, it's important!)

So, if I want to port all of this to a tool that can figure out which things to run in parallel, and run them, what are my options?

I've mostly been looking at Python-based stuff, but I'm willing to go in all kinds of directions. Here are my general requirements:

Parallel execution
Feasible to work with Python virtualenvs
Possible to specify that specific commands should allow execution to continue if they exit with a specific non-zero exit code, but be called out as failing in the final summary. (This code is usually 1, but it's 2 for coverage report, so I need to be able to specify this per-command.)
Any incremental build features can handle conditions like "something changed in subproject A that is a dependency of subproject B, so both A and B need to rerun everything (except flake8 against B, I guess)". ("No incremental build features" would entirely satisfy this requirement.)
A nice-to-have would be that ctrl-c propagates into the processes and interrupts them instead of just stop execution outright. I don't know how feasible/widespread this is, I just know that drone exec doesn't do this, and seemingly isn't planned to.
The tool should not assume the use of a specific VCS, because I'm not using any of the popular ones.

I've looked at a few different projects that handle parallel execution, and I haven't found anything saying that any of them support the third feature given above, and I'm not sure I'd trust any to handle the fourth, without testing it myself.

So far, I've looked, relatively superficially, at Ninja, doit, Bazel, and SCons.
It looks to me like Ninja and doit won't handle the third point above without requiring something really strange, and if Bazel or SCons documents things one way or the other, I haven't found it yet.

So, are any of those tools a better fit than I realize? Are there other tools you'd recommend?

DEV Community: mwchase

Looking Beyond Nox