Three weeks ago I asked an uncomfortable question about my own LLM regression suite: if a model quietly got worse in a way I care about, would any check actually go red?
I didn't reason about it. I built a tool that answers it mechanically, the way mutation testing answers it for ordinary code: inject a known defect into the system under test, run the eval suite, and report which checks stayed green. A surviving mutation is a hole in the eval. Not an argument — a hole, with a name and a reproduction.
It's called evalmut, and it's now public:
-
pip install evalmut— the CLI runs against a plain Python suite file - 18 mutation operators, every one provenance-gated: an operator only exists if a real, documented defect exists that it reproduces (mined from production failures and issue trackers — never authored to pad a coverage number)
- Deterministic end to end. No LLM judges anything. Red/green is reproducible.
- The catalog of what it caught — including in suites I trusted — ships in the repo as FINDINGS.md
The part I'd defend in a design review: the tool went through eight rounds of adversarial cold-critique before I trusted it, because a mutation tester that's wrong is worse than none — it hands out false confidence about false confidence. Early rounds found real false positives in the tool itself. By round six, tool-fault false positives on a well-formed suite reached zero and stayed there. An empty suite exits nonzero on purpose: a mutation tester must never report "no holes" on a suite that checked nothing.
I work with AI agents openly — Claude Code wrote much of this under an adversarial loop I operate; the discipline above is how I keep either of us from grading our own homework.
Repo: github.com/egnaro9/evalmut (MIT, tagged v0.1-paper — there's a short paper in /paper if you want the method written up properly)
If you run an eval suite you trust: point this at it before you trust it more. I'd genuinely like to hear what survives.
Top comments (0)