I built a QA agent that audits itself nightly. It has filed 79 findings against its own releases.

Evgenii Menshikov — Sat, 05 Sep 2026 23:44:01 +0000

Most "AI QA agents" audit your repo from scratch every time and end with LGTM. I wanted the opposite: a tester with memory, a failure taxonomy, a verdict contract, and no ability to touch the code it judges.

So in Verdict the model only judges. A stdlib harness measures everything else — timestamps, SHAs, test counts, diff coverage, a signed run history — and a validator refuses any state that claims more than was measured.

What a run looks like

A state file carries the baseline, so every run is a delta:

NEW / STILL_OPEN / RESOLVED / REGRESSED, regressions ranked first, every finding with a stable ID and an age.
Every red test is classified: real defect, stale expectation (with a citation), brittle, environment, or flaky — quarantined with an expiry, not deleted.
Every run ends in one of pass | pass with risks | blocked | fail, and a pass must name what was not tested.
It never edits your code: no Edit tool, a hook that confines writes to the QA root, a strict-mode Bash guard.

Then I pointed it at itself

Every release is audited by the previous one, nightly. It has filed 79 findings against its own harness so far — including defects in the fixes for earlier findings, three times in a row, until the process itself changed: before closing a finding, name the axis the fix did not vary, and test one point on it.

The mutation catalogue that scores whether the test suite defends each rule was itself enumerated wrong twice (from the tests, then from the fix list) before it was enumerated from the code — and only then did it find the four gaps no fix-list could reach.

Everything is published: the eval scores including the misses, the run history, the findings it filed against itself, the accepted risks with their citations.

A stranger run: `pallets/itsdangerous`

To see what a newcomer hits, I ran the published wheel on a library I had never opened, in a fresh clone, solo mode. Eighteen minutes, verdict pass with risks, nine findings, all filed as proven, checkout untouched.

Sev	Finding
Major	Signature verification accepts non-canonical base64: four distinct token strings unsign to the same payload, plus a signature with `@@@@` or trailing `=` appended.
Major	The `max_age` boundary is unpinned — the test steps age 1→11 across `max_age=10` and skips the boundary; injecting `>=` leaves 297/297 green.
Major	`hmac.compare_digest` can be replaced with `==` and the suite stays green: the library's only timing-attack defence has no guard test.
Minor	Unbounded `zlib.decompress` on unverified input via `loads_unsafe`: 39 KB → 30 MB. Held to Minor because `loads()` verifies before decompressing — measured with a call counter, 0 calls on a bad signature.
Minor	`sep in _base64_alphabet` is a substring test, not a set test: `Signer(sep=b"az")` is accepted and 12/2000 legitimately signed values fail their own round-trip.
Minor	Three smaller ones: a docstring that promises "never fails" on a call that raises, a coverage config naming a different project, a test parameter that asserts nothing.

The part people actually pay a tester for is the verified intact list: verification precedes decompression; key rotation revokes (mutant → 3 failures); salts separate contexts (mutant → 13 failures); wire-format golden masters pin signatures across versions; suite deterministic across 17 executions.

And what it asked the maintainers instead of deciding for them: is age == max_age meant to be accepted? Is token-string canonicality a guarantee, or must callers key off the payload?

The one thing that broke was Verdict's own runner, not the library: verdict-run . read . as a project key, declared a completed run lost, and ran it again. Fixed the same evening. A stranger would have hit that before reading a single finding — which is exactly why the run was done.

Why won't it fix the bugs?

Independence. A tester that patches what it judges is grading its own homework. It returns an ordered fix list instead, and the next run verifies each fix by re-injecting the defect and watching the suite fail — not by noting the defect's absence.

Try it

/plugin marketplace add ArtJack/verdict
/plugin install verdict@verdict
/verdict:run

Or headless: uvx --from verdict-qa-mcp verdict-run .

Repo: https://github.com/ArtJack/verdict — MIT, Python 3.9+, stdlib only, no network, no telemetry. If you run it on something and it misses, I want the report: the eval suite publishes its misses, and yours would be the first external one.