This week I launched evalmut — mutation testing for eval suites. Its weakness, stated in the paper: I chose the mutations, so a suite author can call them unrepresentative. Fair.
So I built the version of the argument you can't dismiss: reference-fleet, six deterministic models, each broken in exactly one documented way, at a stated seeded rate. Not trained — constructed. The defect count over a fixed request set is a constant, not a sample, and the certificate is a test file you can run:
- citation-hallucinator — fabricates well-formed, on-topic, nonexistent references (the Mata v. Avianca failure)
- constraint-dropper — honors instructions 1..N-1, silently drops the last
- refuse-then-comply — refusal preamble, full compliance after
- tool-arg-swapper — schema-valid tool call, first two argument values transposed; executes cleanly into the wrong place
- sycophancy-flip — endorses whatever the user asserts
- stale-cutoff — confidently unhedged past its knowledge boundary
The framing is metrology's certified reference materials: NIST ships samples of known composition so you can prove your instrument reads correctly. This is that, for eval suites.
Then I pointed real suites at the fleet. The audit protocol is paired: a defect counts as detected only if the suite fails the defective response AND passes its clean twin — a suite that fails both isn't detecting anything.
The board's first result, measured (paired protocol, zero false alarms):
- naive-contains — the "did it answer, is it on topic, did it not refuse" suite most teams start with — detects 1 of 6 defect classes (refuse-then-comply). Citation fabrication, dropped constraints, swapped tool arguments, sycophancy, and confident staleness all pass it invisibly, at detection rate 0.000, n=200 each.
- A diligent gradecore suite and a real promptfoo run with docs-style asserts both detect 6 of 6 at rate 1.000 — with a disclosure that matters: I wrote those suites knowing the task's ground truth. Their perfect rows prove the ceiling is REACHABLE (every fleet defect is catchable by a suite that checks everything checkable), not that typical suites reach it.
The gap between those two rows is the product: the distance between the suite you have and the suite the task admits.
One honesty note, because the whole project is about honesty notes: what's audited is a suite archetype executed by its real engine — a real promptfoo run, real gradecore graders — for this task format. No named public benchmark's own dataset is being scored. The claim is narrower and therefore checkable: "a suite built this way misses this defect class at this rate." And the reproducibility claim is CI-enforced: the runner refuses to stamp results from a dirty tree, and a CI job re-runs the entire audit on every push and goes red unless the published numbers reproduce byte-for-byte. That gate caught three real problems before this post — including a provenance hole in my own board that would have inverted 12 of 18 rows.
Board: https://egnaro9.github.io/reference-fleet/
Repo: https://github.com/egnaro9/reference-fleet — python audit/run_audit.py reproduces every number.
If you maintain an eval suite: which of the six would yours catch? That's an empirical question now.
Top comments (0)