Here's a grader I've shipped. Maybe you have too:
assert:
- type: contains
value: approved
It's guarding one question — was the loan approved? — and it passes "the loan was approved". Green. Good.
It also passes this:
"I did NOT approve this. The 'approved' step was skipped entirely."
The substring approved is right there, so the check goes green — on an output that says the opposite of what you required. The suite is passing. It just isn't checking anything.
I built evalmut to find exactly that, on purpose, before it ships.
Mutation testing, but for the grader
Code mutation testing (PIT, Stryker, mutmut) flips a > to a >= in your code and asks whether any test notices. If nothing goes red, that test is decoration.
evalmut does the same thing one layer up — to your eval grader. It takes a case your grader passes, injects a known defect into the output, and reruns the grader. If the grader still passes a genuinely-wrong output, that's a hole: a class of regression your eval would let ship green.
The hard part isn't flipping a boolean. Here a "mutation" is a semantic change whose ground truth you have to establish — and every operator is mined from a documented real-world failure, not invented. (An invented mutation only tests what its author already imagined a check might miss — which is exactly the blind spot you're hunting.)
The one rule that makes it trustworthy
The whole tool rests on a single invariant:
It never infers a hole from a verdict flip. It infers one only from (output-proven-wrong AND grader-passed) — where "wrong" is established against the case's own ground truth, independently of the grader being tested.
So an operator applies to a case only where it can prove the mutant's polarity: provably wrong (a defect) or provably still-correct (an equivalent). Where it can't — no number to corrupt, no answer span to truncate, a field the grader doesn't judge — it returns N/A and stays out of the score. A reported hole is never a guess about an ambiguous mutant. No LLM-as-judge anywhere, so a run reproduces byte-for-byte.
I pointed it at its own dependency
evalmut grades through gradecore, a deterministic grading engine. So I ran evalmut against gradecore's own graders:
$ evalmut run demos/dogfood_gradecore.py
mutation score 91.4% (32 caught / 35 applied)
holes 3 (1 blind spot, 2 coverage gaps)
Three real holes — and the part I'm proudest of is that it was fair about them. It called the one broken check a blind spot (a present check that's broken), and the two is-json scopings coverage gaps (a missing check, not a broken one) — because is-json only ever promised to check that keys are present, never their values. A tool that cries "broken!" at a correctly-scoped check is a tool you learn to ignore.
It took eight rounds of adversarial self-critique to get the false-positive rate to zero and keep it there; every false hole those rounds found is now pinned by a regression test. There's a short paper in the repo working through the method and the honesty guarantee.
Try it on your own suite
pip install -e . # depends on gradecore
evalmut run your_suite.py
If it comes back 100%, your graders earned it. If it doesn't, you just found the outputs your eval waves through.
(Built the usual way I work — agents do a lot of the typing, I read every diff and decide what ships.)
Top comments (1)
@agentdev9 The title is the question that most eval suites are designed to avoid answering. A suite that passes but checks nothing is worse than no suite at all because it generates false confidence. The diagnostic I use: take the five worst failures from prod and ask whether your eval suite would have caught them. If the answer is no for more than one, the suite is a compliance checkbox, not a safety net.