Last week I wrote about tracelint, a linter that catches structural bugs in agent traces — the classic one being an agent that calls charge_card, gets a failure back, and just... keeps going and tells the customer their order shipped.
The response was better than I expected, and one question kept coming up in different forms: "Okay — but wouldn't my eval suite have caught that anyway?"
Fair question. So I checked. On the case I'll show you, it didn't. The suite was green. The charge was declined. The agent said "payment successful," and every eval passed.
This post is about the tool I used to find that — muteval — and what happened when I pointed it at the exact failure tracelint was built for.
The blind spot in a passing eval suite
Here's the thing about a green eval run: it tells you your system passed today's tests. It tells you nothing about whether those tests would notice if the system silently got worse.
That's not hypothetical. Your contains("refund") check passes whether or not the model still follows the refund rule someone deleted last week. Your faithfulness judge passes a confidently-wrong answer as long as it's grounded in the context. The eval is green, and the regression walks straight through it.
In normal software we have a name for "are my tests actually any good?" — mutation testing. You deliberately inject bugs (mutants) into the code, rerun the test suite, and measure how many the tests catch. mutmut and Stryker do this. muteval does it for eval suites — except the "code" it mutates is the system under test: the prompt, the retrieved context, the tool outputs, the model.
Degrade the system → rerun your existing evals → report the percentage of injected regressions they caught. The ones they miss are survivors — concrete coverage gaps.
The declined charge
Back to the agent. Here's a deliberately naive payments agent: it calls charge_card and reports success. Its eval suite has one realistic semantic check — does the reply confirm the charge went through?
muteval has an operator called deny_tool_output. It takes a tool's result and turns it into the nastiest kind of failure: a domain failure returned as transport success — an HTTP 200 whose body says {"status": "declined"}. Structured-error handling never fires, because nothing technically errored. The agent proceeds. The final answer still reads "Your payment was successful."
Rerun the eval suite against that mutant:
[HIGH] SURVIVED [deny_tool_output]
tool output #1 returned a domain failure (HTTP 200 + status:declined)
Mutation score: 0%. The suite caught nothing. The card was declined, the agent lied, and the contains("successful") check passed — because the answer does still say "successful." That's a survivor, and it's exactly the kind of blind spot you'd never see from a green CI run.
(To be clear about the number: 0% here means of the regressions I injected, your evals caught none of them. Not "your system is broken" — "your tests wouldn't have noticed.")
Closing the gap — where the two tools meet
Here's the part I think is actually interesting.
The thing that catches "declined charge reported as success" isn't a smarter output judge — the output looks fine. It's a structural check on the trace. Which is exactly what tracelint does.
So I wired tracelint in as one of muteval's evals. tracelint reads the agent's trace, and if the tool declares what failure looks like (failure_when: {"pointer": "/status", "in": ["declined"]}), it flags the declined charge deterministically — no judge, no key.
Add that one check and rerun:
verdicts on the declined-charge mutant:
semantic eval (confirms "successful") -> PASS (misses it)
tracelint (declared failure contract) -> FAIL (kills it)
mutation score: semantic eval alone 0% -> + tracelint 100%
muteval didn't just tell me the gap existed — it let me prove the fix closes it: the mutant that survived the semantic suite dies the moment the structural check is in the suite, and the baseline stays green. That's the whole loop — a survivor names a missing eval, you add it, the score goes up.
A tool fault is a mutant; a trace-lint rule is an eval. Mutation testing on one side, deterministic trace checking on the other, and they compose.
What this does NOT tell you (the part I care about most)
A diagnostic you can't trust is worse than none, so:
- A survivor is a candidate, not a verdict. muteval tells you your eval missed an injected change. Whether that change could really happen, and would really be bad, is a human call. Some survivors won't survive that question.
- Mutation coverage is not validity. A suite can be highly sensitive to mutations and still be wrong — sensitivity isn't the same as agreeing with ground truth or a human. That needs labels; no label-free tool gets you there.
- The mutations are rule-based, synthetic edits. They model real regressions; they aren't identical to them. Whether mutants predict the failures your system actually experiences is the open question I'm still chasing.
- It's a per-suite diagnostic, not a universal flaw-finder. It tells you where your suite has a hole — not a new universal truth about evals.
- It needs a re-runnable system. muteval degrades the system and needs a fresh output per mutant. A frozen CSV of outputs can't be mutated.
muteval fails closed on all of this: a red baseline, too few mutants, or too many errored mutants means it refuses to emit a score rather than hand you a confident-but-meaningless number.
Try it
There's a demo that runs with no key — a mock model, so you can watch the mutate → survive → fix loop end to end in about a minute. (Running it on your own suite calls your model + a key, like any eval.)
git clone https://github.com/AshwinUgale/muteval && cd muteval
pip install -e ".[tracelint]"
python examples/agent_tool_fault/run_demo.py # keyless, ~1s
Repo: https://github.com/AshwinUgale/muteval (Apache-2.0) · tracelint: https://github.com/AshwinUgale/tracelint (MIT).
And the genuine ask, same as last time: if you write evals for agents or RAG, I'd like to know what your suite would miss. What's the regression you're most afraid of shipping — and if you injected it, would it survive your evals? Issues, replies, and war stories all welcome.
Top comments (2)
Really enjoyed this one. The declined-charge example is spot-on — the tool call is "technically fine," and the customer has already been lied to.
Quick intro: we're building a security gate for AI assistant plugins (pre-install static scanning + runtime monitoring), and your loop turns out to fit us almost perfectly — our rule table is basically our eval suite, but no one has ever systematically tested it. We're now building a corpus along the same lines: hand-crafted escape samples as mutants, each bound to the rule that should kill it, and every survivor gets triaged one of three ways (add a rule / add a runtime check / record it as a known boundary).
Also answering your call: the regression we fear most is a plugin smuggling API keys out through perfectly ordinary-looking network requests — structurally legitimate, semantically theft. My suspicion is this class of thing sails right through most scanners out there, including ours.
One practical thing I'd love to compare notes on: your mutation operators are rule-based and auto-generated. On our side, auto-generated variants rarely resembled real attacks, so we ended up crafting samples mostly by hand. Has mutation over prompts / tool outputs worked well in practice for you? Or did you hit the same wall of "synthetic mutants don't look like real threats"?
This is a great fit — "our rule table is basically our eval suite, but no one's ever systematically tested it" is the gap stated better than I managed. And it holds all the way down: your three-way triage (add a rule / add a runtime check / record as a known boundary) is exactly what I call survivor triage (write an eval / it's a real gap / equivalent-or-intentional mutant). We converged from different ends.
On your question — yes, I hit that wall, and I'd trust your instinct to hand-craft. The split I've landed on:
Auto-generated rule-based mutants are decent proxies for accidental degradation — prompt drift, a trimmed instruction, a retriever dropping a doc, a flaky tool. Real regressions of that kind often are mundane edits, so a weakened modal or a dropped line resembles what actually happens. That's where auto-generation earns its keep.
They're bad proxies for adversarial degradation. A real attack is structurally legitimate and semantically hostile — your key-exfil case exactly. A generator can't synthesize "looks legitimate, is theft," because that property is the whole point. Security is adversarial, so hand-crafting samples bound to the rule that should kill them isn't a compromise — it's the correct method.
Your exfil case is the same shape as the declined charge: transport-legitimate, semantically wrong. And notice what caught the charge — not a smarter check on the request, but the tool declaring what failure looks like (tracelint calls it a failure_when contract). The parallel might be declaring, per plugin, what legitimate egress looks like — allowed destinations, data shapes — so a structurally-fine but out-of-contract request trips a rule. It won't catch novel exfil you never anticipated, but it flips "ordinary-looking" from invisible to a declared-contract violation.
Where I think we're both headed past the synthetic-mutant wall: sourcing mutants from reality instead of generating them — hand-crafted escape samples, disclosed attacks, real incidents — each bound to the rule that should kill it. Auto-generation buys coverage of the mundane; real samples buy the threats. That "bind each sample to the rule that should kill it" discipline is the part most people skip, and it's what makes a survivor mean something. Would genuinely love to compare corpus notes as you build yours.