Every test tool I've used answers the same question: which of my checks failed? It runs your assertions and tells you which ones went red. Useful — but it can only ever report on tests that exist. It is structurally blind to the test you never wrote.
For LLM systems that blind spot is the expensive one. The regression that reaches a user is rarely "a check I had went red." It's usually "there was no check there at all." And a green suite looks identical in both cases.
Here's the smallest concrete version I can show you.
A suite that's green for the wrong reason
A support-bot prompt with three rules: cite a source, never promise a refund, and reply in English. There's a promptfoo suite for it that asserts the first two — a contains for the citation, a not-contains for refund language. It passes. In CI it's a green check, and you'd move on.
Now delete the "reply in English" line from the prompt and rerun the same suite. The bot's output changes — it can now answer in another language. And every assertion still passes, because nothing in the suite ever looked at the language:
#1 [HIGH] SURVIVED [drop_instruction_lines]
dropped line: "- Always reply in English."
fix: add checks.llm_judge("the reply still follows: - Always reply in English.")
The suite wasn't green because the system was fine. It was green because it wasn't looking. No individual assertion is wrong; the coverage has a hole exactly the shape of the rule nobody thought to test.
Finding the hole instead of guessing at it
You can't grep for a test that isn't there. So muteval comes at it from the other side: it degrades the system on purpose — deletes a rule, weakens a modal, drops a retrieved doc — reruns your existing suite against each degraded version, and watches for the case where the output changed but every check still passed. That case is a "survivor," and a survivor is the tool telling you: there is a behavior here your suite has no opinion about.
That's the whole idea, and it's a different question than any assertion-runner asks. A normal suite tells you which of your tests failed. This tells you which test you're missing — the absence, not the failure. (It ran offline, no API key, in about a second — a deterministic mock model standing in for the bot, so the survivor is the suite's blind spot, not model noise.)
What it is, and isn't
- A survivor is a candidate gap, not a verdict. Some are behaviors you intentionally don't test; you look and decide. muteval surfaces the hole, you rule on whether it matters.
- It's a per-suite diagnostic — "your suite doesn't cover this," not "here's a universal flaw in evaluation."
- It can only degrade the system in ways it knows how to (delete a line, corrupt a doc, …); a behavior no mutation exercises won't surface. It finds the holes near the edits it can make, not every possible hole.
Those limits are real, and it's still the only thing I've found that answers "what am I not testing?" instead of "what did I test that broke?"
The question
Think about your own eval suite for a second: if someone quietly deleted one line from your system prompt tonight, is there a check that would go red before a user noticed? If you're not sure — that uncertainty is the gap, and it's exactly the thing that's invisible until you go looking for it.
Repo (the offline demo above is in it, keyless): https://github.com/AshwinUgale/muteval
Top comments (2)
"Structurally blind to the test you never wrote" is the right framing — coverage tools can only grade checks that exist, so an absence never shows up in any report. Deleting a line and watching nothing go red is a nice inversion: the suite's silence after a deliberate break is the actual signal. The part that resonates most is your caveat that a survivor is a candidate gap, not a verdict. I've watched a team wire mutation survivors straight into their defect backlog and drown within a sprint — an unexercised branch often just means "not exercised by these mutations yet," and "you rule on whether it matters" is what keeps it a diagnostic instead of noise. Deterministic mock judge is a good call too; if the grader itself is stochastic, you chase flakes instead of blind spots. When a survivor turns out to be intentional (untested by design), do you record that anywhere, or does it just resurface on every run?
You've put your finger on the exact thing that decides whether this stays a diagnostic or becomes backlog noise — and the honest answer today is no, it just resurfaces on every run. muteval reports survivors fresh each time; there's no place to record "this one is untested by design." So an intentional survivor comes back every run, which is precisely the failure mode you watched that team drown in — a candidate list with no memory turns into a defect firehose within a sprint.
That's a real gap, and it's the missing piece between "good diagnostic" and "usable in CI over time." What it should be is a suppression baseline: mark a survivor as accepted once, keyed on a stable signature of the mutation (the operator + the specific edit, not the run), so on later runs it drops out of the actionable/new set but is still shown as "accepted (N, by design)" — visible and auditable, never silently muted. Basically what a lint baseline does, or how mutmut lets you tag an equivalent mutant, so you triage each hole exactly once.
One design subtlety I'd want to get right: the signature has to be tied to the prompt text the survivor came from, which means if you later edit that part of the prompt, the "accepted" mutation legitimately re-surfaces as new — and that's correct, not a bug: you changed the thing you'd ruled on, so it deserves another look. The wrong version would be a blanket mute by operator that hides real regressions.
Going on the list — it's the difference between "muteval finds the holes" and "muteval finds the holes and remembers which ones you already decided about." Thanks for the field report; the "drowned within a sprint" detail is exactly the argument for building it.