Mutation testing, but for LLM evals — early experiment, would love feedback

I write evals for LLM features, wire them into CI, and move on. But a passing eval suite has a blind spot: it tells you nothing about whether it would actually fail if the model quietly got worse. Green isn't the same as good.

So I built muteval to measure that directly. It borrows mutation testing from software engineering — you deliberately break the thing under test and check whether your tests catch it. muteval degrades the system (weakens a prompt rule, drops a retrieved doc, swaps in a weaker model), reruns your existing eval suite against each degraded version, and reports what fraction of those injected regressions your evals caught. The ones they miss are concrete coverage gaps.

Mutation score: 33% (2/6 caught)
SURVIVED: deleted "if the answer isn't in the context, say you don't know"

A bit on how it works. It ships around 18 of these mutations, each modeling a way a real system quietly degrades: softening a "must" to a "should", flipping a "do not", dropping or corrupting a retrieved document, shuffling context, swapping in a cheaper model, or breaking a tool's output for agent setups. Before mutating anything it confirms your suite passes on the original system — if it doesn't, there's nothing meaningful to measure — then reruns your evals against each mutation, ranks whatever survives by severity, and suggests an eval that would close each gap.

Does it actually find anything real? I ran it on Vectara's open-rag-eval. Its citation check caught mutations that made the model stop citing its sources — but it completely missed one that removed the rule telling the model to say "I don't know" when the answer isn't in the context. The model kept citing sources while being free to make things up, and a citation check simply can't catch that. Once I added a check for that behavior, it caught it.

It's pure Python, no required dependencies, and works with your deepeval/RAGAS/promptfoo metrics or its own checks:

pip install muteval
muteval init      # scaffold a config
muteval check     # validate it, then muteval run

Where I think this goes next is a tool that rates an eval suite across several angles and suggests the evals you're missing — but I'd like to hear how other people approach it first. If you write LLM evals: how do you know they're any good?

Repo (Apache-2.0): https://github.com/AshwinUgale/muteval