You have a RAG eval suite. Probably faithfulness and answer-relevancy — the two metrics almost every RAG suite starts with. It's green in CI.
Here's a question that suite can't answer for you: if someone weakened your system prompt next week — inverted "do not invent facts," dropped "cite the source," trimmed half the instructions — would those two metrics go red?
I tested it. On the setup below, they didn't.
The setup
A small RAG assistant, graded by a faithfulness judge and a relevancy judge. Standard stuff. I then degraded the prompt one change at a time — the kind of regression a bad edit or prompt drift actually produces — and reran the same suite against each degraded version. This is mutation testing: break the system on purpose, rerun the evals, and count how many breaks the evals catch. The ones they miss are the coverage gaps.
The result — and why it isn't a gotcha
Across 24 injected prompt regressions, the faithfulness + relevancy suite caught zero. Not the inverted "do not invent facts," not the dropped "cite the source," not deleting half the prompt. The faithfulness judge returned a flat top score on every one.
Before you close the tab: this is not "faithfulness is broken," and it's worth being precise about why. Faithfulness measures whether the answer is grounded in the retrieved context — it does not measure whether the answer is correct, and it has no opinion about your prompt. On an answerable case, where the answer sits in the context, a prompt regression that doesn't change the grounded answer is simply invisible to it. The metric is doing its job; its job just isn't "notice prompt regressions." Competent teams already know this and cover it with labeled/correctness evals. The point is that this surfaces the gap on your specific suite, automatically, and names which regressions would walk through.
One caveat that genuinely matters: if your retrieved documents are your source of truth — the docs can't be wrong — then faithful is correct, and this isn't a gap for you. Read every result as per-suite, never universal.
Closing the gap
The fix wasn't a fancier metric. I added one unanswerable case — a question whose answer is not in the context — plus one check: does the assistant abstain ("if it's not in the context, say you don't know") instead of inventing an answer?
That single addition caught exactly the 6 mutants that break the don't-invent guardrail — the inversion, the two operators that delete that line, and both truncations that chop off the prompt tail where it lives. The mutation score moved from 0/24 to 6/24, and each newly-caught mutant was the right one. The remaining survivors are honest, named gaps — citation, tone — each a concrete "write this eval next."
I reran the same experiment against real DeepEval and real Ragas metric objects (not my reimplementation), to check it wasn't an artifact of one rubric set. Same shape: on answerable cases the reference-free metrics sat at the top of their scale on every mutant. It's the metric class, not the library.
What this is, and isn't
- A survivor is a candidate gap you triage — "could this regression happen, and would it be bad?" Some won't survive that question.
- It's a per-suite diagnostic, not a claim about faithfulness in general.
- The mutations are rule-based edits that model real prompt drift; whether they match the regressions your system actually ships is a separate question I'm still working on, and I'll be honest about it in a later part.
If you want to see the loop with no API key, there's an offline demo (mock model) that finds a gap in about a second; the numbers above are a live gpt-4o-mini run, and the configs are in the repo: https://github.com/AshwinUgale/muteval (FINDINGS.md has the reproduction).
The question I'd actually like answered
If you run RAG evals: what's in your suite beyond faithfulness and relevancy? And if you inverted "do not invent facts" in your prompt tonight — would anything in CI go red before a user noticed? I'd genuinely like to hear what people add to cover this.
Top comments (8)
0/24 to 6/24 is partly a property of the mutation set rather than the suite. The 6 caught mutants are the ones that touch the don't-invent line, which is the line the new abstention case asserts on, so adding ten more operators that chop the same prompt tail would raise the score without a single new eval. A discriminator that separates the two readings, with the suite frozen: add an operator family aimed at something the current checks say nothing about, such as reordering instructions or weakening a numeric threshold in the prompt, and see whether the score falls in proportion to the new operators.
On your question, the thing I would not run without is a control I know must fail — a mutant the suite is supposed to catch — because that is what separates a real 0/24 from a harness that is not scoring at all.
This is the sharpest methodological point the post has gotten, and you're right: the jump is confounded by the operator distribution, not just the suite. All six caught mutants touch the don't-invent line the abstention case asserts on, so ten more tail-chopping operators would inflate the score with no new eval. Which means a mutation score is only interpretable against a fixed operator set — comparing two suites, or celebrating a jump, is meaningless unless the operators are held constant, and I haven't been careful enough about that.
Your discriminator is exactly the experiment: freeze the suite, add an operator family the current checks say nothing about, and watch whether the score falls in proportion. Reordering is already an operator; weakening a numeric threshold in the prompt isn't, and it's a good one to add precisely because nothing in a faithfulness/relevancy suite speaks to it — the score should fall toward zero as I add it, and if it doesn't, the number was an artifact of operator mix all along. Honestly this is the strongest argument I've seen for reporting coverage per behavior class instead of one aggregate — "covers instruction-drop, not threshold-weakening" survives operator reweighting in a way a single percentage never does.
And the control you wouldn't run without — a mutant the suite is supposed to catch — is a real gap. The baseline gate is a negative control (the original must pass); there's no positive control, and you've named exactly why it matters: it's what separates a genuine 0/24 from a harness that isn't scoring at all. A canary mutant any competent suite must kill, that fails the run if it survives — I'm adding that. This is the most useful critique the post's had.
The mutation family has a second axis, and it sharpens the caveat you already wrote. Everything in the current suite mutates the prompt side, and the proposed additions reorder or threshold the retrieval, but the retrieved context itself is never touched. Mutate that: swap an entity inside a chunk, stale a date, flip a number, and count catches. For faithfulness the score is knowable before the run, zero of N by construction, because an answer faithful to a corrupted chunk is exactly what the metric rewards. That is not a broken judge, it is the cleanest possible statement of where the metric class ends, which is the framing you were careful to keep for the prompt-side result. And it forces a useful conclusion: catches on context mutations can only come from checks that compare the answer or the chunk against something outside the context, a live-state probe, a provenance check, a contradiction test against a record known to be current. The abstention case caught six of twenty four because unanswerable is the one condition where groundedness and correctness point the same way. Context corruption is the condition where they point in opposite directions, which makes it the strongest mutation operator a suite can carry.
Yes — and this is the sharper knife. Part 1 only showed the prompt side, but the tool already carries context operators: drop a retrieved doc, corrupt one (it flips the first number or a polarity verb in a chunk — your "flip a number" exactly), swap one for an irrelevant hit. There's an offline demo where corrupting a chunk flips the grounded answer, faithfulness happily passes the now-wrong-but-grounded response, and the weak eval misses it → a high-severity survivor. So your by-construction prediction isn't just knowable, it's observed: on context corruption a reference-free faithfulness metric scores ~0/N catches, because "faithful to the corrupted chunk" is precisely what it rewards.
And you've stated the boundary better than I did. The prompt-side result was groundedness and correctness being orthogonal; context corruption is where they point in opposite directions — the metric doesn't just fail to help, it actively rewards the wrong answer. That forces your conclusion: a catch here can only come from a reference outside the context — provenance, a live-state probe, a contradiction test against a record known to be current. No amount of grounding-based scoring reaches it.
Your line about the abstention case — unanswerable being the one condition where groundedness and correctness align, which is why it caught six — is the framing I wish I'd led with. Stealing it: the metric is safe exactly where grounded = correct, blind exactly where they diverge, and context corruption is the maximal-divergence case. That's the whole reference-free story in one sentence.
Zero out of 24 is a striking number, and the reason it is not a scandal is worth restating: with good retrieved context, a competent model produces a grounded answer whether or not you told it to, so the instruction you deleted was never what was carrying the behaviour on those inputs. The suite is not blind, the test set just never contains a case where the instruction matters. That suggests the fix is in the fixtures more than the metrics - to catch a dropped "cite the source" you need at least one case where citations are required and absent, and to catch an inverted "do not invent facts" you need a question whose answer is genuinely not in the corpus. Both are unpleasant to author, which is exactly why most suites do not have them. Mutation testing against the prompt is a good way to find that out cheaply, and I would run the same trick against the retrieval config, since dropping the reranker or halving top-k tends to sail through the same green suite.
This is the clearest statement of it: the suite isn't blind, the instruction just wasn't carrying the behavior on those inputs, so nothing in the fixtures ever put it under load. Grounded ≠ instructed is why deleting the line changed nothing.
And "the fix lives in the fixtures, not the metrics" is really what the survivor is telling you — your test set has no case where that instruction is load-bearing. The abstention check only works because it comes with the fixture that makes it bite (a question whose answer genuinely isn't in the corpus); the eval alone catches nothing. Fixture + eval, together. And the reason those fixtures are rare is exactly that they're unpleasant to author — which is the point of doing this at all: it doesn't write the adversarial case for you, but it turns "you should probably have one" into "here's the specific regression your current fixtures can't see."
On retrieval — agreed, and muteval already carries the content side: operators that drop, corrupt, or swap a retrieved doc, graded against the mutated context, so a faithfulness metric passes a confidently-wrong-but-grounded answer → survivor. That's the stronger, less-gimmicky version of the 0/24. The honest gap is the level you're pointing at: those mutate the retrieved content (the effect of worse retrieval), whereas reranker-off / top-k-halved mutate the retrieval config so a different set gets retrieved organically — muteval doesn't expose those knobs today. Same green-suite blindness is waiting there, and it's the right next rung.
The failure that faithfulness doesn't catch in my experience is confident answers about the wrong entity. Our retriever sometimes pulls chunks about a different company that just shares a name pattern, and then faithfulness scores fine because the answer is faithful to those chunks. I spent a few weeks last year adding a probe where I deliberately ask about an entity that has a common name doppelganger and then verify which one the model is actually talking about. Most eval suites I've seen don't have anything like that.
This is exactly the thought process behind muteval, but generalizing it is where the real challenge lies, because the faults are many times very unique to systems, I would love to know what you think about muteval if you get a change to try it out, or would love to discuss ways to make the mutations cover more use cases