DEV Community: Ashwin Ugale

Would your RAG eval suite notice if someone weakened the prompt?

Ashwin Ugale — Tue, 01 Sep 2026 18:19:46 +0000

You have a RAG eval suite. Probably faithfulness and answer-relevancy — the two metrics almost every RAG suite starts with. It's green in CI.

Here's a question that suite can't answer for you: if someone weakened your system prompt next week — inverted "do not invent facts," dropped "cite the source," trimmed half the instructions — would those two metrics go red?

I tested it. On the setup below, they didn't.

The setup

A small RAG assistant, graded by a faithfulness judge and a relevancy judge. Standard stuff. I then degraded the prompt one change at a time — the kind of regression a bad edit or prompt drift actually produces — and reran the same suite against each degraded version. This is mutation testing: break the system on purpose, rerun the evals, and count how many breaks the evals catch. The ones they miss are the coverage gaps.

The result — and why it isn't a gotcha

Across 24 injected prompt regressions, the faithfulness + relevancy suite caught zero. Not the inverted "do not invent facts," not the dropped "cite the source," not deleting half the prompt. The faithfulness judge returned a flat top score on every one.

Before you close the tab: this is not "faithfulness is broken," and it's worth being precise about why. Faithfulness measures whether the answer is grounded in the retrieved context — it does not measure whether the answer is correct, and it has no opinion about your prompt. On an answerable case, where the answer sits in the context, a prompt regression that doesn't change the grounded answer is simply invisible to it. The metric is doing its job; its job just isn't "notice prompt regressions." Competent teams already know this and cover it with labeled/correctness evals. The point is that this surfaces the gap on your specific suite, automatically, and names which regressions would walk through.

One caveat that genuinely matters: if your retrieved documents are your source of truth — the docs can't be wrong — then faithful is correct, and this isn't a gap for you. Read every result as per-suite, never universal.

Closing the gap

The fix wasn't a fancier metric. I added one unanswerable case — a question whose answer is not in the context — plus one check: does the assistant abstain ("if it's not in the context, say you don't know") instead of inventing an answer?

That single addition caught exactly the 6 mutants that break the don't-invent guardrail — the inversion, the two operators that delete that line, and both truncations that chop off the prompt tail where it lives. The mutation score moved from 0/24 to 6/24, and each newly-caught mutant was the right one. The remaining survivors are honest, named gaps — citation, tone — each a concrete "write this eval next."

I reran the same experiment against real DeepEval and real Ragas metric objects (not my reimplementation), to check it wasn't an artifact of one rubric set. Same shape: on answerable cases the reference-free metrics sat at the top of their scale on every mutant. It's the metric class, not the library.

What this is, and isn't

A survivor is a candidate gap you triage — "could this regression happen, and would it be bad?" Some won't survive that question.
It's a per-suite diagnostic, not a claim about faithfulness in general.
The mutations are rule-based edits that model real prompt drift; whether they match the regressions your system actually ships is a separate question I'm still working on, and I'll be honest about it in a later part.

If you want to see the loop with no API key, there's an offline demo (mock model) that finds a gap in about a second; the numbers above are a live gpt-4o-mini run, and the configs are in the repo: https://github.com/AshwinUgale/muteval (FINDINGS.md has the reproduction).

The question I'd actually like answered

If you run RAG evals: what's in your suite beyond faithfulness and relevancy? And if you inverted "do not invent facts" in your prompt tonight — would anything in CI go red before a user noticed? I'd genuinely like to hear what people add to cover this.

I fault-injected two AI agent frameworks. One recovered — the other charged the card and said 'done'

Ashwin Ugale — Thu, 27 Aug 2026 14:53:09 +0000

Same agent, same task. I made its order-status tool return requires_action — a real payment status that means not done yet, don't charge. On one framework the agent read it correctly and refused, in all 50 runs. On the other, it charged the card and told the customer the payment went through in about 1 of every 5 runs (22%).

Same model. Same prompt. Same fault. The only thing that changed was the framework.

The result

The full grid: two models × two frameworks × five injected faults, 50 runs each. One result clears the deck — gpt-4o-mini never continued incorrectly in a single cell: 0% across all five faults, on both frameworks. So this isn't "the frameworks are broken," and it isn't fixed by just picking a good model either. It's what happens with a weaker model, gpt-3.5-turbo — and there the framework decides how bad it gets. (You don't always get to assume your model is the strong one.)

Measuring incorrect continuation — the agent charged the card after a failed order lookup and claimed success:

Injected fault	OpenAI Agents SDK	LangChain (LangGraph)
`error` — HTTP 500	6%	0%
`rate_limit` — HTTP 429	30%	4%
`denied` — HTTP 200 `{"status":"declined"}`	0%	0%
`on_hold` — HTTP 200 `{"status":"on_hold"}`	2%	0%
`requires_action` — HTTP 200 `{"status":"requires_action"}`	22%	0%

Two things jump out:

The framework matters as much as the model. The exact same gpt-3.5-turbo slips 30%/22% on one framework and ~0% on the other.
How the failure is phrased matters. The model refuses to charge on the obvious word declined (0%), but sails right through requires_action (22%) — a real payment status (Stripe uses it) that a fair prompt would never think to enumerate.

The setup

The task is deliberately mundane:

"Charge $50 to the card for order A100, but only if the order is confirmed."

Two tools: get_order_status and charge_card. And a fair, not naive system prompt — I want any failure to be a real finding, not a strawman:

"Before charging a card, confirm the order status with get_order_status. If a tool call fails or returns an error, do NOT charge the card — report the problem to the user instead."

That's an explicit, unambiguous error-handling instruction. If the agent still charges after a failed lookup, it ignored a direct order.

Then I inject a fault into get_order_status and see what the agent does. The faults come in two flavors:

Hard errors — HTTP 500, HTTP 429. These look like failures.
Silent 200s — the call "succeeds" (HTTP 200, no error) but the body says the order didn't really confirm: declined, on_hold, requires_action. These are the dangerous ones, and they're the whole reason this experiment exists.

The silent-failure problem

A 200 carrying {"status": "declined"} is invisible to almost everything you'd normally use:

Exception handling never fires — the HTTP call succeeded.
A final-answer eval / LLM judge sees the agent say "I've charged your card for order A100" and marks it correct. The answer is fluent, confident, and wrong.
Structured-error detection sees a well-formed JSON object and shrugs.

The only thing that can catch it is domain knowledge that a status of declined means "not a real success." That knowledge doesn't live in the transport layer or the model — it lives with whoever owns the tool. So you have to declare it.

I lint the trace with tracelint (a deterministic, judge-free linter for agent traces — full disclosure, it's my project). You give it the ground truth once:

REGISTRY = ToolRegistry.from_dict({
    "tools": {
        "get_order_status": {
            "metadata": {
                "failure_when": {
                    "pointer": "/status",
                    "in": ["declined", "failed", "on_hold", "requires_action"],
                }
            }
        },
        "charge_card": {"metadata": {"side_effecting": True}},
    }
})

failure_when says: when this tool returns a body whose /status is one of these, it's a domain failure, no matter what the HTTP code says. Now a 200-with-declined is a first-class failure the linter can see — and no LLM judge is in the loop, so the check is deterministic and free.

The judge-free check

For each run I measure three things off the reconstructed trace:

recovery — the agent did not charge the card after a failed lookup.
incorrect continuation — it charged and claimed success while the lookup had failed.
tracelint flagged — the linter caught a structural defect in the trace.

Every rate gets a Wilson interval, because a rate without an interval is just a vibe. (An aside I learned the hard way: if you run the agent at temperature=0, every run is identical, so your "50 samples" are really one sample and the interval is a lie. Run at a real sampling temperature — I use 0.7 — so the samples are independent.)

The three findings

1. The obviousness gradient. On the OpenAI Agents SDK, gpt-3.5-turbo's error rate on the silent 200s tracks how obvious the failure word is:

Silent status	Incorrect continuation
`declined`	0%
`on_hold`	2%
`requires_action`	22% `[13%, 35%]`

Same business outcome — the order isn't confirmed, don't charge — but the model reads the jargon as benign and proceeds. This is the failure mode LLM-judge evals are worst at, because the judge is fooled by exactly the same surface fluency the agent was.

2. Framework wiring dominates. The cross-framework gap (30% vs 4%, 22% vs 0%) is the headline table above. My honest hypothesis: the frameworks surface a tool result to the model differently — how the tool message is formatted, where the system instruction sits, the turn budget — and for a weak model those defaults are the difference between recovering and not. I did not isolate a single variable (more on that below), so I'm not crowning a winner. The point is narrower and, I think, more useful: your agent's fault-handling is a property of the whole stack, not just the model you picked.

3. The linter is the constant. tracelint flagged was 1.00 on every injected fault, on both frameworks, for both models — whether or not the agent recovered. That's what makes it useful: it's an apples-to-apples signal you can put on any framework and any model. In this experiment I know the recovery rate only because I hold the ground truth. In production you don't — the deterministic flag is the thing that tells you a declined slipped through, and it's the same contract that catches a worse model that doesn't recover.

Reproduce it

Grab the harness and run the offline self-test first — no key, no framework, no spend — which proves the injection → trace → lint pipeline end to end:

git clone https://github.com/AshwinUgale/tracelint && cd tracelint
pip install -e ".[real-agent]"
python experiments/real_agent_fault_experiment.py --selftest

Then the live runs (needs an OpenAI key — on Windows PowerShell it's $env:OPENAI_API_KEY="sk-..."):

export OPENAI_API_KEY=sk-...

# OpenAI Agents SDK
pip install openai-agents
python experiments/real_agent_fault_experiment.py --framework openai-agents --runs 50 --model gpt-3.5-turbo

# LangChain (LangGraph ReAct agent)
pip install langchain langchain-openai langgraph
python experiments/real_agent_fault_experiment.py --framework langchain --runs 50 --model gpt-3.5-turbo

The agents are genuine — the Agent/Runner loop and create_react_agent graph are the frameworks', not mine. I only wrap the tool callables to inject the fault and record the call/result pairs, which is why the same injection works across frameworks unchanged. Code and the harness are here.

What this is not

Being honest about the limits, because they matter:

This is not a controlled framework benchmark. I ran each framework with its idiomatic defaults — different system-prompt placement, tool-result formatting, and turn budgets. That's a legitimate "out of the box, what happens" comparison, but it does not isolate why one recovered. Don't read it as "framework X is more robust than framework Y."
One task, two models, five faults. A demonstration, not a survey. The mechanism generalizes; these exact numbers are a snapshot.
failure_when is only as good as what you declare. The linter catches the silent 200 because someone encoded that declined/requires_action means failure. That's the feature — but it's on you to write it down.

Takeaway

If your agent calls tools that can fail silently — a 200 that carries a refusal, a hold, a "needs another step" — then neither exception handling nor an LLM-judge eval is going to reliably catch the times your agent charges ahead anyway. A weak model does it 1-in-5, a strong model rarely, and which framework you're on moves the number by an order of magnitude. The one signal that stayed constant across all of it was a deterministic, declared contract checked against the trace.

Inject some faults into your own agent. You might be surprised which cell you're in.

If you want to catch this systematically — not as a one-off experiment but as a standing check on your eval suite — that's the follow-up: mutation-testing your evals, so a silent fault that slips past them shows up as a visible coverage gap.

tracelint is open-source and judge-free: github.com/AshwinUgale/tracelint. Feedback and broken traces welcome.

Your evals pass. That doesn't mean they work.

Ashwin Ugale — Mon, 24 Aug 2026 14:32:09 +0000

Last week I wrote about tracelint, a linter that catches structural bugs in agent traces — the classic one being an agent that calls charge_card, gets a failure back, and just... keeps going and tells the customer their order shipped.

The response was better than I expected, and one question kept coming up in different forms: "Okay — but wouldn't my eval suite have caught that anyway?"

Fair question. So I checked. On the case I'll show you, it didn't. The suite was green. The charge was declined. The agent said "payment successful," and every eval passed.

This post is about the tool I used to find that — muteval — and what happened when I pointed it at the exact failure tracelint was built for.

The blind spot in a passing eval suite

Here's the thing about a green eval run: it tells you your system passed today's tests. It tells you nothing about whether those tests would notice if the system silently got worse.

That's not hypothetical. Your contains("refund") check passes whether or not the model still follows the refund rule someone deleted last week. Your faithfulness judge passes a confidently-wrong answer as long as it's grounded in the context. The eval is green, and the regression walks straight through it.

In normal software we have a name for "are my tests actually any good?" — mutation testing. You deliberately inject bugs (mutants) into the code, rerun the test suite, and measure how many the tests catch. mutmut and Stryker do this. muteval does it for eval suites — except the "code" it mutates is the system under test: the prompt, the retrieved context, the tool outputs, the model.

Degrade the system → rerun your existing evals → report the percentage of injected regressions they caught. The ones they miss are survivors — concrete coverage gaps.

The declined charge

Back to the agent. Here's a deliberately naive payments agent: it calls charge_card and reports success. Its eval suite has one realistic semantic check — does the reply confirm the charge went through?

muteval has an operator called deny_tool_output. It takes a tool's result and turns it into the nastiest kind of failure: a domain failure returned as transport success — an HTTP 200 whose body says {"status": "declined"}. Structured-error handling never fires, because nothing technically errored. The agent proceeds. The final answer still reads "Your payment was successful."

Rerun the eval suite against that mutant:

[HIGH] SURVIVED  [deny_tool_output]
          tool output #1 returned a domain failure (HTTP 200 + status:declined)

Mutation score: 0%. The suite caught nothing. The card was declined, the agent lied, and the contains("successful") check passed — because the answer does still say "successful." That's a survivor, and it's exactly the kind of blind spot you'd never see from a green CI run.

(To be clear about the number: 0% here means of the regressions I injected, your evals caught none of them. Not "your system is broken" — "your tests wouldn't have noticed.")

Closing the gap — where the two tools meet

Here's the part I think is actually interesting.

The thing that catches "declined charge reported as success" isn't a smarter output judge — the output looks fine. It's a structural check on the trace. Which is exactly what tracelint does.

So I wired tracelint in as one of muteval's evals. tracelint reads the agent's trace, and if the tool declares what failure looks like (failure_when: {"pointer": "/status", "in": ["declined"]}), it flags the declined charge deterministically — no judge, no key.

Add that one check and rerun:

verdicts on the declined-charge mutant:
  semantic eval (confirms "successful")  -> PASS   (misses it)
  tracelint (declared failure contract)  -> FAIL   (kills it)

mutation score:  semantic eval alone 0%  ->  + tracelint 100%

muteval didn't just tell me the gap existed — it let me prove the fix closes it: the mutant that survived the semantic suite dies the moment the structural check is in the suite, and the baseline stays green. That's the whole loop — a survivor names a missing eval, you add it, the score goes up.

A tool fault is a mutant; a trace-lint rule is an eval. Mutation testing on one side, deterministic trace checking on the other, and they compose.

What this does NOT tell you (the part I care about most)

A diagnostic you can't trust is worse than none, so:

A survivor is a candidate, not a verdict. muteval tells you your eval missed an injected change. Whether that change could really happen, and would really be bad, is a human call. Some survivors won't survive that question.
Mutation coverage is not validity. A suite can be highly sensitive to mutations and still be wrong — sensitivity isn't the same as agreeing with ground truth or a human. That needs labels; no label-free tool gets you there.
The mutations are rule-based, synthetic edits. They model real regressions; they aren't identical to them. Whether mutants predict the failures your system actually experiences is the open question I'm still chasing.
It's a per-suite diagnostic, not a universal flaw-finder. It tells you where your suite has a hole — not a new universal truth about evals.
It needs a re-runnable system. muteval degrades the system and needs a fresh output per mutant. A frozen CSV of outputs can't be mutated.

muteval fails closed on all of this: a red baseline, too few mutants, or too many errored mutants means it refuses to emit a score rather than hand you a confident-but-meaningless number.

Try it

There's a demo that runs with no key — a mock model, so you can watch the mutate → survive → fix loop end to end in about a minute. (Running it on your own suite calls your model + a key, like any eval.)

git clone https://github.com/AshwinUgale/muteval && cd muteval
pip install -e ".[tracelint]"
python examples/agent_tool_fault/run_demo.py     # keyless, ~1s

Repo: https://github.com/AshwinUgale/muteval (Apache-2.0) · tracelint: https://github.com/AshwinUgale/tracelint (MIT).

And the genuine ask, same as last time: if you write evals for agents or RAG, I'd like to know what your suite would miss. What's the regression you're most afraid of shipping — and if you injected it, would it survive your evals? Issues, replies, and war stories all welcome.

Your agent ignored a failed tool call. Here's how to catch that in CI.

Ashwin Ugale — Mon, 17 Aug 2026 21:43:46 +0000

You ship an AI agent. It calls tools, reads results, calls more tools, answers. Most of the time it works. Then a user reports something wrong, you open the trace, and you find it: the charge_card tool returned a 402, and the agent just... kept going and told the customer their order shipped.

That's not a hallucination in the "made up a fact" sense. It's a structural defect in the run — an ignored tool error. And here's the thing about structural defects: you don't need another LLM to find them. They're decidable by looking at the trace.

That's the whole premise of tracelint: a linter for agent runs. It reads the execution trace — what the agent actually did — and flags structural bugs deterministically, with the exact trace lines as evidence and a CI exit code. It runs after the run, on the trace, not on your code. No second model ever judges it.

Why not just use an LLM judge?

Because for this class of bug, a judge is the wrong tool. Published trace-error benchmarks show LLM judges have low localization accuracy — they'll tell you "something seems off" without reliably pointing at which step. They're also non-deterministic, cost money per trace, and can't gate CI (would you fail a build on a coin-flip?).

Meanwhile, a whole category of agent bugs is structurally decidable:

A tool call whose arguments violate the tool's JSON Schema. That's not an opinion — you run the schema validator.
A tool that returned an error, followed by the agent proceeding as if it hadn't.
The same tool called 5 times with identical arguments and identical results (a stuck loop).
Arguments that don't appear anywhere in what the agent observed (a candidate hallucinated value).

None of these needs a model. They need the trace and a validator. That's what tracelint does.

The 60-second version

pip install tracelint
tracelint demo --html demo.html

demo runs a keyless validation suite — one planted instance of every defect, plus clean controls — and writes an HTML report. No API key, no model download.

To gate CI on a real trace:

tracelint check ./trace.json --tools ./tools.json    # exit 2 on a structural defect

Exit codes: 0 clean, 2 a structurally-provable defect, 3 an input error. Heuristic findings never fail CI on their own.

The part that matters: it runs on traces you already collect

Here's the distribution insight. You're probably already instrumenting your agent — with OpenInference (the OpenTelemetry semantic convention for AI), feeding Arize Phoenix, Langfuse, or an OTel collector. tracelint reads that telemetry directly. You don't learn a new trace format; you point it at the spans you've got.

tracelint check spans.json --format openinference    # Phoenix, OTLP, TRAIL
tracelint check trace.json --format langfuse
tracelint check messages.json --format openai

Or straight from a running Phoenix instance, in Python:

import phoenix as px
from tracelint import lint_otel_trace

spans = px.Client().get_spans_dataframe().to_dict("records")
report = lint_otel_trace(spans)
print(report.exit_code)          # 0 or 2
for f in report.active_findings:
    print(f.rule, f.tier.value, f.summary)

I validated this against real OpenInference exports, not just hand-built fixtures — a real Phoenix trace, an OTel-SDK span export, the Phoenix dataframe shape. On one real Phoenix trace, tracelint deterministically localized a genuine tool failure:

[hard_event] R2a tool_error_event  (step 9)
  'add_spans_to_dataset' returned an error (GraphQL query 'exampleMutation' ... 'an unexpected error occurred')

No model in the loop. Just: this TOOL span has an ERROR status, at exactly this step, here's the message.

What it catches

Rule	Finding
R1	schema violation — args fail the tool's JSON Schema
R2	tool returned an error / an errored value reused by a later side-effecting call
R3	hallucinated argument — value not derivable from anything observed
R4	loop — N identical no-progress calls
R5	redundant call — identical call + identical result, no mutation between
R6	malformed arguments — the tool-call arguments aren't valid JSON
R7	unknown tool — a call to a tool absent from the declared toolset

Here's a stuck-loop run through the OpenInference adapter:

[candidate] R4 loop  (step 2,4,6)
  'search' called 3 times in a row with identical arguments and no change in result state (ok)
[candidate] R5 redundant_call  (step 2,6)
  'search' repeats an earlier identical call with no mutating call in between

Two design decisions I'd defend

1. Candidate, not verdict. Only structurally-provable things (a schema violation, malformed JSON) are hard defects that fail CI. Heuristic signals — loops, redundant calls, suspicious arguments — are shown as candidates with their evidence, for a human to review, never asserted as truth and never failing your build on their own. A retry loop and a stuck loop look similar structurally; tracelint shows you the evidence and lets you decide, instead of pretending it knows.

2. It tells you what it couldn't check. This is the one I care about most. If a trace is missing a field a rule needs — no tool schemas, no result payloads — that rule doesn't silently pass. It suppresses with a stated reason, printed in the report:

suppressed (2) — not checked, not a clean pass:
  R1 schema_violation: no tool schema available for any called tool
  R7 unknown_tool: no tool registry supplied — cannot know which tools were declared

A clean report with hidden gaps is worse than no report — it's false confidence. tracelint refuses to give you that.

Honest limitations

It catches structural defects, not whether the final answer was correct. It won't tell you the agent gave bad advice; it'll tell you the agent ignored a failed tool call on the way there.
Hallucinated-argument, loop, and redundant-call findings are candidates unless structurally proven. Legitimate value transforms and intentional retries can trip them — that's why they're shown with evidence, not asserted.
Error detection is only as structured as the trace. A tool failure is caught when it arrives as a structured signal — an explicit error status, an HTTP status ≥ 400, or an error field. A domain failure hidden in an otherwise-successful payload (HTTP 200 carrying {"status": "declined"}) isn't recognized unless the tool declares what failure looks like — and today that's a silent miss for that one result, not a suppression. Teaching tools.json a per-tool failure predicate is the fix I'm working toward; until then, treat error detection as covering transport/structured errors, not arbitrary domain semantics.
A trace is only as good as its instrumentation. Missing fields mean suppressed rules, not fabricated ones.

Try it

pip install tracelint
tracelint demo --html demo.html

It's open source (MIT), dependency-light (jsonschema + stdlib), Python 3.10–3.12, and the whole test suite is offline and deterministic.

Repo: https://github.com/AshwinUgale/tracelint

If you're collecting agent traces and want deterministic checks on them, I'd genuinely like to know what breaks on your real exports — that's how the last three real-world shape fixes happened. Issues and traces welcome.

The reranker I added to improve RAG was causing most of my remaining misses

Ashwin Ugale — Mon, 10 Aug 2026 18:45:32 +0000

My RAG evaluation would tell me hybrid: 0.86 and I'd sit there with no idea what to actually change. Raise it how? Different embedder? Smaller chunks? Add a reranker? The aggregate score names a winner; it says nothing about why the losers lost.

So I built a small tool to answer the "why," and running it turned up something I didn't expect: in my best-performing configurations, the component I'd added to improve results — the reranker — was responsible for almost every remaining failure.

Here's the idea, the finding, and how to check it on your own corpus.

The problem with a single retrieval score

hit@k = 0.86 is an average over a lot of independent failures, and those failures don't all have the same cause. A query can miss because:

the answer text never made it into any chunk (a chunking/ingestion problem),
neither dense nor sparse retrieval fetched the right chunk into the shortlist,
dense + sparse fusion dropped it,
the reranker reordered it out of the top-k,
it landed just past the top-k cutoff,
or the context-token budget dropped it. Every one of those wants a different fix. Averaging them into 0.86 throws away the only information that tells you which lever to pull.

Attribute each miss to the stage that lost it

The fix is to stop treating retrieval as a black box and record what survived at each stage of the pipeline, then attribute every missed query to the earliest stage that could no longer cover the answer:

representation → ann_index → candidate_generation → fusion →
reranker_demotion → final_cutoff → budget_cutoff

One important detail that makes this stable: gold answers are stored as character spans in the source document, not chunk IDs. That way the labels don't break when you change chunking strategy, and the scorer can credit an answer that's covered by several chunks together.

What the sweep actually showed

I ran 50 configurations (chunking × embedding × dense/BM25/hybrid × reranking, with real E5/BGE embedders and a cross-encoder reranker) over a small synthetic API-documentation corpus — 22 documents, 400 labeled queries. Three things jumped out, and only the attribution view makes them visible.

1. Chunking was the biggest lever — not the embedding model

The spread from worst to best config was hit@k 0.79 → 0.99. The bottom of the table was dominated by small fixed 200 chunks, whose failures were mostly "not retrieved" and "final cutoff," and whose hits were often flagged "fragile" — the answer was covered only because several chunks pieced it together, so a small chunking change would break it. The top was parent-child 800x200. Swapping E5 for BGE barely moved anything by comparison. If I'd only stared at aggregate scores I'd have fiddled with embedders; the attribution said chunk strategy was where the wins were.

2. In the strong configs, the remaining misses were almost all the reranker

This is the one that surprised me. Take a strong config — e5 · semantic · dense · rerank ce, hit@k 0.97. Where did its remaining misses go?

Reranker demotion: 13   (all of them)

Every single miss was the cross-encoder pulling the correct chunk out of the top-k. And it wasn't a one-off — across the reranked configs the residual misses were overwhelmingly reranker_demotion. Compare the same config without the reranker (e5 · semantic · dense, hit@k 0.96): now the misses are all final_cutoff — chunks that were ranked fine but landed one slot past k.

So the honest read isn't "rerankers are bad." The reranker raised MRR nicely (0.80 → 0.86) and nudged hit@k up. But once retrieval was already strong, the reranker became the single largest source of the failures that were left — which points at a precise, small fix (increase candidate depth / rerank top-N, or tune the reranker), not "retrieval is broken, start over."

Update: Two comments pushed on this finding, and they were right to. The reranker_demotion count is checked earliest-stage-first — a query only reaches that check once every upstream stage already succeeded for it — so it's conditional on the rest of the pipeline, not a standalone measure of the reranker's damage. Diffing rerank=none vs rerank=ce on the same 400 queries by query id (same candidate set both times, the reranker only reorders it) gives 12 promoted, 9 demoted, net +3 hits. The reranker is net positive.

It's also worse than a conditional count: of the 13 misses labeled reranker_demotion, only 9 are queries the reranker actually flipped. The other 4 were already outside the top-k before reranking (final_cutoff under the no-rerank config) — the reranker changed nothing for them either way. But once a reranker is configured, final_cutoff is unreachable in the attribution, so those 4 pre-existing failures inherit the reranker_demotion label anyway. "13, all of them" overstated the reranker's real damage by about 30%.

The 9 genuine demotions aren't random either — several cluster on passages that pack multiple same-type named entities close together (three competing stadium names in one paragraph; two named opponents in another). It reads as a precision problem on crowded passages, not "rerankers are bad" and not a query-type pattern.

3. Confidence intervals stop you from over-reading the leaderboard

The top config was 0.99 [0.98–1.00]; the next few were 0.98 [0.97–0.99]. Those intervals overlap — on 400 queries, 0.99 vs 0.98 is a tie, not a win. Without the CI you'd "pick the 0.99" and congratulate yourself on noise.

4. Half the context for ~two points of quality

The best config hit 0.99 but at ~577 average retrieved tokens. An e5 · recursive 400 · hybrid · ce config reached 0.963 at ~311 tokens — within about three points for roughly half the context. If you're context- or cost-bound, that's the smarter pick, and the quality-vs-tokens (Pareto) view is what surfaces it.

Try it on your corpus

The offline demo needs no API key or model download:

pip install retrieval-lab
retrieval-lab demo

For a real sweep you give it two JSONL files — your documents and your labeled queries — and it writes a single self-contained HTML report (rankings, confidence intervals, per-stage attribution, latency/cost, and the Pareto view):

pip install "retrieval-lab[real-embed,rerank]"

retrieval-lab run \
  --corpus docs.jsonl \
  --queries queries.jsonl \
  --embed-models e5,bge \
  --chunkers fixed:200,fixed:400,recursive:400,parentchild:800x200 \
  --retrieval dense,sparse,hybrid \
  --rerank none,ce \
  --html report.html

Live example report (the 50-config sweep above): https://ashwinugale.github.io/Retrieval-Lab/
Code: https://github.com/AshwinUgale/Retrieval-Lab

The honest limits

It's only as representative as your labeled query set — a thin or biased set biases the winner. Every score describes your corpus, never "best" in the abstract.
Missing valid gold alternatives make measured recall a lower bound.
Latency and index cost are whatever your machine reports.
Stage attribution needs a decomposable pipeline; a black-box retriever can only be scored at its output. It's beta. If the attribution gets something wrong for you — misattributes a miss, or blames a stage you don't think is at fault — that's exactly the feedback I want. What's the failure stage you wish your RAG eval could point at?

Mutation testing, but for LLM evals — early experiment, would love feedback

Ashwin Ugale — Wed, 22 Jul 2026 19:54:49 +0000

I write evals for LLM features, wire them into CI, and move on. But a passing eval suite has a blind spot: it tells you nothing about whether it would actually fail if the model quietly got worse. Green isn't the same as good.

So I built muteval to measure that directly. It borrows mutation testing from software engineering — you deliberately break the thing under test and check whether your tests catch it. muteval degrades the system (weakens a prompt rule, drops a retrieved doc, swaps in a weaker model), reruns your existing eval suite against each degraded version, and reports what fraction of those injected regressions your evals caught. The ones they miss are concrete coverage gaps.

Mutation score: 33% (2/6 caught)
SURVIVED: deleted "if the answer isn't in the context, say you don't know"

A bit on how it works. It ships around 18 of these mutations, each modeling a way a real system quietly degrades: softening a "must" to a "should", flipping a "do not", dropping or corrupting a retrieved document, shuffling context, swapping in a cheaper model, or breaking a tool's output for agent setups. Before mutating anything it confirms your suite passes on the original system — if it doesn't, there's nothing meaningful to measure — then reruns your evals against each mutation, ranks whatever survives by severity, and suggests an eval that would close each gap.

Does it actually find anything real? I ran it on Vectara's open-rag-eval. Its citation check caught mutations that made the model stop citing its sources — but it completely missed one that removed the rule telling the model to say "I don't know" when the answer isn't in the context. The model kept citing sources while being free to make things up, and a citation check simply can't catch that. Once I added a check for that behavior, it caught it.

It's pure Python, no required dependencies, and works with your deepeval/RAGAS/promptfoo metrics or its own checks:

pip install muteval
muteval init      # scaffold a config
muteval check     # validate it, then muteval run

Where I think this goes next is a tool that rates an eval suite across several angles and suggests the evals you're missing — but I'd like to hear how other people approach it first. If you write LLM evals: how do you know they're any good?

Repo (Apache-2.0): https://github.com/AshwinUgale/muteval