One Quality Score Is a Lie: Split Your RAG Judge Into Retrieval, Groundedness, and Relevance

Ask an LLM judge to score your RAG agent's answer and it will hand you a number. A 7. A 4.2. A crisp green 0.85. That number feels like signal. It is actually three different failures averaged into mush, and the average is engineered to hide the one you most need to see.

I'll make the case with a concrete .NET reference implementation (maf-evals), a three-tier agent evaluation built on Microsoft Agent Framework and .NET 8. Every number below came from running it, not from reasoning about it.

The problem with a single score

A retrieval-augmented answer can be wrong in at least three independent ways:

• The retrieval was bad — wrong documents came back, or the knowledge base doesn't cover the question.
• The groundedness was bad — the retrieved documents were fine, but the answer made claims they don't support. That's a hallucination.
• The relevance was bad — the answer is perfectly grounded in real documents and still doesn't answer what the user asked.

These have nothing to do with each other. A great retrieval can feed a hallucinated answer. A flawlessly grounded answer can miss the question entirely. When you collapse them into one "quality" score, a mediocre-everywhere answer and a perfect-retrieval-but-hallucinating answer can land on the same 3.5. You cannot tell which knob to turn, because the score was designed to be indifferent to which knob is broken.

So the maf-evals RAG triad scores all three separately, and never averages them:

| Score        | Isolates                                        |
| Retrieval    | a bad knowledge base or a bad query             |
| Groundedness | claims the retrieved context doesn't support    |
| Relevance    | well-grounded answers that miss the question    |

Why the average actively lies

Here's the part that turned my opinion from "nice to have" to "non-negotiable." When you calibrate these judges against human labels — and you must calibrate them, a threshold picked without calibration is just taste — the Groundedness judge produces this:

| Score        | Exact | Within 1 | MAE  | Bias  | Same band |
| Groundedness | 42%   | 67%      | 1.17 | -0.17 | 75%       |

Look at that bias: -0.17. Nearly zero. If you were tracking a single blended quality score, that -0.17 would tell you the judge is essentially unbiased and calibrated. Ship it.

It is not calibrated. The Groundedness judge fails in two opposite directions at once. It scores outright fabrication at exactly 3.0 every time — a hallucination sails through as a mild warning — and it penalises well-grounded answers for being slightly off topic. The two errors point opposite ways, so they cancel. The near-zero bias is the average of a systematic over-score and a systematic under-score, and it looks healthy precisely because both are broken.

The metrics that expose it are the ones the average destroys: a mean absolute error of 1.17 (over a 1–5 scale, that's enormous) and band agreement of only 75%. You only see those if you refuse to blend.

That single insight — that fabrication was being scored 3.0 — is why the Groundedness floor moved from 3.0 to 3.5. At 3.0, every hallucination slipped through as a warning. Moving the floor to 3.5 lifted band agreement from 50% to 75%. A blended score would never have surfaced the problem to move the floor over.

Deterministic first, judge last

The triad isn't the first thing that runs. In the pull-request gate (Tier 2), five checks run cheapest-first, and the judge is last:

1. Rules — the same rule engine the live agent uses, so a rule can't drift between production and CI.
2. Retrieval — did the expected document IDs come back? Exact, free, no model involved.
3. Tool calls — right tool, right arguments? Compared, not judged.
4. Meaning — embeddings, for cases where wording is free to vary. Deterministic, ~1000x cheaper than a judge.
5. RAG triad — the judge, and only now.

The ordering is doctrine, not convenience. Evidence ranks on an independence axis — independent to corruptible — not a cost axis. A document-ID match is independent: it means the same thing every run. A judge score is corruptible: it moves between runs. So the exact checks run first and always block, while the judge scores get two thresholds — a floor that blocks and a target that warns — because a number that wobbles has no business being a hard gate on its own.

This is why Retrieval scoring is advisory in maf-evals, even though it's part of the triad. Calibration caught the judge returning 5, 2, 4, 5, 2 for the same input — 17% of cases would flip a merge decision at random. So expectedChunkIds, an exact and free check, does the actual gating. The judge only annotates.

What it looks like in code

A golden case declares the deterministic expectation right next to the judged one. Gating lives in the exact fields; the triad adds color:

{
  "id": "refund-within-limit",
  "query": "Order A-31905 arrived damaged. Please refund me 120 for it.",
  "expectedChunkIds": ["refunds#3"],
  "expectedToolCalls": [
    { "name": "issue_refund", "arguments": { "orderId": "A-31905", "amount": 120 } }
  ],
  "semanticExpectations": [
    { "name": "confirms_refund", "anyOf": ["Your refund of 120 has been issued."], "minSimilarity": 0.55 }
  ]
}

expectedChunkIds gates retrieval for free. The triad's Retrieval score, being corruptible, only advises on top of it. Groundedness and Relevance — the two that genuinely need a judge, because they're about meaning — get their floor-and-target bands. Nothing is averaged.

And it's cheap to be this careful about which score means what, but not free: the judge costs about 250x the agent. A full Tier 2 run is ~$0.16. The thing being tested is nearly free; measuring it is the entire bill. That's exactly why you don't want to pay a judge to produce a blended number you then can't act on.

The takeaway

One quality score is a management metric wearing an engineering costume. It goes up and to the right and tells you nothing you can fix. Split it, and each score points at a specific, actionable failure: bad KB, hallucination, or off-topic. Then gate on the checks that don't wobble, and let the judge report on the two that genuinely require meaning.

If you want the whole thing — the triad, the calibration harness that caught the 3.0-hallucination bug, the deterministic checks, and the cost tracker that proves the judge is your real bill — it's all runnable .NET here: github.com/sauravbhattacharya001/maf-evals. Clone it, run dotnet run --project src/EvalRunner -- calibrate --repeat 3, and watch a near-zero bias hide a broken judge on your own screen.

Comments

[Maya Andersson]

The -0.17 is the most useful number in this post, because it is a mean of two errors pointing in opposite directions and a mean is the one statistic guaranteed to hide that. The cheap replacement is signed error stratified by the true label instead of pooled: a judge that runs high on fabrication and low on well-supported answers shows up immediately as two bands with opposite signs, and never shows up in the pooled figure. The other thing worth publishing next to MAE is agreement in the narrow region around your gate threshold, since overall agreement is dominated by the easy ends of the scale where no decision is at stake. A judge can look calibrated across the full range and still sit near a coin flip exactly where the gate fires.