Hope Is Not a Metric
Last week you swapped the embedding model. Or changed the chunk size. Or rewrote the system prompt. Simple question: is the system better or worse now?
If the honest answer is "it feels better," you are not engineering, you are hoping. Nobody would merge a backend change with no tests and a shrug, yet most RAG systems in production ship exactly like that: a demo that impressed someone once, a few cherry-picked queries, and no way to tell whether Tuesday's change quietly broke the answers that used to work.
The fix is not exotic. It is the same discipline you already apply to code: a versioned test suite, metrics that localize the failure, a live signal for traffic you did not anticipate, and a gate that refuses to ship a regression. This post is the tactical version of how we build that at Oronts.
One Pipeline, Two Failure Surfaces
A bad RAG answer has two very different root causes, and you cannot fix what you cannot locate.
query
|
v
+-------------+ measuring point 1:
| RETRIEVAL | did the right chunks come back at all,
+-------------+ and how high did they rank?
|
top-k chunks
|
v
+-------------+ measuring point 2:
| GENERATION | given those chunks, did the model write
+-------------+ a faithful, relevant, attributed answer?
|
v
answer
If you only score the final answer, a failure is a black box. Retrieval might have returned the wrong documents, or retrieval was fine and the model invented something anyway. Those two problems have completely different fixes (chunking and embeddings versus prompt and model), so the first rule of RAG evaluation is: instrument both stages, always.
The Test Set Is the Asset
Everything else hangs off one artifact: a versioned test set that lives in git next to the code. Not a hundred easy questions. A deliberate mix:
| Category | What goes in | Why |
|---|---|---|
| Common | The questions users actually ask most | Protects the bulk of traffic |
| Hard | Ambiguous, multi-part, edge-of-corpus questions | Finds where the system slips |
| Known failures | Every production bug you ever found | The same bug can never return unnoticed |
| Adversarial | Prompts built to trigger hallucination or leakage | Tests the guardrails, not the happy path |
Each entry pairs a question with what correct looks like at both stages: which chunks should be retrieved, and what the answer must contain.
- id: cancellation-same-day-001
category: known_failure
question: "What is the cancellation policy for same-day bookings?"
expected_chunks:
- policy-cancellations#chunk-3
answer_must_state:
- "same-day bookings are non-refundable"
answer_must_cite:
- policy-cancellations
origin: "support escalation 2026-03, system claimed a 24h refund window"
added: "2026-03-14"
The origin field matters more than it looks. Every entry that came from a real failure is a permanent regression check with a paper trail. Six months from now, when someone proposes a chunking change that re-breaks this case, the suite fails with a message that explains exactly why the case exists.
Rules we hold ourselves to:
- The test set is version-controlled and reviewed like code.
- Every production bug adds an entry before the fix merges, the same way you write a failing test first.
- The set grows monotonically. Deleting a case requires the same justification as deleting a test.
Score the Two Stages Separately
With the test set in place, score each stage against its own metrics:
| Stage | Metric | Question it answers |
|---|---|---|
| Retrieval | recall at k | Are the expected chunks anywhere in the top k? |
| Retrieval | first-relevant rank | Is the best chunk near position 1, or buried at 9? |
| Generation | faithfulness | Does the answer stay inside the retrieved facts? |
| Generation | answer relevance | Does it answer the question that was asked? |
| End to end | attribution | Did the model actually use the context, and cite it? |
Retrieval metrics are cheap and deterministic: you know which chunks should come back, so scoring is set membership and rank. Generation metrics need a judge, either a rubric-driven LLM judge or a human sample, and both are fine as long as the rubric is fixed and versioned too. An unstable judge is just a second system you cannot trust.
The payoff is diagnosis. A run after a change reads like a stack trace instead of a mood:
run: 2026-07-18 change: hybrid search weights 0.7/0.3 -> 0.5/0.5
retrieval
recall@5 0.86 (prev 0.81) +0.05
first-relevant rank 1.9 (prev 2.6) improved
generation
faithfulness 0.94 (prev 0.94) flat
relevance 0.91 (prev 0.92) noise
regressions: 1
FAIL cancellation-same-day-001
expected policy-cancellations#chunk-3 in top 5, got rank 8
Read that output. The change helped retrieval on average and broke one specific known-failure case. Without the split you would see "quality roughly the same" and ship a re-broken bug. With it, you know the fix is on the retrieval side, and you know which document to look at.
One calibration note, because the industry keeps lying to itself here: a well-built system on a fair test set lands in real numbers. Extraction and answer accuracy in the 88 to 95 percent range is what good looks like on hard, representative cases. If your suite reports 99.9 percent, your test set is too easy, and it is measuring your optimism.
A Confidence Score for Traffic You Have Never Seen
Offline evaluation needs known answers. Production traffic does not come with any. So the second instrument is a confidence score: a composite the system computes on every single response, at runtime, with no ground truth required.
We run this in Exfinity, our own platform, where every response is scored on four factors we can measure live:
confidence = 0.4 * retrieval // did retrieval return usable results
+ 0.2 * policy // did validation and policy checks pass
+ 0.2 * completeness // substantive answer, not a stub
+ 0.2 * grounding // enough context to anchor the answer
score < threshold => flag as low confidence, route to review
The weights are less important than the shape: retrieval dominates because a response built on a retrieval miss is wrong no matter how fluent it sounds. Each flagged response also gets a classified reason (retrieval miss, policy block, hallucination signal such as a stale date presented as current), so review starts with a diagnosis instead of a raw transcript.
This is the online half of evaluation: a quality signal on 100 percent of real traffic, not just on the questions you thought to write down. Offline catches regressions before users do; the confidence score catches what your test set never anticipated. You need both, and the online half deserves the same rigor as your other production signals (our AI observability guide covers that side in depth).
Close the Loop or the Score Is Just a Warning Light
A confidence score that only flags things is a dashboard nobody acts on. Wire it into a loop:
response --> confidence score --> below threshold --> human review
|
+-------------------------+
v
approved answer folded back into the knowledge base
|
v
same question retrieves a good answer next time
When a reviewer approves a flagged answer that should have been good, that answer goes back into the knowledge base, so the retrieval gap closes itself. In Exfinity the loop goes one step further and tunes its own threshold: if reviewers approve nearly everything that gets flagged, the threshold drops to catch more; if they reject a large share, it rises to cut noise. The system improves from being used instead of merely accumulating history.
And every rejected review case? It becomes a new entry in the offline test set. The online loop feeds the offline suite. That is the whole architecture in one sentence.
Gate the Deploy
The last step is the one most teams skip, and it is the one that changes behavior: make the offline suite a CI gate, not a report.
change pushed
|
v
offline eval vs versioned test set
|
+-- any known-failure case regresses --> build fails
+-- recall or faithfulness below bar --> build fails
+-- all green --> deploy
Same semantics as a failing unit test. Nobody debates whether to ship a red build; the pipeline already decided. This is the practical difference between teams whose AI quality trends up and teams whose quality drifts while everyone insists it feels fine. A report gets skimmed. A gate gets respected.
For the retrieval architecture underneath all of this (chunking, hybrid search, reranking), see our enterprise RAG systems guide. We are Oronts, a founder-led software company in Munich; our founder and solution architect Refaat Al Ktifan has a standing rule for AI work: if we cannot show a client the number that moved, we do not claim the improvement.
Key Takeaways
If you cannot say whether a change made your RAG better or worse, you are hoping, not engineering. The answer is a number from a suite, never a vibe.
The versioned test set is the highest-leverage artifact you can build. Common, hard, known-failure, and adversarial cases, in git, growing with every bug, never shrinking.
Score retrieval and generation separately. Recall and rank for retrieval, faithfulness and relevance for generation. The split turns "quality dropped" into "chunk 3 fell to rank 8."
Real accuracy on a fair test set is 88 to 95 percent. A suite reporting 99.9 is measuring its own weakness.
A runtime confidence score evaluates the traffic your test set never saw. Composite of retrieval, policy, completeness, and grounding signals, with a threshold that routes low-confidence answers to review.
Close the loop. Approved answers fold back into the knowledge base, rejected ones become new offline test cases, and the threshold tunes itself from reviewer behavior.
Gate the deploy. A quality regression should fail the build with the same finality as a failing unit test. Reports get skimmed; gates get respected.
Top comments (0)