How Do You Know Your RAG Actually Works?

It's Working Really Well(phase 5A)

👦 Nephew: Uncle, good news. I added reranking, the relevance threshold, citations, hybrid search — everything from last time. I tested it for two days. Our chatbot is working really well now.

👨‍🦳 Uncle: How do you know?

👦 Nephew: ...I asked it a bunch of questions. The answers looked right.

👨‍🦳 Uncle: Looked right to whom?

👦 Nephew: To me.

👨‍🦳 Uncle: And how many questions did you ask?

👦 Nephew: Maybe fifteen.

👨‍🦳 Uncle: Out of every question your users will ever type — across every department, every bad phrasing, every month, forever?

👦 Nephew: ...when you say it like that, fifteen sounds like nothing.

👨‍🦳 Uncle: It is nothing. Years ago I spent two full days debugging a chatbot that kept giving the wrong policy number. I blamed the model. I blamed the prompt. I lowered the temperature, raised it again, rewrote the system prompt four times. Turns out none of that was broken. Retrieval had simply never found the right document in the first place — I was debugging the wrong component for two days because I never checked the one thing that actually failed first.

👦 Nephew: So you can't tell if the LLM is lying, or if it was just never shown the truth.

👨‍🦳 Uncle: Exactly. And "it looks fine" can't tell those two apart. Only measurement can.

The funny thing? I used to think evaluation was something only big companies worried about — something for teams with a dedicated ML platform, not for someone building a demo alone on a weekend. The first time I built a RAG system myself, I just kept asking the same five questions, over and over, until I'd convinced myself it worked. Looking back, I wasn't testing the system. I was testing my own optimism.

There's also a quieter reason this matters. Every time someone tweaks a prompt, swaps the embedding model, upgrades the reranker, changes the chunk size, or imports a fresh batch of documents — any one of those can silently make retrieval worse. Nobody notices, because nobody's watching a number. They find out three weeks later, from a customer complaint. Evaluation is what turns "we think it's still fine" into "we know exactly what changed."

Why Fifteen Random Questions Prove Nothing

👨‍🦳 Uncle: Here's the trap. When you test manually, you unconsciously ask questions you already expect to go well. You skip the weird ones. And when an answer comes back a little off, you're generous — "eh, close enough" — because you're not grading it, you're just glancing at it.

👦 Nephew: That's... uncomfortably accurate.

👨‍🦳 Uncle: Every engineer does it the first time. The fix is boring but non-negotiable: a fixed set of questions, run the same way every time, scored against an answer you already know — not against your mood that day. That's called evaluation, and it splits into two:

OFFLINE EVALUATION                 PRODUCTION MONITORING
Before you ship                    After you ship
Questions you wrote                Questions real users type
You know the right answer          You often don't
Catches regressions early          Catches what you never
                                    thought to test

We're building the offline half today — you can't monitor production well until you know how to grade an answer at all.

The Golden Dataset — Your Answer Key

👨‍🦳 Uncle: Imagine handing a new employee a test with no answer key, then judging their score by vibes. That's what testing without a golden dataset is. A golden dataset is simply: a question, paired with what a correct result actually looks like — written down before the system ever runs.

👦 Nephew: So for every question I also write the expected chunk, the expected answer, and the source it should cite?

👨‍🦳 Uncle: Exactly. And here's the part almost everyone gets wrong: they only write down questions they expect the system to answer correctly. A good golden dataset needs four kinds of questions, not one — and let's not stay stuck on HR policy the whole time, since your readers won't all work in HR tech:

ANSWERABLE
"What does error code ERR_4521 mean?"           → should answer, correctly, from docs

UNANSWERABLE
"What is the CEO's birthday?"                    → should say "I don't know"

PARTIAL
"What's the loan approval time for freelancers?" → policy only covers salaried applicants

MULTI-DOCUMENT
"Does this patient's prescription conflict
 with their listed allergy?"                     → answer spans two separate records

👦 Nephew: I never once tested the "should say I don't know" case. I only ever tested things I expected it to get right.

👨‍🦳 Uncle: That sentence right there is the whole reason today's article exists. Nobody tests for the honest "I don't know." Everyone tests for the impressive correct answer. Production breaks in the gap between those two habits.

Grading Retrieval Before You Grade the Answer

👨‍🦳 Uncle: Here's a question for you. If the final answer comes out wrong, is that the LLM's fault, or retrieval's fault?

👦 Nephew: ...I genuinely don't know how I'd tell.

👨‍🦳 Uncle: Neither can most people, because they only ever look at the final answer. Retrieval happens before the LLM sees anything — so we grade it on its own, first.

Here's a question, before I define anything. Imagine I hand you five documents in response to something you asked. Three of them actually answer your question. Two are completely unrelated. Would you call that a good search?

👦 Nephew: Maybe... 60%?

👨‍🦳 Uncle: Congratulations. You just calculated Precision without knowing its name.

Documents handed to you: 5
Actually useful:         3
Precision = 3/5 = 0.6

Let's make it concrete. Imagine this is Stripe's documentation search. Someone asks about error code ERR_4521. Five chunks come back.

1. "ERR_4521 means the webhook signature is invalid"       ✅
2. "How to reset your dashboard password"                   ❌
3. "ERR_4521 often follows a clock drift on your server"     ✅
4. "Our pricing tiers explained"                             ❌
5. "Retry policy for failed webhook deliveries"              ✅

Same math — Precision@5 = 3/5 = 0.6. Low precision means you're handing the LLM more distraction than signal, regardless of whether it's your five-document thought experiment or Stripe's real docs.

Now flip it. Say there are eight chunks across Stripe's knowledge base that actually explain ERR_4521 — some in the API reference, some in old incident postmortems, some in a troubleshooting FAQ. Your search only surfaced six of them.

Relevant chunks that exist:  8
Relevant chunks you found:   6
Recall = 6/8 = 0.75

👦 Nephew: So Recall is "how much of the truth did we actually find," and Precision is "how clean was what we found"?

👨‍🦳 Uncle: Exactly right. So — if Recall is high, we're basically done, yes?

👦 Nephew: ...I want to say yes.

👨‍🦳 Uncle: Imagine retrieving 100 chunks just to be extra safe. You'll find almost everything — Recall shoots close to 1.0. But now the one chunk that actually answers the question is buried under 99 nearly-irrelevant ones, and the LLM has to wade through all of it before it even gets there. High Recall alone can quietly murder your Precision.

Hit Rate — sometimes you don't need a fraction at all. Forget precision and recall for a second. Did at least one genuinely correct chunk show up anywhere in your results — yes or no?

Q1: correct chunk somewhere in Top-5?  YES
Q2: correct chunk somewhere in Top-5?  NO
Q3: correct chunk somewhere in Top-5?  YES

Hit Rate = 2/3 = 0.67

Good for a fast health check across a hundred questions before you dig any deeper.

MRR (Mean Reciprocal Rank) — Hit Rate tells you whether the right chunk showed up. It says nothing about where. Think about a GitHub-style docs bot: someone asks about rate limits, and the correct chunk is sitting at rank 1 for one user's question, and rank 18 for another's.

Q1: correct chunk at rank 1   → 1/1  = 1.0
Q2: correct chunk at rank 18  → 1/18 = 0.06

MRR = (1.0 + 0.06) / 2 = 0.53

👦 Nephew: Rank 1 is basically a perfect score, and rank 18 barely counts, even though technically it's "found"?

👨‍🦳 Uncle: Right — and that gap matters, because the LLM (and your token budget) pays far more attention to what's near the top.

NDCG — one more wrinkle, and it's a subtle one. Not all "relevant" chunks deserve equal weight. Think of a medical assistant: one chunk comes from the patient's official allergy record, another is a nurse's informal note that happens to mention the same allergen in passing.

👦 Nephew: Both count as "relevant" if we're just checking yes or no.

👨‍🦳 Uncle: Right, but if the informal note ranks above the official record in your results, something is off, even though both are technically correct. NDCG accounts for graded importance and position together — it's ranking quality with a sense of what actually matters more, not just what's relevant at all.

Now Check What the LLM Actually Said

👨‍🦳 Uncle: Say your retriever just scored 96% across all five of those metrics. Does that mean your chatbot is trustworthy?

👦 Nephew: ...I want to say yes, but I have a feeling you're about to say no.

👨‍🦳 Uncle: No. Retrieval passing only means the LLM had the chance to be right. A 96% retriever can still hand over a perfect chunk, and the LLM can still make something up anyway. Now we check what it actually did with that chance — sorted into four buckets: correct, partial, wrong, or contradictory, where contradictory means the answer disagrees with the very source it cited.

Groundedness — Did the Claim Come From Anywhere Real?

👨‍🦳 Uncle: A loan-approval assistant retrieves a chunk saying the maximum loan tenure is 20 years. The LLM answers: "You can get a 25-year tenure." Grounded?

👦 Nephew: No — 25 doesn't exist anywhere in what it was given. It just invented it.

👨‍🦳 Uncle: That's groundedness in one sentence: does every claim trace back to something actually retrieved? There's a third, sneakier outcome beyond grounded and not-grounded — partially grounded, where most of the answer is true and one invented detail hides quietly inside it. That one is more dangerous than a fully wrong answer, because the true parts make the whole thing feel trustworthy.

Faithfulness — the Subtler Cousin

👨‍🦳 Uncle: Harder one now. The chunk says: "Patients with a penicillin allergy should not be prescribed amoxicillin." The LLM answers: "Amoxicillin is generally best avoided for most patients with allergy concerns." Grounded?

👦 Nephew: I mean... nothing's technically false, so... yes?

👨‍🦳 Uncle: Look closer. "Patients with a penicillin allergy" — a specific, named condition — quietly became "most patients with allergy concerns," a vague generalization. Nothing here is an invented fact, but the precision and scope of a medical instruction just eroded in translation, and in a domain like this, that erosion is the whole danger. Groundedness asks "is this true." Faithfulness asks "did you say it the way the source actually meant it."

Citation Validation — Do the Footnotes Point Where They Claim?

👨‍🦳 Uncle: The LLM says: "Your API rate limit is 1,000 requests per minute [2]." You open chunk 2. It actually says 100 requests per minute.

👦 Nephew: So the citation existed — it just pointed at the wrong thing.

👨‍🦳 Uncle: Right, and that's exactly what people forget to check. A citation number being present is not proof of being correct. You're checking three separate things: does every claim have a citation at all (coverage), does the cited source actually say that (correctness), and is it citing the best available source, not just a technically-not-wrong weaker one (quality).

Negative Testing — Does It Know When to Shut Up?

👨‍🦳 Uncle: Ask it something that exists nowhere in any document — "What is the CEO's birthday?" — and watch what happens.

👦 Nephew: Can I just ask ChatGPT if my RAG system is good instead of doing all this?

👨‍🦳 Uncle: You can. It will confidently tell you your baby is beautiful.

Across your whole golden dataset, track one number: abstention rate — of all the genuinely unanswerable questions, how many did the system correctly refuse instead of confidently guessing? This is the metric most teams skip, and the one that costs them the most in public, because every missed abstention is a confident lie reaching a real user.

The Whole Thing, End to End

                    Golden Dataset
                         ↓
                    Run Retrieval
                         ↓
        Precision · Recall · Hit Rate · MRR · NDCG
                         ↓
                     Run the LLM
                         ↓
     Grounded? · Faithful? · Citations correct? · Abstained?
                         ↓
                  Final Scorecard
                         ↓
          Pass → ship it     Fail → fix it first

This pipeline is intentionally language- and stack-agnostic. Whether your retriever runs in Node, Python, Go, or anything else, the four questions stay the same: did search find the right things, did the model tell the truth, did it point to the truth correctly, and did it know when to stay silent?

If you strip away every diagram in this article down to just one, it's this:

                       Does my RAG work?

                             │
              ┌──────────────┴──────────────┐
              │                              │
      Did retrieval work?            Did the LLM work?
              │                              │
        Precision                     Groundedness
        Recall                        Faithfulness
        Hit Rate                      Citation Check
        MRR                           Abstention
        NDCG
              │                              │
              └──────────────┬───────────────┘
                             │
                     Production Ready?

Interview Questions

• What's the difference between retrieval evaluation and answer evaluation?
• What is Precision@K vs Recall@K, and why do they trade off against each other?
• Why can high Recall quietly hurt Precision?
• What does MRR tell you that Hit Rate doesn't?
• What is groundedness? How is it different from faithfulness?
• Why include unanswerable questions in a golden dataset? What metric tracks that?
• How would you verify a citation actually supports the claim beside it?
• Why is manual, ad-hoc testing not enough for a production RAG system?

Where This Leaves Us

👨‍🦳 Uncle: So — where does that leave you?

👦 Nephew: We can finally prove the system works instead of just feeling like it does. Precision, recall, groundedness, abstention — all measurable, all repeatable.

👨‍🦳 Uncle: Good. Last week, a company uploaded one PDF into their support knowledge base. Just one. Their chatbot read it — and obeyed it. It ignored its own system prompt. It started leaking details from completely unrelated customer documents to whoever asked.

👦 Nephew: Wait... a PDF can do that?

👨‍🦳 Uncle: That's exactly what we're building next.