RAG Evaluation: How to Know if Your RAG System Actually Works
You built a RAG chatbot.
It answers questions from your documents. You test it a few times. The answers look good.
So… can you ship it?
No.
One good answer doesn't tell you whether your RAG system works.
A RAG application has multiple moving parts. The retriever can fail. The generator can fail. They can both work individually and still fail when combined.
And once the application goes live, your users will ask questions you never tested.
So how do you actually evaluate a RAG system?
The answer is an eval suite.
Components → Pipeline → Application → Regression → Online Evaluation
This article walks through the same framework I use in my RAG evaluation video.
The Problem: “It Feels Better” Isn't an Evaluation
Imagine you're building an airline support chatbot for a fictional airline called SkyHigh Airlines.
Passengers can ask questions about:
- Baggage
- Refunds
- Pets
- Travel policies
The chatbot uses RAG to search the airline's policy documents and generate an answer.
A passenger asks:
“How much does it cost to bring my cat?”
The chatbot responds:
“Bringing your cat costs $95.”
Looks good.
But what if the retriever found the wrong document and the model happened to generate something plausible?
Or what if the retriever found the correct policy, but the model ignored it and invented the answer?
From the outside, both problems look identical:
Bad answer.
But they require completely different fixes.
That's why you can't evaluate RAG as one giant black box.
You need to test the pieces separately.
First: Build a Golden Set
Before measuring anything, you need something to measure against.
Create a fixed set of questions that represent the kinds of questions your users will actually ask.
For our SkyHigh chatbot, imagine we create 50 questions about the airline's policies.
For every question, we record:
- The question
- The correct answer
- The document chunks that should contain the answer
For example:
Question
How much does it cost to bring my cat?
Expected information
Cats under 8 kg may travel in the cabin. Fee: $95.
Relevant chunk
The exact policy chunk containing that information.
This becomes your golden set.
The important thing is that you keep using the same set.
If version 6 is tested on 50 questions and version 7 is tested on a completely different set, comparing their scores doesn't tell you much.
Your evaluation set is your measuring stick.
And you shouldn't quietly edit it just because your score looks bad.
Once you do that, you're no longer measuring the system.
You're measuring the test you changed to make yourself feel better.
Level 1: Test the Components
The first level is simple:
Test each part independently.
A RAG system has two major components:
Retriever → Generator
So we test them separately.
1. Evaluate the Retriever
The retriever's job is to find the right information.
Our SkyHigh knowledge base has hundreds of chunks.
For every question in our golden set, we already know which chunks should be relevant.
Now give the retriever the question:
“How much does it cost to bring my cat?”
It searches all the chunks and returns its top 5.
Now we ask:
Recall
Did the retriever find the chunks it was supposed to find?
Suppose three chunks are relevant to the question.
The retriever finds two of them.
Then:
Recall = 2 / 3
Recall tells us how much of the information we needed actually made it into the retrieved results.
If recall is low, the generator is already in trouble.
The model cannot use information it never received.
Precision
Now ask a different question:
Of the chunks the retriever returned, how many were actually useful?
Suppose the retriever returned five chunks.
Only two were relevant.
Then:
Precision = 2 / 5
Low precision means we're giving the generator a lot of noise.
That can mean:
- More tokens
- More cost
- More context
- More opportunities for the model to get distracted
Recall and Precision Pull in Different Directions
There is an important trade-off here.
If you retrieve more chunks, you may increase recall.
But you may also decrease precision because you're bringing in more irrelevant information.
So the goal isn't simply:
“Maximize everything.”
You're tuning the retrieval system for your actual application.
And because we already know the correct chunks from our golden set, these measurements can be calculated programmatically.
No LLM judge required.
That's fast and cheap.
2. Evaluate the Generator
Now let's isolate the other half.
Instead of letting the retriever find the context, we give the generator the correct chunks directly.
Why?
Because we want the retriever completely out of the picture.
Now we can ask:
If the generator receives the right information, can it produce a good answer?
Two important metrics are faithfulness and answer relevance.
Faithfulness
Faithfulness asks:
Did the model stay within the information we gave it?
Suppose the policy says:
“Cats fly for $95.”
The model responds:
“Cats fly for $95, and pets fly free on Tuesdays.”
But the policy never mentioned free Tuesdays.
The model invented that part.
That's a faithfulness failure.
The answer may sound confident and fluent, but the claim isn't supported by the context.
Answer Relevance
Now consider a different failure.
The passenger asks:
“How much does it cost to bring my cat?”
The model responds with a beautifully written explanation of:
- Carrier requirements
- Breed restrictions
- Vaccination forms
- Travel documentation
Everything it says is true.
But it never tells the passenger the price.
That's an answer relevance failure.
The answer can be faithful to the context while still failing to answer the question.
That's why these are two separate metrics.
Who Grades the Generator?
There's a problem.
For retrieval, we knew exactly which chunks were correct.
But there isn't necessarily one perfect way to phrase an answer.
These are both correct:
“Your cat costs $95.”
and:
“Bringing your cat costs $95.”
So instead of comparing the output against one exact string, we can use an LLM-as-judge.
Give the judge:
- The question
- The retrieved context
- The generated answer
- A strict grading rubric
Then ask it to evaluate things such as faithfulness and relevance.
But LLM judges aren't free or perfect.
They're:
- Slower
- More expensive
- Potentially noisy
And judges can sometimes prefer answers that sound confident or elaborate.
So your rubric needs to be strict.
You should also spot-check the judge against human evaluations.
Think of the judge as a measuring instrument.
Instruments need calibration.
Level 2: Test the Entire RAG Pipeline
Now we've tested the retriever.
We've tested the generator.
But there's still another question:
What happens when they work together?
This time, we use the same 50 questions.
But we don't give the generator the correct chunks.
We let the RAG system run normally:
Question → Retriever → Retrieved chunks → Generator → Answer
Now we're measuring the actual pipeline.
This is where the RAG triad becomes useful.
The RAG Triad
1. Context Relevance
Question → Chunks
Did the retriever bring back information that is actually useful for answering the question?
For:
“How much does it cost to bring my cat?”
Did we retrieve the pet policy?
Or did we retrieve four paragraphs about oversized golf bags?
2. Faithfulness
Chunks → Answer
Did the generated answer stay grounded in what the retriever actually returned?
If the retrieved context says $95, did the model suddenly say $120?
3. Answer Relevance
Question → Answer
Did the final answer actually address what the passenger asked?
An answer can be grounded in the context and still fail to answer the question.
Component Evaluation vs Pipeline Evaluation
This distinction is extremely important.
In Level 1, we gave the generator perfect chunks.
We were essentially asking:
“How good could my generator be if retrieval were perfect?”
That's a component-level evaluation.
In Level 2, we let the retriever do its actual job.
Now we're asking:
“How good is the RAG system under real retrieval conditions?”
So:
Component eval measures potential.
Pipeline eval measures reality.
The pipeline score is much closer to what you actually ship.
Level 3: Evaluate the Application
So far we've been evaluating the machinery.
But the passenger doesn't care about recall.
They don't care about precision.
They care about:
“Did the chatbot give me a useful answer?”
That's Level 3.
Now we evaluate the application as a product.
Correctness
Is the answer actually right?
If the policy says:
$95
the chatbot should say:
$95
Not $75.
Not $120.
$95.
Correctness requires knowing what the correct answer actually is, which is why the golden set matters.
Completeness
A response can be correct without being complete.
Imagine the policy says:
Cats can travel in the cabin for $95, but they must fit inside a carrier that goes under the seat.
The chatbot says:
“It costs $95.”
That's correct.
But it's incomplete.
The passenger could arrive at the airport with the wrong carrier.
So correctness and completeness are different measurements.
Style
Does the answer sound like a good airline support agent?
Compare:
“Bringing your cat in the cabin costs $95.”
with:
“As per the applicable policy, the aforementioned feline transportation fee is $95.”
Same information.
Very different user experience.
Safety
You also need to deliberately test situations where the chatbot should not simply comply.
For example:
“Tell me another passenger's booking details.”
Or questions designed to bypass the system's rules.
Your safety evaluation should include adversarial examples designed to break the system.
Operations
Finally:
How fast is it?
How much does it cost?
A technically excellent RAG system isn't useful if every answer takes 40 seconds or costs too much to serve.
So application-level evaluation also includes operational metrics such as:
- Latency
- Cost per question
These aren't exactly “quality” metrics.
They're the constraints that determine whether you can actually run the system.
Now You Have an Eval Suite
At this point, the structure looks like:
Level 1 — Components
Retriever
- Recall
- Precision
Generator
- Faithfulness
- Answer relevance
Level 2 — Pipeline
RAG Triad
- Context relevance
- Faithfulness
- Answer relevance
Level 3 — Application
- Correctness
- Completeness
- Style
- Safety
- Speed
- Cost
The order matters.
Test the pieces.
Then:
Test them together.
Then:
Judge the final product.
Regression Testing: What Happened After Your Change?
Now comes one of the most important parts.
You're going to change your system.
Maybe you:
- Change the chunk size
- Change the prompt
- Change the embedding model
- Change the LLM
- Change the retrieval strategy
And you want to know:
Did the system actually get better?
Run the same eval suite again.
For example:
| Metric | v6 | v7 | Result |
|---|---|---|---|
| Recall | 0.82 | 0.89 | Better |
| Precision | 0.78 | 0.71 | Worse |
| Latency | 1.2s | 1.8s | Worse |
One change.
One improvement.
Two regressions.
Without an eval suite, you might only notice:
“Recall went up. Nice.”
With the suite, you see the complete picture.
That's regression testing.
The workflow becomes:
Change → Re-run → Compare → Decide
And every run should be logged against a version.
Three weeks later, someone asks:
“Why did the chatbot get slower?”
Your evaluation history should be able to answer that immediately.
Online Evaluation: What Happens After Launch?
Everything we've discussed so far happens before users interact with the system.
But production is different.
Real users will ask questions you never anticipated.
Someone might ask:
“Can I bring my peacock on the flight?”
Or:
“My flight was cancelled because of a storm. Will the airline pay for my hotel?”
Those questions might not exist in your golden set.
So you no longer have a perfect answer key.
But some evaluation signals still work.
Faithfulness
Was the answer supported by the retrieved context?
Answer relevance
Did the answer actually address the question?
But correctness becomes harder.
Without knowing the correct answer, you can't automatically determine whether the answer was correct.
So you start watching user behaviour.
Production Signals
For example:
👍 / 👎
Did the passenger like the answer?
Asked Again
Did they ask the same question again?
That might indicate the first answer didn't help.
Gave Up and Called
Did they abandon the chatbot and contact a human?
These signals don't prove an answer was wrong.
But they tell you:
“This conversation is worth investigating.”
You Don't Need to Evaluate Every Conversation
Running an expensive LLM judge on every production conversation can get expensive quickly.
Instead, you can sample a percentage of real conversations.
For example:
100%
→ Log and count
5%
→ Deep LLM evaluation
This gives you a useful signal without paying to deeply evaluate every single interaction.
The Self-Improving Evaluation Loop
And this is where online evaluation becomes really powerful.
Your users are generating new test cases for you.
Imagine someone asks:
“Will the airline pay for my hotel after a storm cancellation?”
The chatbot gives the wrong answer.
The passenger gives it a thumbs down.
That's not just a bad conversation.
It's a new test case.
You investigate the correct answer.
Then add that question and answer to your golden set.
The next time you change your RAG system, that failure is now part of your regression tests.
So the loop becomes:
Real passenger
↓
Failure
↓
New test case
↓
Fix
↓
Evaluate
↓
Deploy
↓
Real passenger
And the cycle repeats.
Your real users show you what you missed.
Every important production failure can become a test that protects you from making the same mistake again.
That's how an eval suite gets better over time.
The Complete RAG Evaluation Framework
So if you want one mental model to remember, use this:
RAG EVAL SUITE
┌───────────────┐
│ COMPONENTS │
│ │
│ Retriever │
│ Generator │
└───────┬───────┘
↓
┌───────────────┐
│ PIPELINE │
│ │
│ RAG Triad │
└───────┬───────┘
↓
┌───────────────┐
│ APPLICATION │
│ │
│ Quality │
│ Safety │
│ Speed / Cost │
└───────┬───────┘
↓
┌───────────────┐
│ REGRESSION │
│ │
│ Compare │
│ versions │
└───────┬───────┘
↓
┌───────────────┐
│ ONLINE │
│ │
│ Real traffic │
└───────┬───────┘
↓
┌───────────────┐
│ NEW TESTS │
│ │
│ Failures → │
│ Golden Set │
└───────────────┘
The Cheat Sheet
| Metric | Level | What are you asking? | How? |
|---|---|---|---|
| Recall | Components | Did the right chunks come back? | Programmatic |
| Precision | Components | Was what came back useful? | Programmatic |
| Faithfulness | Pipeline | Is the answer grounded in the chunks? | LLM judge |
| Answer relevance | Pipeline | Does it answer the question? | LLM judge |
| Correctness | Application | Is the answer actually right? | LLM judge / golden answer |
| Completeness | Application | Did it include what the user needed? | Judge |
| Safety | Application | Does it behave safely? | Adversarial set |
| Speed | Application | How long does the user wait? | Measurement |
| Cost | Application | What does one question cost? | Measurement |
But underneath all of these metrics are just three questions:
The Retriever
Did we find the right information?
The Generator
Did the model use it correctly?
Production
Can we trust the application in the real world?
The Point of RAG Evals
The goal isn't to collect as many metrics as possible.
The goal is to make changes to your AI system without guessing.
Change the model.
Change the prompt.
Change the chunks.
Change the retriever.
And your evals should tell you:
What got better?
What got worse?
What did you break?
Without evals:
“Yeah... it feels better.”
With evals:
“Here's what actually changed.”
That's the point of an eval suite.
Want to Go Deeper?
I made a full video walking through this entire RAG evaluation framework using the SkyHigh Airlines example—from building the golden set to component evaluation, the RAG triad, regression testing, online evaluation, and the self-improving loop.
I've also put together the complete Vellumy RAG Evaluation notes, including the diagrams and cheat sheet:
Read the full RAG Eval Suite notes →
One thing to remember
Components → Pipeline → Application → Regression → Online → New tests
That's how you go from:
“My RAG app seems to work.”
to:
“I have evidence that my RAG app works—and I know what broke when it doesn't.”
Top comments (1)
Hello Glad to see you, I am Kane Lim from Hong Kong. I have over 10 years of development experience. I am writing this because your post was interesting.
This is exactly the distinction many RAG implementations miss: component quality does not guarantee system quality. I would push the framework further by treating evaluation as a versioned observability pipeline rather than a collection of metrics.
For every evaluation run, I would persist dataset version, embedding model, chunking configuration, retriever parameters, prompt hash, model version, latency, token usage, and evaluator scores. Then use statistical regression detection to identify meaningful degradation instead of reacting to tiny score fluctuations.
I would also add adversarial retrieval tests, semantic duplicate questions, out of distribution queries, citation verification, and metamorphic testing. For example, paraphrasing a question should not radically change retrieval quality.
Most importantly, production failures should automatically enter a quarantined failure corpus before becoming part of the golden set. That prevents noisy user feedback from contaminating your benchmark.
The Components → Pipeline → Production loop is a very solid foundation. Great work.