If you've worked with RAG, you've probably seen this:
The answer is wrong.
So you change the prompt.
Try another model.
Increase the context window.
Maybe even switch to a bigger LLM.
And somehow...
the answer is still wrong.
I've started to think that we sometimes look at the wrong part of the system.
The problem may have started before the LLM ever saw the question.
First, look at the whole flow
A simple RAG system can be thought of as:
User Question
↓
Retrieval
↓
Context
↓
LLM
↓
Answer
The LLM is only one part of this.
Before it generates anything, the system has already made several decisions:
- What information should be searched?
- How were the documents split?
- Which results are relevant?
- How many results should be returned?
- Should some results be filtered out?
- Which results should appear first?
- What finally goes into the model's context?
So when the final answer is bad, the model isn't necessarily where things went wrong.
Here's a simple example
Imagine you're building an internal support assistant.
Someone asks:
“What's our refund policy for prepaid orders?”
The company has the answer somewhere in its documentation.
Your LLM is capable of understanding the policy.
But your retrieval system returns these:
Refunds are available for eligible purchases.
Customers can contact support regarding refunds.
Refund requests are reviewed within 5 business days.
Prepaid orders are processed immediately.
Refunds may be issued to the original payment method.
Everything looks relevant.
But there's one problem:
The actual rule for prepaid orders wasn't retrieved.
Now the LLM has incomplete information.
It might still produce a very confident answer.
And we might say:
“The LLM hallucinated.”
Maybe.
But the problem actually started earlier.
We gave the model the wrong context.
Where can things go wrong?
There are a few common places.
1. The document was split badly
A policy might say:
Customers can request a refund within 30 days.
And immediately after:
Prepaid orders are subject to different conditions.
If those two statements are split into different chunks, retrieval might find one without the other.
The model gets half the story.
2. The right document wasn't retrieved
This is probably the easiest one to overlook.
The answer may be sitting inside your knowledge base.
But your search never brings it back.
So you:
change the prompt → change the model → change the temperature → try again
...while the correct document is still sitting in the database.
Sometimes the model isn't failing.
The system simply didn't give it the information it needed.
3. Similar doesn't always mean useful
Suppose your system retrieves the five most similar chunks.
That sounds reasonable.
But imagine all five are about refunds while the one chunk containing the actual prepaid-order exception is ranked sixth.
You've technically retrieved relevant information.
But you haven't retrieved the right information.
That's where things like better search, filtering and re-ranking start becoming important.
4. Old information can look very relevant
Imagine your knowledge base contains:
2022 Refund Policy
2023 Refund Policy
2024 Refund Policy
2025 Refund Policy
A question about the 2025 policy might still retrieve the 2023 document because the wording is almost identical.
This is where metadata can matter:
date → source → department → document type → version
Sometimes the system needs more than semantic similarity to find the right answer.
So... should we just use a bigger model?
Not necessarily.
A bigger model can certainly help with reasoning and generation.
But it can't reliably answer from information it never received.
Think about it this way:
Bad context
↓
Good model
↓
Bad / unreliable answer
versus:
Good context
↓
Appropriate model
↓
Much better chance of a useful answer
That's why I'm starting to look at RAG less as:
“Search + LLM”
and more as:
“Getting the right information to the right model at the right time.”
What I would check first
If a RAG application is giving poor answers, I'd be tempted to check the retrieval pipeline before immediately replacing the LLM.
A simple checklist:
1. Retrieval
Did we actually find the right information?
2. Chunking
Did we split the information in a useful way?
3. Ranking
Did the useful result make it near the top?
4. Metadata
Are we filtering by things like date, source or version?
5. Context
Is the model actually receiving enough of the right information?
6. Generation
Only then — is the model struggling to turn that context into a good answer?
The Chapter 2 material I've been going through goes quite deep into these areas, including embeddings, vector databases, chunking, hybrid search, re-ranking and retrieval evaluation.
The bigger takeaway
The more I work through this, the more I think:
Building a good RAG system isn't just about choosing a smart model.
It's about building a good path between:
the question → the information → the model → the answer.
If the information going into that path is wrong, incomplete or badly ranked, throwing a smarter model at the end of it may not solve much.
And that's probably the part of RAG I find most interesting right now.
What has caused more trouble in your RAG projects?
Retrieval or generation?
I'd genuinely be interested to hear what others have run into.
Top comments (0)