After building and evaluating dozens of retrieval systems, I stopped thinking retrieval was the real problem.
Over the last couple of years I've spent a lot of time building and evaluating RAG systems.
Some were quick prototypes.
Some powered internal knowledge bases.
Some eventually became parts of products.
And I kept seeing the same pattern.
A user would ask a perfectly reasonable question.
The search would work.
The LLM would answer confidently.
And the answer would still be wrong.
Not because the model hallucinated.
Not because embeddings were bad.
Not because vector search failed.
The system had simply retrieved the wrong kind of information.
A question every RAG system should answer
Imagine a company with a thousand engineering documents.
Architecture decisions.
Meeting notes.
RFCs.
Runbooks.
ADRs.
Internal documentation.
Now someone asks:
Why did we choose Kafka over RabbitMQ?
A typical RAG pipeline looks like this:
Documents
↓
Chunking
↓
Embeddings
↓
Vector Database
↓
Top-K Retrieval
↓
LLM
Everything seems reasonable.
The question becomes an embedding.
The nearest chunks are retrieved.
The chunks are added to the prompt.
The LLM generates an answer.
Simple.
Except this is often where things start going wrong.
Semantic similarity isn't the same as relevance
Suppose your knowledge base contains fifteen documents about Kafka.
Architecture guides.
Broker configuration.
Monitoring.
Deployment.
Developer documentation.
Vector search will happily retrieve them.
And technically, it's correct.
They are semantically close to the question.
But none of them explain why Kafka was chosen.
The actual answer might live inside a two-page Architecture Decision Record written eighteen months earlier.
That document mostly discusses RabbitMQ, latency, throughput, operational complexity, and load testing.
The word Kafka barely appears.
From the perspective of embeddings, it isn't particularly similar.
From the perspective of an engineer, it's exactly the document you needed.
That's the disconnect.
Search found similar documents.
It didn't find the decision.
Then we start optimizing
Once teams notice retrieval isn't great, the usual optimization cycle begins.
First we reduce Top-K.
Instead of retrieving 100 chunks, we retrieve 10.
Then we shrink chunk sizes.
500 tokens become 250.
250 become 150.
Then we add rerankers.
Cross-encoders.
Hybrid retrieval.
Bigger context windows.
I tried all of these.
Most of them help.
None of them completely solve the problem.
Because they're still optimizing retrieval inside the same mental model.
The document slowly disappears
One thing surprised me more than anything else.
The smaller our chunks became, the less context survived.
Eventually the system remembered paragraphs.
But forgot the document.
Headings disappeared.
Structure disappeared.
Arguments disappeared.
What remained was a collection of isolated text fragments.
I started calling this chunk amnesia.
It's not a scientific term.
Just a name for something I kept observing.
Humans don't search that way
This is where my thinking changed.
When I need information, I don't start by looking for a paragraph.
I usually ask myself:
Which document was this?
Then:
What decision did we make?
Only after that do I look for implementation details.
It's almost hierarchical.
Document
↓
Decision
↓
Evidence
↓
Implementation
Most RAG systems reverse that process.
They search for paragraphs first and hope context reconstructs itself.
Sometimes it does.
Often it doesn't.
Bigger context windows don't fix this
A common reaction is:
"Let's just send more context."
I don't think that's the answer either.
A 64K context window can hold far more text.
It doesn't magically restore structure.
The model still receives disconnected fragments.
More fragments don't automatically become a coherent narrative.
Large prompts reduce one bottleneck.
They don't eliminate the retrieval problem.
What changed my thinking
Eventually I stopped thinking of retrieval as a single search problem.
Instead, I started thinking about navigation across multiple levels of abstraction.
Before asking:
Which paragraph matters?
I first wanted to answer:
Which document matters?
Then:
Which decisions inside that document matter?
Only then should retrieval dive into individual chunks.
That's also how people naturally explore documentation.
We rarely jump directly into paragraph number seventeen.
We skim.
We orient ourselves.
Then we zoom in.
This isn't just an LLM problem
After a while I realized this discussion isn't really about language models.
It's about organizational memory.
Companies rarely lose information because documents disappear.
They lose the relationships between documents.
The ADR exists.
The meeting notes exist.
The design proposal exists.
The Slack discussion exists.
But the connections slowly fade.
Search retrieves text.
People are trying to recover context.
Those aren't always the same problem.
I'm still exploring this
I don't have a universal solution.
In fact, this realization is one of the reasons I started building Retineo.
Not because I thought I had the answer.
Because I couldn't find a system that treated organizational knowledge as something richer than independent chunks inside a vector database.
Maybe the answer isn't hierarchical retrieval.
Maybe it's something else entirely.
I'm still experimenting.
I'd love to hear your experience
If you've built RAG systems at scale, what's the failure mode you've seen most often?
For me, it isn't hallucination.
It's this:
The documents aren't missing.
The relationships between them are.
Discussion
Have you encountered similar problems?
Did hierarchical retrieval, knowledge graphs, rerankers, or another approach help?
I'm genuinely interested in how others are tackling this.
Top comments (0)