DEV Community: ValeryKot

Bigger Context Windows Didn't Make Our RAG Smarter

ValeryKot — Wed, 08 Jul 2026 10:23:41 +0000

We stopped measuring retrieval quality by how many tokens we could fit into the prompt.

When long-context models became available, many of us made the same assumption.

If an LLM can read 128K tokens, retrieval suddenly feels less important. Why spend time carefully selecting documents if the model can simply read everything?

It sounds reasonable.

In practice, it wasn't.

More context, worse answers

Imagine asking your internal assistant:

Why did we abandon microservices?

Retrieval returns thirty documents.

an architecture decision record
a few Jira tickets
Slack discussions
meeting notes
a glossary page

Everything is related.

Almost nothing answers the question.

The actual decision lives in a single ADR written months earlier. It explains the trade-offs: team size, latency, deployment complexity, operational cost.

But that document isn't especially similar to the query. It doesn't repeat the same vocabulary. It doesn't even mention "microservices" very often.

So it gets buried.

The model now receives thirty relevant documents and does what language models are very good at: it produces a coherent explanation.

The problem is that coherence is not the same thing as faithfulness.

Instead of recovering the original decision, it often synthesizes one from recurring themes across the retrieved documents.

The answer sounds plausible.

It just isn't the answer that was originally made.

Bigger windows don't fix retrieval

Research has already shown that models struggle with information buried inside very long contexts. The Lost in the Middle paper is probably the best-known example.

Our experience suggested something slightly different.

Sometimes the answer isn't lost because the context is long.

It's lost because the retrieval stage couldn't distinguish the document that contains the decision from documents that merely discuss the same topic.

Adding more context doesn't necessarily solve that problem.

Sometimes it simply gives the model more material to average together.

We were optimizing the wrong thing

For a while we treated retrieval as a packing exercise.

How many useful chunks can we fit into the prompt?

Over time the question changed.

Why is this document here?

Should it be here at all?

Does it explain the decision, or does it merely mention the same technology?

Those questions turned out to matter much more than the size of the context window.

Retrieval is a selection problem

The biggest shift wasn't moving from 8K to 128K tokens.

It was realizing that retrieval isn't about fitting more information into a prompt.

It's about selecting the few pieces of information that actually explain the answer.

Large context windows are incredibly useful.

They just don't compensate for weak retrieval.

If anything, they make weak retrieval look convincing.

Next time I'll look at another assumption I no longer believe: that documents should be treated as bags of chunks.

Why RAG Finds Documents but Still Misses the Answer

ValeryKot — Mon, 06 Jul 2026 10:22:20 +0000

After building and evaluating dozens of retrieval systems, I stopped thinking retrieval was the real problem.

Over the last couple of years I've spent a lot of time building and evaluating RAG systems.

Some were quick prototypes.

Some powered internal knowledge bases.

Some eventually became parts of products.

And I kept seeing the same pattern.

A user would ask a perfectly reasonable question.

The search would work.

The LLM would answer confidently.

And the answer would still be wrong.

Not because the model hallucinated.

Not because embeddings were bad.

Not because vector search failed.

The system had simply retrieved the wrong kind of information.

A question every RAG system should answer

Imagine a company with a thousand engineering documents.

Architecture decisions.

Meeting notes.

RFCs.

Runbooks.

ADRs.

Internal documentation.

Now someone asks:

Why did we choose Kafka over RabbitMQ?

A typical RAG pipeline looks like this:

Documents
↓
Chunking
↓
Embeddings
↓
Vector Database
↓
Top-K Retrieval
↓
LLM

Everything seems reasonable.

The question becomes an embedding.

The nearest chunks are retrieved.

The chunks are added to the prompt.

The LLM generates an answer.

Simple.

Except this is often where things start going wrong.

Semantic similarity isn't the same as relevance

Suppose your knowledge base contains fifteen documents about Kafka.

Architecture guides.

Broker configuration.

Monitoring.

Deployment.

Developer documentation.

Vector search will happily retrieve them.

And technically, it's correct.

They are semantically close to the question.

But none of them explain why Kafka was chosen.

The actual answer might live inside a two-page Architecture Decision Record written eighteen months earlier.

That document mostly discusses RabbitMQ, latency, throughput, operational complexity, and load testing.

The word Kafka barely appears.

From the perspective of embeddings, it isn't particularly similar.

From the perspective of an engineer, it's exactly the document you needed.

That's the disconnect.

Search found similar documents.

It didn't find the decision.

Then we start optimizing

Once teams notice retrieval isn't great, the usual optimization cycle begins.

First we reduce Top-K.

Instead of retrieving 100 chunks, we retrieve 10.

Then we shrink chunk sizes.

500 tokens become 250.

250 become 150.

Then we add rerankers.

Cross-encoders.

Hybrid retrieval.

Bigger context windows.

I tried all of these.

Most of them help.

None of them completely solve the problem.

Because they're still optimizing retrieval inside the same mental model.

The document slowly disappears

One thing surprised me more than anything else.

The smaller our chunks became, the less context survived.

Eventually the system remembered paragraphs.

But forgot the document.

Headings disappeared.

Structure disappeared.

Arguments disappeared.

What remained was a collection of isolated text fragments.

I started calling this chunk amnesia.

It's not a scientific term.

Just a name for something I kept observing.

Humans don't search that way

This is where my thinking changed.

When I need information, I don't start by looking for a paragraph.

I usually ask myself:

Which document was this?

Then:

What decision did we make?

Only after that do I look for implementation details.

It's almost hierarchical.

Document
↓
Decision
↓
Evidence
↓
Implementation

Most RAG systems reverse that process.

They search for paragraphs first and hope context reconstructs itself.

Sometimes it does.

Often it doesn't.

Bigger context windows don't fix this

A common reaction is:

"Let's just send more context."

I don't think that's the answer either.

A 64K context window can hold far more text.

It doesn't magically restore structure.

The model still receives disconnected fragments.

More fragments don't automatically become a coherent narrative.

Large prompts reduce one bottleneck.

They don't eliminate the retrieval problem.

What changed my thinking

Eventually I stopped thinking of retrieval as a single search problem.

Instead, I started thinking about navigation across multiple levels of abstraction.

Before asking:

Which paragraph matters?

I first wanted to answer:

Which document matters?

Then:

Which decisions inside that document matter?

Only then should retrieval dive into individual chunks.

That's also how people naturally explore documentation.

We rarely jump directly into paragraph number seventeen.

We skim.

We orient ourselves.

Then we zoom in.

This isn't just an LLM problem

After a while I realized this discussion isn't really about language models.

It's about organizational memory.

Companies rarely lose information because documents disappear.

They lose the relationships between documents.

The ADR exists.

The meeting notes exist.

The design proposal exists.

The Slack discussion exists.

But the connections slowly fade.

Search retrieves text.

People are trying to recover context.

Those aren't always the same problem.

I'm still exploring this

I don't have a universal solution.

In fact, this realization is one of the reasons I started building Retineo.

Not because I thought I had the answer.

Because I couldn't find a system that treated organizational knowledge as something richer than independent chunks inside a vector database.

Maybe the answer isn't hierarchical retrieval.

Maybe it's something else entirely.

I'm still experimenting.

I'd love to hear your experience

If you've built RAG systems at scale, what's the failure mode you've seen most often?

For me, it isn't hallucination.

It's this:

The documents aren't missing.

The relationships between them are.

Discussion

Have you encountered similar problems?

Did hierarchical retrieval, knowledge graphs, rerankers, or another approach help?

I'm genuinely interested in how others are tackling this.