DEV Community

Onkar Deokate
Onkar Deokate

Posted on

Every new model has a 1M-token context window. Do you still need retrieval?

Look at the model releases from October alone: DeepSeek V4.1 Flash, MiniMax M3.1 Flash, Kimi K3.1 Pro, Reflection's Beam 1.0, Mercury 3. One million tokens of context each. A year ago that was a headline feature. Now it's the default.

So the obvious question: why are we still building retrieval pipelines? Just put everything in the prompt.

Sometimes that's right. I changed one of my own designs for exactly that reason. But "the window is big enough" was never the only reason for retrieval.

When stuffing the context wins

In September I redid the maths on a text-to-SQL design. The whole database schema was about 20k tokens. The original plan retrieved the top 5 relevant tables per question, to save tokens.

Two things changed my mind:

  1. Cache reads got cheap. A fixed schema in a cached prefix costs very little per question.
  2. Retrieval was failing in a way I hadn't measured. A commenter, Reid, described it well: a search for "stale leads" pulls the leads table, but it drops the join table that links leads to activities, because the join table doesn't look like the question. The model then invents a join.

When the content is small, stable and connected, giving the model all of it beats making it guess which part matters.

When retrieval still wins

1. The data changes. A 1M window doesn't help if half of what you'd put in it is out of date. Retrieval reads current data at question time. A cached prefix is a snapshot.

2. You pay per token, every time. Caching helps a stable prefix. It doesn't help a context that's different for every user. A million tokens per request isn't cheap on any price list.

3. Latency. The time before the first token grows with the input. Fine for a background job, not for a search box.

4. Accuracy across a long context isn't flat. A model that accepts 1M tokens doesn't use every part of it equally well. Test your task at your size. Don't trust the number on the model card.

5. Permissions. Retrieval is where you enforce "this user can only see their own records". Put everything in the prompt and you've given everything to everyone who can phrase a clever question.

The rule I use now

Content Approach
Small, stable, connected (schemas, API specs, style guides) Put all of it in a cached prefix
Large, changing, or per-user Retrieve at question time
Mixed Cache the stable part, retrieve the rest, and put the cached part first

And the step people skip: measure both on your own questions. In the text-to-SQL case, the eval compares the rows each query returns against a reference query. Without that, both designs feel fine. With it, one is clearly better.

What I'd tell you before you start

A bigger window is a new option, not a new answer. Start with "what does the model need to see to get this right?", then pick the cheapest way to show it that. Sometimes that's the whole schema. Sometimes it's three rows.


Release details: "AI Model Releases: October 2026 Tracker", Digital Applied. My earlier post: "Opus 5.5 Made Cache Reads 60% Cheaper. I Redid the Math on My Text-to-SQL Architecture".

Top comments (0)