Your model says it has a one-million-token context window. Its real working memory is a lot smaller than that.
On long-context benchmarks, models start failing well below the number printed on the box. And here's the part nobody warns you about: past a certain point, adding more context makes your answers worse. Not better. Worse.
Prefer to watch? Full walkthrough with the attention-cost animation:
A context window is not memory
Let's kill one idea first. A context window is not memory.
The model doesn't remember your last message — it's stateless. It keeps nothing between calls. Every single call, you send the whole conversation again. All of it.
Look at what you actually ship over the wire:
- Turn one: a system prompt and one question. Small.
- Turn ten: the same system prompt, plus nine questions, plus nine answers, plus the new one.
The window is just how much text fits in that box before the API says no. And everything in the box gets processed. Every time.
What attention actually does
Inside the model, attention has a job that's simple to say: every token looks at every other token and decides how much it cares about each one.
- 10 tokens → 100 pairs
- 1,000 tokens → 1,000,000 pairs
- 100,000 tokens → 10,000,000,000 pairs
Ten times the context isn't ten times the work. It's a hundred times the work. (That's just arithmetic: cost grows with the square of the input.)
That's also why the first reply in a long chat feels slow and the rest feel fast — the model caches the keys and values it already computed. But the cache saves you compute. It doesn't save your answer quality. That's a separate problem, and it's the one you actually feel.
Here are the three reasons big context hurts.
Reason 1 — attention is a budget
For every token, the attention weights add up to one. Always one — that's what softmax does.
So when you paste in fifty more documents, you don't hand the model more focus. You split the same focus into more pieces.
Think about search on your laptop. One folder, ten files — you find it in a second. Same search, ten thousand files, same query, and the right file is now somewhere on page four. The model has that exact problem. Except it never shows you page four. It just answers.
Reason 2 — lost in the middle
Researchers took one correct answer and moved it around inside a long prompt. Same prompt, same question, different position. Then they measured accuracy.
- Near the beginning: good.
- Near the end: good.
- In the middle: much worse.
(Liu et al., Lost in the Middle.) So your most important paragraph, sitting at 60% of a long prompt, is in the worst possible spot in the whole file.
Reason 3 — distractors in your own data
This is the one I hit most in real code. The famous needle-in-a-haystack test is easy, because the needle looks nothing like the hay. Your production data is not like that.
You've got version two of the doc. Version three. An old changelog. And a chat thread that contradicts all three. Four chunks that all look right. One of them is right — and the model can't tell which.
It gets worse as the input grows: accuracy drops steadily with input length, even on tasks the same model nails at short length. So the number on the box tells you what fits. It doesn't tell you what works.
Four things you can actually change
1. Send less, but better. If you're doing retrieval, stop pushing the top fifty chunks into the prompt. Search wide, re-sort the results, then send the top three to five. Fewer clean chunks beat more chunks, every time.
2. Use the edges. Instruction at the top. Data in the middle. Then repeat the actual question at the bottom, right before the model answers. That last line costs you nothing, and it works.
3. Stop reusing one giant thread. Split the job into steps. Each step gets a fresh, small context with only what it needs. Between steps, pass a short summary — not the full transcript. That's most of what agent frameworks are doing for you under the hood.
4. Measure it on your own data. Take twenty real questions from your app. Run them with 2,000 tokens of context, then run the same twenty with 20,000. Compare the answers side by side. You'll find your own limit — and it'll be lower than the spec sheet.
The takeaway
The context window is a hard limit on what fits. It was never a promise about what the model can use. Treat it like bandwidth you're paying for, and send the smallest thing that answers the question.
What's the biggest prompt you've ever shipped to production? And when you cut it down, did it get better or worse? Tell me below.
I make Vlad's Stack — how the AI tools you use every day actually work, for people who write code. Full video walkthrough is above.
Top comments (0)