An LLM has no memory. Everything it "knows" about your conversation lives inside one finite window of text — and the moment something scrolls out of that window, it's gone. Understanding the context window explains most of what frustrates people about these models.
The context window is the model's entire working memory, and it behaves nothing like human memory. Here's what it is, why bigger isn't automatically better, and how to work with its limits.
What the context window actually is
Every time you call an LLM, you send it a block of text — the system instructions, the conversation so far, any retrieved documents, and the new question. That entire block has to fit inside the context window, measured in tokens. The model reads all of it fresh, every single call. It doesn't "remember" the previous turn; the previous turn is re-sent as part of the input.
This is the crucial mental shift: the model is stateless. What feels like memory in a chat is just the whole conversation being re-fed on every message. Run out of window, and the oldest content must be dropped — the model doesn't forget gracefully, it simply never sees what you truncated.
Bigger windows aren't a free lunch
Context windows have grown enormously, and it's tempting to think you can just stuff everything in. Two problems:
- Cost and latency scale with what you send. You pay per token and wait per token. A giant context on every call gets expensive and slow, fast.
- "Lost in the middle." Models reliably attend to the start and end of a long context and get hazy about the middle. Bury the key fact halfway through 100,000 tokens and the model may effectively miss it. More context can mean worse answers if it dilutes the signal.
So the skill isn't filling the window — it's curating it. Put the right information in, in the right place, and leave the rest out. It's a discipline I lean on across the AI systems I build.
This is why RAG exists
If the model can't hold your entire knowledge base in its window — and it can't — you need to fetch only the relevant slice at query time and place it in the context. That's retrieval-augmented generation in one sentence: don't expand the memory, curate what goes into it. The context window's limits are the whole reason retrieval is an architecture and not an afterthought.
Working with the window, not against it
- Summarize old turns instead of carrying them verbatim, so long conversations don't blow the budget.
- Put the most important instructions and facts at the edges — start and end — where attention is strongest.
- Retrieve, don't dump. Feed the model the few passages that matter, not everything you have.
- Track your token budget like you'd track memory in a constrained system, because that's exactly what it is.
Treat the context window as a scarce, carefully-managed resource and the model gets sharper. Treat it as infinite and it gets slow, expensive, and vague. More on how I manage it at www.divyakush.com.
Related reading
- RAG, explained properly: how retrieval keeps an LLM honest — the architecture built entirely around this limit.
- Fine-tuning vs RAG vs prompting: pick the right lever — when to fine-tune, retrieve, or just prompt.
- Vector databases: how semantic search scales — nearest-neighbor search at scale.
Divyakush Punjabi · Full-Stack & AI Engineer
Portfolio · GitHub · LinkedIn
Top comments (0)