AI is moving fast, and it feels like there's a new concept to learn every week. In an effort to actually understand this whole new world instead of just skimming past it, I've been writing ELI5 articles breaking down concepts that show up constantly in AI conversations but rarely get explained simply. This time, a term that gets thrown around a lot without much explanation: the context window.
AI 101 Recap
The context window is the total number of input and output tokens an LLM can consider while generating a response. Your prompt, the conversation history, and even the model's response all share that same "budget" of tokens. As a conversation with an LLM grows longer, more tokens live in this window.
So far so good. Every AI model has a limit on how many tokens it can hold in its "working memory" at once. So the interesting part with the context window is what happens when you reach these limits.
Why are there limits?
There are several reasons models have context window limits. Processing more tokens requires more memory and computation, making every request slower and more expensive. On top of that, today's models struggle to use very long contexts effectively. They naturally pay more attention to the beginning and end of a conversation than the middle ("lost in the middle" problem).
The Amnesia Problem
An AI model has no persistent memory between conversations. Within a long conversation, it only sees whatever still fits inside its context window. Think of it as a sliding window: as new messages come in, older ones eventually slide out and are no longer visible to the model.
This is why a long conversation can feel like the model has "forgotten" something you told it early on. It doesn't actually forget, it just no longer has access to that part of the conversation, unless the product you're using has built something extra on top to handle it.
How RAG helps
RAG (retrieval-augmented generation) sounds technical, but the idea is simple: look it up before you answer, instead of guessing from memory.
Think of the model as a smart intern who read a huge pile of general knowledge, but has never seen your company's internal docs. If you ask that intern a question about your product, you wouldn't expect them to know the answer off the top of their head.
The flow looks more or less like this:
That's RAG. Before generating a response, the system retrieves the most relevant pieces of information and adds only those to the prompt. Instead of searching through thousands of documents itself, the model receives just the information it needs.
This fixes two problems at once:
- It solves the "too much to fit" problem. You don't need to cram an entire wiki into the context window — just the relevant slice.
- It solves the "the model doesn't know that" problem. Training data has a cutoff and doesn't include your private docs. Retrieval hands the model current, specific facts instead of asking it to make something up.
It's not magic, though. If the search step grabs the wrong page the model's answer will be wrong too. A RAG system is only as good as its retrieval step, not just the model doing the writing at the end.
Why longer context isn't automatically better
Newer models advertise huge context windows, sometimes hundreds of thousands of tokens. It's tempting to think the fix for all of this is simple: just make the window bigger, and pile everything in.
Turns out that doesn't hold up as well as it sounds.
It costs more and takes longer. Every extra token in the prompt means more processing on every single request.
Models don't "read" a huge context evenly. They tend to pay attention to the start and the end of a long conversation, and quietly lose track of things in the middle.
More text means more noise. A huge pile of context gives the model more chances to get distracted by something irrelevant, outdated, or conflicting.
A bigger context window isn't automatically better. It gives the model more information to work with, but also more opportunities to be distracted. That's why RAG matters: retrieving a small set of relevant information is usually more effective than hoping the model can make sense of everything at once.
Resources
Context windows
- Anthropic's context window documentation — official, model-specific limits and behavior
- "Lost in the Middle: How Language Models Use Long Contexts" (Liu et al., 2023) — the research paper behind the finding that models underuse the middle of long contexts
- OpenAI's models documentation — useful for comparing how another provider frames the same constraint
RAG
- "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (Lewis et al., 2020) — the original RAG paper from Meta AI
- Pinecone's guide to RAG — practical, vendor-neutral explanation
- LangChain RAG documentation — hands-on look at how RAG pipelines are built
- AWS: "What is RAG?" — plain-language explainer
- IBM: "What is retrieval-augmented generation?" — another accessible, non-technical framing
Top comments (22)
Thanks for the analogy and explanation. Would you be open to create a new post explaining what loop engineering is and its usage?
This is not a concept is more like a practise, a while loop that let the ai run, test and self correct. I try to write about the concepts and algorithms that are around ai
I really liked the "smart intern" analogy. 😄 I also think a lot of people assume bigger context windows mean the model magically remembers everything. In reality, feeding more text isn't the same as giving it the right information. Good explanation for beginners without oversimplifying it.
I guess people cannot comprehend how "if this box answers me quite accurately doesn't have memories and doesn't remember much". It's very hard to explain easily that everything they see is a statistic probability
The "lost in the middle" point connects nicely to why RAG isn't just about fitting things in. Even when the whole doc fits, burying the answer at 50% depth can hurt you, so good retrieval also decides where in the prompt the relevant chunk lands, not only which chunk. Have you looked at re-ranking yet? It's the natural next ELI5 after this one.
I was caught up with reading about embeddings which are quite fascinating. Re-ranking will come in the future! ^_^
I enjoyed this explanation, especially the emphasis that a context window is closer to working memory than long-term memory.
One thing it made me think about is that we've started treating larger context windows as though they're a substitute for memory. I'm not convinced they are.
To me, the context window is really an execution surface. It's the information available for this reasoning step. Once the inference completes, that context disappears unless something intentionally preserves it.
That's where I think concepts like Memory as Infrastructure become interesting. Instead of assuming memory "just happens," the surrounding system decides what deserves to survive beyond the context window. Decisions, evidence, corrections, provenance, and reasoning history become durable architectural assets instead of transient prompt content.
I also wonder if this is where Prose Tax starts to matter. Bigger context windows certainly allow us to provide more information, but they don't eliminate the cost of recovering intent from verbose or ambiguous language. A million-token context full of loosely organized prose may still be less effective than a much smaller context containing precise, well-structured information.
A larger context window lets an agent think longer.
Memory as Infrastructure lets a system learn longer.
Those feel like related, but fundamentally different, architectural problems.
Exactly. Bigger context windows are a genuine technical improvement, but they're also an easy metric to market because they're simple to compare: 200k, 1M, 10M tokens. It's like megapixels in cameras. More isn't automatically better.
Speaking as the audience this article is actually for: I'm a freelancer in
Indonesia, I subscribe to a couple of chat products, and I have never built a
RAG pipeline in my life.
The amnesia section explained something I'd been misreading for a year. When a
long chat starts giving worse answers, my assumption was that the model was
somehow getting tired or that I'd annoyed it into being lazy. It genuinely felt
like that. Knowing it's a sliding window and the early instructions have simply
scrolled out is much less mysterious, and it changes what I do — start a fresh
chat and re-paste the constraints, instead of repeating myself louder in the
same one.
The intern analogy is the part I'll steal when clients ask me why the AI got
something wrong about their business.
One small thing from the cheap seats: for people at my level, the practical
version of "context window" is mostly "when do I start over." That's not in the
article, and maybe it doesn't belong, but it's the decision the concept actually
maps to for a non-builder.
Thanks, yes a clear window can help, claude also provide the ability to summarise the previous messages so it can free space, but the caveat is that it might lose information
Nice explanation! One distinction I’d add is that people often confuse the context window with memory. A context window is just the information available for a single inference, while memory (when a product supports it) is an application-layer feature that retrieves or stores information across sessions. Keeping those concepts separate helps explain why a model can have a huge context window and still not “remember” previous conversations by itself.
References:
Definitely worth mentioning! Thanks!
Nice explanation. One nuance worth adding is that a context window is closer to the model’s temporary workspace than human memory. The model does not necessarily retain everything inside it equally well, and different products may truncate, summarise or retrieve earlier messages rather than simply letting them slide out.
This is also why huge context windows do not remove the need for RAG or persistent memory. Context determines what the model can see right now whereas RAG determines what information should be brought in and memory systems decide what may be useful again later. They solve related, but different problems.
the "lost in the middle" section is the one that changes how you structure prompts in prod. ran evals where the same fact at position 30% vs 80% in a 50k context had a 15% accuracy gap on retrieval questions — "200k context = just cram everything in" is genuinely bad advice for tasks that need precision on a specific buried fact.
the fix we landed on: anchor critical info right before the instruction. retrieval at the start for richness, the specific thing the model must not miss right at the end.
did you cover the attention distribution research in a followup, or is this series staying more conceptual?
This series is for beginners to understand the basic concepts of how AI systems work.
The RAG caveat at the end is right but understates the failure mode. Good retrieval fixes the "wrong information" problem, it doesn't fix lost-in-the-middle on its own. If retrieval hands the model twenty relevant chunks and you dump all twenty into the prompt, you've just moved the amnesia problem downstream, now it's happening inside the retrieved set instead of the raw conversation.
The part that actually holds up in practice is ranking and trimming what retrieval returns before it goes into the prompt, not just getting the retrieval itself accurate.
That's a fair point, and I agree. This article is intentionally focused on explaining what a context window is, not on RAG or retrieval strategies. I wanted to keep the mental model simple before introducing concepts like ranking, reranking, chunk selection, and other techniques that make RAG effective in practice. Those definitely deserve an article of their own.
Makes sense to scope it that way, the mental model has to exist before the retrieval failure modes are legible anyway. Looking forward to the RAG follow-up, that's the part where "more context" and "better context" stop being the same claim.
The sliding-window framing is where the ELI5 quietly diverges from what actually happens in most chat products. A raw API call doesn't slide anything — if you exceed the limit, it errors out. The "old messages disappear" behavior is something the product implements on top: it truncates or summarizes history before it hits the model. That distinction matters because it means the forgetting isn't a law of physics, it's a policy decision someone made, and different apps make it differently. Some drop the oldest turns, some summarize them, some pin the system prompt and evict the rest. When a user complains the model "forgot," the bug is usually in that eviction logic, not the model.
Worth adding a caveat to the RAG section too: retrieving the wrong page is the loud failure, but the quiet one is retrieving the right page plus four mediocre ones. That's exactly the "lost in the middle" problem you cited biting you again — the correct chunk lands in position 3 of 5 and gets underweighted. So retrieval quality isn't only precision at rank 1; it's how aggressively you prune before stuffing the context. More retrieved chunks is the same trap as a bigger window, just wearing a RAG costume.
Those are good nuances, but I think we're talking about different goals. This article is an ELI5 introduction, not a deep dive into LLM infrastructure.
To explain context windows to someone who's never heard of them, simplifying implementation details is intentional. Going into API behavior, eviction policies, chat product architectures, retrieval ranking, and lost-in-the-middle would make the article harder to understand than the concept it's trying to explain.
All of the points you raised are valid in a more advanced discussion, but they're beyond the scope of an introductory article. Sometimes the goal is to build the right mental model first, then explore the edge cases later.