AI is moving fast, and it feels like there's a new concept to learn every week. In an effort to actually understand this whole new world instead of just skimming past it, I've been writing ELI5 articles breaking down concepts that show up constantly in AI conversations but rarely get explained simply. This time, a term that gets thrown around a lot without much explanation: the context window.
AI 101 Recap
The context window is the total number of input and output tokens an LLM can consider while generating a response. Your prompt, the conversation history, and even the model's response all share that same "budget" of tokens. As a conversation with an LLM grows longer, more tokens live in this window.
So far so good. Every AI model has a limit on how many tokens it can hold in its "working memory" at once. So the interesting part with the context window is what happens when you reach these limits.
Why are there limits?
There are several reasons models have context window limits. Processing more tokens requires more memory and computation, making every request slower and more expensive. On top of that, today's models struggle to use very long contexts effectively. They naturally pay more attention to the beginning and end of a conversation than the middle ("lost in the middle" problem).
The Amnesia Problem
An AI model has no persistent memory between conversations. Within a long conversation, it only sees whatever still fits inside its context window. Think of it as a sliding window: as new messages come in, older ones eventually slide out and are no longer visible to the model.
This is why a long conversation can feel like the model has "forgotten" something you told it early on. It doesn't actually forget, it just no longer has access to that part of the conversation, unless the product you're using has built something extra on top to handle it.
How RAG helps
RAG (retrieval-augmented generation) sounds technical, but the idea is simple: look it up before you answer, instead of guessing from memory.
Think of the model as a smart intern who read a huge pile of general knowledge, but has never seen your company's internal docs. If you ask that intern a question about your product, you wouldn't expect them to know the answer off the top of their head.
The flow looks more or less like this:
That's RAG. Before generating a response, the system retrieves the most relevant pieces of information and adds only those to the prompt. Instead of searching through thousands of documents itself, the model receives just the information it needs.
This fixes two problems at once:
- It solves the "too much to fit" problem. You don't need to cram an entire wiki into the context window — just the relevant slice.
- It solves the "the model doesn't know that" problem. Training data has a cutoff and doesn't include your private docs. Retrieval hands the model current, specific facts instead of asking it to make something up.
It's not magic, though. If the search step grabs the wrong page the model's answer will be wrong too. A RAG system is only as good as its retrieval step, not just the model doing the writing at the end.
Why longer context isn't automatically better
Newer models advertise huge context windows, sometimes hundreds of thousands of tokens. It's tempting to think the fix for all of this is simple: just make the window bigger, and pile everything in.
Turns out that doesn't hold up as well as it sounds.
It costs more and takes longer. Every extra token in the prompt means more processing on every single request.
Models don't "read" a huge context evenly. They tend to pay attention to the start and the end of a long conversation, and quietly lose track of things in the middle.
More text means more noise. A huge pile of context gives the model more chances to get distracted by something irrelevant, outdated, or conflicting.
A bigger context window isn't automatically better. It gives the model more information to work with, but also more opportunities to be distracted. That's why RAG matters: retrieving a small set of relevant information is usually more effective than hoping the model can make sense of everything at once.
Resources
Context windows
- Anthropic's context window documentation — official, model-specific limits and behavior
- "Lost in the Middle: How Language Models Use Long Contexts" (Liu et al., 2023) — the research paper behind the finding that models underuse the middle of long contexts
- OpenAI's models documentation — useful for comparing how another provider frames the same constraint
RAG
- "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (Lewis et al., 2020) — the original RAG paper from Meta AI
- Pinecone's guide to RAG — practical, vendor-neutral explanation
- LangChain RAG documentation — hands-on look at how RAG pipelines are built
- AWS: "What is RAG?" — plain-language explainer
- IBM: "What is retrieval-augmented generation?" — another accessible, non-technical framing
Top comments (1)
Nice explanation! One distinction I’d add is that people often confuse the context window with memory. A context window is just the information available for a single inference, while memory (when a product supports it) is an application-layer feature that retrieves or stores information across sessions. Keeping those concepts separate helps explain why a model can have a huge context window and still not “remember” previous conversations by itself.
References: