If you've ever had a long ChatGPT or Claude session where the model suddenly "forgets" something you told it ten minutes ago, you've hit the context window. It's not a bug and it's not the model getting confused — it's a hard architectural limit, and understanding it changes how you prompt.
The Core Concept
A context window is the total amount of text — measured in tokens, not words or characters — that a model can process in a single inference call. Anything outside that window is invisible to the model. It doesn't get "forgotten" the way a person forgets; it simply was never in front of the model for that call. Both your input (the prompt plus the entire conversation history) and the model's output count against the same limit.
Tokens and words aren't 1:1. A rough rule of thumb: one token is about three-quarters of an English word, so 1,000 tokens is roughly 750 words, or about a page and a half of prose.
Why AI "Forgets" in Long Conversations
When a conversation grows past the window size, older messages get dropped — typically from the beginning. The model isn't experiencing amnesia. It literally never receives those tokens on that particular call. If it seems to contradict something you said earlier in a long session, that's usually window pressure pushing your earlier instruction out of view, not the model changing its mind.
Context Window Sizes You'll Actually Work With
Sizes vary a lot by model, and they've grown fast — from GPT-3's 2,048 tokens in 2020 to windows over 1,000,000 tokens within a few years, according to model providers' own documentation.
The table below is a historical snapshot of the 2024 generation, not a current spec sheet. It's here to show the shape of the range, because that shape is what changes how you design. Do not build against these numbers — look up your model's current figure.
| Model (2024 generation) | Context window as published then | Roughly |
|---|---|---|
| GPT-3.5 Turbo | 16,385 tokens | ~12,000 words |
| GPT-4o | 128,000 tokens | ~96,000 words |
| Claude 3.5 Sonnet / Claude 3 Opus | 200,000 tokens | ~150,000 words |
| Llama 3.1 405B | 128,000 tokens | ~96,000 words |
| Mistral Large 2 | 128,000 tokens | ~96,000 words |
| Gemini 1.5 Pro / Flash | 1,000,000 tokens | ~750,000 words |
Two orders of magnitude separate the low end from the high end, and that gap has kept widening since. Every provider publishes the current number in its model reference page — that page, not a blog post, is your source of truth. Anything written down here has a shelf life measured in months.
Where This Bites You in Practice
- Paste a long PDF and ask questions → if the document plus your questions exceed the window, earlier parts silently get dropped from what the model actually sees.
- Long coding sessions → the model "forgets" function signatures or conventions you established early on.
- Multi-session research → you have to re-supply context in each new session; nothing persists across separate conversations by default.
- Mid-conversation contradictions → almost always window pressure, not the model changing its stance.
Workarounds That Actually Help
- Summarize and restart — paste a compact running summary at the top of a new conversation instead of the full history.
- Chunk the task — break a large job into sequential, focused sub-tasks (prompt chaining) rather than one giant request.
- Retrieval-Augmented Generation — instead of stuffing everything into context, an external index retrieves only the relevant passages per query. See our RAG explainer if you haven't used this pattern yet.
- System prompt anchoring — put your most critical, non-negotiable instructions in the system prompt. Most tool implementations prioritize it and are slower to drop it under pressure.
- Explicit recall prompts — when you sense drift, say something like the template below to force the model to re-register key facts before continuing:
Here's what we've established so far:
- [fact 1]
- [fact 2]
- [constraint or decision you don't want dropped]
Continue from here with: [next instruction]
That template alone fixes most "the model forgot my instruction" complaints in long sessions — it's cheaper than restarting the conversation and more reliable than hoping the model re-derives the constraint on its own.
FAQ
What is a context window in AI?
The total amount of text, measured in tokens, that a model can process — as input and output combined — in one inference call.
Why does ChatGPT forget things in long conversations?
It's not forgetting in the human sense. Once the conversation exceeds the context window, older messages fall outside what the model receives on that call.
How many tokens can a model handle?
It depends entirely on the model — anywhere from roughly 16K tokens on older models to over 1,000,000 tokens on the largest current windows. Check your provider's docs for the exact figure.
What happens when you exceed the context window?
Behavior varies by implementation, but typically the oldest messages are dropped or truncated to make room for new input, and the model simply never sees the dropped portion.
What's the difference between a token and a word?
A token is a chunk of text a model processes as one unit — often a word, but sometimes a word fragment or punctuation. In English, one token is roughly three-quarters of a word on average.
How do I work around a small context window?
Summarize periodically, chunk large tasks into smaller sequential prompts, use retrieval instead of stuffing everything into the prompt, and anchor critical instructions in the system prompt.
Originally published at my-blog.org.
Top comments (0)