A chatbot that behaves perfectly in short conversations and turns vague after twenty turns is rarely a model problem. It is a context window problem, and the reason it is hard to catch is that nothing throws an error when it happens.
What Actually Shares The Window
Every model call has one fixed token budget, and everything competes for it: the system prompt, few-shot examples, documents retrieved from a vector store, the full conversation history, and the response being generated right now.
Put numbers on it. A 4,000 token system prompt, a ten turn conversation that has accumulated 6,000 tokens, and 3,000 tokens of retrieved context add up to 13,000 tokens before the model writes a single word. On a 16,000 token model that leaves 3,000 tokens for the answer. Every feature you add to the prompt takes that space from somewhere else.
The Advertised Window Is Not The Usable One
Context windows have grown enormously, from 2,048 tokens in the GPT-3 era to 128k, 200k and 1M today, but growth shifted the economics rather than removing the constraint. Cost scales linearly with input tokens, so a 100k token prompt costs ten times a 10k one, and latency rises with it.
Quality moves too. Attention degrades in the middle of very long contexts, the effect usually called lost in the middle, where information near the center of a long prompt influences the answer less than the same information at the start or end. Measurable accuracy loss shows up once input passes about 40 percent of the maximum window, so an application on a 128k model should be designed around roughly 50k usable tokens rather than the number on the box.
Overflow Fails Silently
When you exceed the limit, behavior depends on the provider. Some return an error. Some truncate the input silently, dropping the oldest messages with no warning. A few truncate from the middle.
The silent cases are the dangerous ones. The system prompt gets cut to make room for history, so the model loses the instruction it was given and answers confidently anyway. In a RAG pipeline the retrieved documents get dropped because the template reserved too little space for dynamic content. In a coding assistant the file being edited falls out of context. Users describe all of this as the assistant getting dumber, which is why it so often gets misdiagnosed as a model regression.
Four Strategies That Hold Up
A sliding window with summarization keeps recent turns verbatim and compresses older ones into a paragraph. The quality of that summary determines long term coherence: "the user is building a REST API in Python with FastAPI and PostgreSQL" preserves what matters, "the user discussed technology choices" does not.
Selective retrieval uses the current query to pick the three to five most relevant previous messages regardless of recency. This beats a static window whenever the topic shifts, because a question about CSS should pull the frontend part of the conversation, not simply the last twenty messages.
Token-aware prompt design allocates a fixed budget per section and enforces it in code: so many tokens for the system prompt, so many for retrieval, so many for history, so many reserved for the response. When a section approaches its budget the matching reduction strategy runs. Overflow becomes impossible and cost becomes predictable, because usage is bounded by the budget instead of growing with the conversation.
External memory moves durable knowledge out of the prompt entirely and pulls back only what the current query needs. This is the one that scales without limit, since a model with a small window connected to a hundred thousand stored memories has access to far more than a huge window stuffed with everything inline. The trade is infrastructure: storage, retrieval logic, and memory management instead of a longer string. The full breakdown of context window management covers how these combine in production.
The Takeaway
Stop treating the prompt as the place where knowledge lives. Treat it as a budget you spend deliberately on every call, measure the token count before you send, and keep the things you always need out of the window and in a store you control. That single shift removes most of the failures people blame on the model.
Top comments (1)
The silent truncation issue gets especially messy once agent tool calls enter the loop. A single unformatted JSON response or dirty error stack from a tool can dump five thousand tokens into the history in one turn. When the provider quietly trims from the top or middle to fit the window, the agent loses its initial tool definitions or system guardrails, then spins on empty inputs without throwing an error. Hard clamping tool output payloads before appending them to the history buffer has saved me far more debugging time than tweaking window sizes.