The context window number in your model's docs is a capacity spec, not a promise. A model that advertises 200K tokens will happily accept 200K tokens, but the quality of what it does with them starts sliding long before you reach that ceiling. If you build production features assuming the whole window works equally well, you will ship bugs that only show up on long inputs. Here is how token budgets actually behave, and how to stop the window from quietly lying to you.
The number on the box is not the number you get
Advertised context windows describe capacity, not usable quality. Work on long context behavior shows models paying far more attention to the beginning and end of the input, while content buried in the middle gets missed even when you stay well inside the stated limit. People call this the lost in the middle problem, and it comes from how attention works, not from a bug you can prompt your way around.
The gap is wider than most teams expect. Models claiming a 200K window show measurable quality degradation around 130K tokens in practice. That is not the model refusing to answer. It is the model getting quietly worse at using the tokens you paid to send. If a critical instruction sits in the middle of a huge prompt, treat it as maybe read, not definitely read.
Everything shares one budget
The biggest misconception is that the context window is only for your input. It is not. The limit applies to the total of input and output tokens combined. Your system prompt, the conversation history, any retrieved documents, the user query, and the model's own response all draw from the same pool.
That has a consequence people hit constantly. A generous system prompt plus a long chat history can leave almost no room for the answer. The model does not warn you first. It runs out of budget mid thought and the response gets truncated, or the API rejects the request outright. Once you accept that output competes with input for the same space, you stop being surprised by cut off answers on your longest sessions.
Why big contexts get slow and expensive
Long prompts do not just risk quality. They cost you time and money on every call. The attention step compares every token to every other token, so the core computation grows with the square of the input length. The QK^T matrix is n by n, which means doubling your context roughly quadruples the work the model has to do. One study measured a 7x latency increase at 15,000 words of context. That is the difference between a snappy reply and a spinner your users abandon.
Cost follows the same curve, because LLM APIs charge per token for both input and output. Every extra token of history or retrieved context is money you spend on every single request, whether or not it earned its place. If your bills keep climbing, oversized prompts are usually part of the story, and trimming them is one of the fastest ways to reduce inference costs without changing your model.
Count tokens before you send them
You cannot manage a budget you never measure. Before firing a request, count what each part of the prompt actually costs. This one habit surfaces the system prompt bloat and runaway history that silently eat your window.
import { encoding_for_model } from "tiktoken";
const enc = encoding_for_model("gpt-4o");
const count = (text) => enc.encode(text).length;
const parts = {
system: count(systemPrompt),
history: count(conversation.map((m) => m.content).join("\n")),
docs: count(retrievedChunks.join("\n\n")),
query: count(userQuery),
};
const inputTokens = Object.values(parts).reduce((a, b) => a + b, 0);
console.log(parts, "input total:", inputTokens);
Run this once against a real session and you will usually find one part hogging the budget. Nine times out of ten it is either a bloated system prompt nobody has trimmed in months, or an unbounded history that grows every turn.
Watch the budget in production
Local counting is step one. In production you want the budget checked on the hot path and an alert before you hit the wall, not after. The rule that works: log token usage on every call, and fire an alert when usage crosses 80 percent of the context limit. That gives you room to react before requests start failing.
const CONTEXT_LIMIT = 128000; // set this to your model's real limit
function checkBudget({ inputTokens, maxOutputTokens }, requestId) {
const projected = inputTokens + maxOutputTokens;
const usage = projected / CONTEXT_LIMIT;
console.log(JSON.stringify({
requestId,
inputTokens,
maxOutputTokens,
usagePct: Math.round(usage * 100),
}));
if (usage > 0.8) {
notifyOncall(`Context at ${Math.round(usage * 100)}% on ${requestId}`);
}
return usage <= 1;
}
When you cross the line, the fix is rarely a bigger model. It is sending less. Summarize old turns, drop stale retrieved chunks, and lean on retrieval so you fetch only the passages a query needs instead of stuffing everything into the prompt. If you are wiring up that retrieval layer, a solid RAG to manage context setup does most of that trimming for you.
Three things to verify right now
- Log the token count of your system prompt today. If it runs over a couple thousand tokens, it is probably carrying instructions you no longer need.
- Put the beginning and end of your prompt to work. Move the single most important instruction to the very top or the very bottom, never the middle.
- Add the 80 percent usage alert before your next deploy, so the window tells you it is nearly full instead of failing silently.
None of this needs a fancy platform. A token counter, one budget check, and an 80 percent alert will catch most context problems before your users do. If you want to see what oversized prompts are doing to your bill, run the numbers through the LLM pipeline cost calculator and size the budget against real pricing.
If you want a deeper look at cutting your model bill, I cover it in more detail on my site.
If you want this wired up on your own site end to end, that is exactly the kind of work I take on.
Drop a comment if your setup looks different. Curious what token budgets people are actually running in production.
Top comments (1)
Token budget is a product constraint, not just a model number. The useful question is what context the system can reliably carry into the decision that matters. Once summarization, retrieval, and tool output enter the loop, nominal window size becomes only one part of the real budget.