Intro
Model providers keep shipping bigger context windows: 100k tokens, 200k, over a million in some cases. The marketing pitch is simple: paste in everything, the model will figure out what matters. It's a seductive idea, and it's the wrong default for anything that has to run in production.
Here's the gap. In a tutorial or a demo, "long context" means pasting a document and asking a question once. Nobody times it, nobody pays for the tenth call, and nobody notices what got ignored. In production, the same request runs thousands of times a day, against a system that has to be fast, cheap, and correct every single time. The capacity to accept 100k tokens says nothing about whether stuffing that much into every request is a good idea. Below are four failure modes that show up once the naive approach hits real traffic. (The scenarios are illustrative composites, not case studies from a specific client.)
The latency tax nobody notices until it's live
Say a support tool feeds the model the full history of a customer's last 40 tickets on every single message, "just in case it's relevant." In the demo, that's one call, and it feels instant. In production, every reply now takes 10-12 seconds instead of 2, because the model has to process tens of thousands of tokens before it writes a word. Users don't experience "more context." They experience a slow bot, and slow bots get abandoned mid-conversation.
The margin killer
Token costs compound in a way that's easy to miss in a pilot and brutal at scale. A team billing customers per seat, but paying per token behind the scenes, can end up with COGS that scale with context size, not with usage. If every request carries the same bloated payload regardless of whether that history is relevant, the unit economics quietly break as the customer base grows, well after the pricing model has already been sold.
Lost in the middle
Even when the tokens are technically "in the window," attention isn't uniform across it. Models are demonstrably better at using information near the start and end of a context than information buried in the middle. Picture a compliance assistant given a 100-page policy document, where the one answer relevant to a question sits on page 47. The model can have "read" every token and still miss it, because being in context and being attended to are not the same thing.
Noise that turns into hallucination
Retrieval systems that grab too much, "let's include the top 15 chunks just to be safe," aren't playing it safe. Excess context isn't neutral filler. Loosely related documents give the model more material to blend, misattribute, and confidently combine into an answer that sounds right and isn't. The system doesn't fail loudly. It fails convincingly.
Context management is data engineering wearing an LLM's clothes
Every one of these failure modes is a symptom of the same root cause: treating context as an unlimited bucket instead of a resource with a cost and a decay curve. The fix isn't a bigger window. It's a pipeline: chunking, filtering, ranking, summarizing, retrieving only what the current task actually needs. That's not prompt engineering. It's the same discipline that's underpinned good search and recommendation systems for a decade, applied to a new kind of consumer.
Build it or buy it
If context management is core to your product's differentiation, and it usually is once you're past the demo stage, it's worth owning the retrieval and filtering layer directly rather than treating it as an afterthought bolted onto whichever framework shipped a quickstart. Off-the-shelf vector stores and orchestration libraries are a reasonable starting point for early prototypes, but the ranking logic, the summarization strategy, and the decision about what's relevant to a given task are where the actual product value lives. That's not something you want to be entirely dependent on someone else's defaults for, once you're optimizing for cost and correctness at scale.
What's the worst context-bloat bug you've shipped or debugged? Curious what patterns other people have run into.
Top comments (0)