Every model release now advertises a context window in the millions of tokens. The number is real, and it is also mostly unusable at scale, because the constraint was never the window. It was the memory that the window occupies while the model is thinking.
A transformer keeps a key-value cache for every token it has already processed. That cache is what lets it attend to earlier text without recomputing it. It is also the fastest-growing cost in any long-conversation product, and it grows linearly with context while your revenue per conversation does not.
Part One: What the cache actually costs
For a 70B-parameter model in FP16, one million tokens of context needs roughly 160 GB of KV cache. A single accelerator has 141 GB. The cache does not fit, which means the serving system must either spread the conversation across several devices, evict and recompute, or refuse the request.
Spread across devices, the arithmetic gets worse. Attention at long context requires each device to see the keys and values of the others, so the interconnect carries cache traffic proportional to context length. At some point the network, not the compute, becomes the bottleneck.
This is why long-context demos usually run at batch size one. A batch of thirty-two conversations at 200,000 tokens each is not a bigger version of the same job. It is a different job, with different economics.
Part Two: Three ways to shrink the bill, and what each one costs
The industry has three practical answers.
Quantise the cache. Store keys and values at 8 bits or 4 bits instead of 16. This cuts memory four-fold and usually costs a small amount of quality, because the cache is read many times and small errors compound. It is the cheapest intervention and the one most widely deployed.
Evict and recompute. Keep only recent tokens plus a summary of the rest. This is what most production chat systems actually do, whatever the marketing says. The cost moves from memory to compute, and the quality loss shows up as forgetting — the model cannot answer a question about paragraph three after paragraph forty.
Change the attention structure. Grouped-query and multi-query attention share key-value heads across query heads, which shrinks the cache by a factor of four to eight at almost no quality cost. This is now standard, and it is the reason modern long-context models are feasible at all.
None of these is free. The question is which cost you can pay.
Part Three: The economics decide the architecture
Consider a product that summarises contracts. A contract is 80,000 tokens. A user asks six follow-up questions. If you keep the full cache resident for the whole session, you are holding 12 GB of accelerator memory per active user, and your cost per session is set by concurrency, not by tokens.
If instead you compress the cache to 4 bits and keep a rolling summary, memory drops to roughly 1.5 GB per session, and you can serve eight times as many users on the same hardware. The quality difference on a contract question is usually small, because the answer lives in a specific clause and the retrieval is local.
Now consider a product that reasons over a 500-page deposition. Here eviction is fatal, because the question may connect a statement on page 40 to one on page 400. Compression is the only option that preserves the whole text, and the quality loss is measurable but tolerable.
The same feature — a long window — is a different product depending on whether you can afford to keep the cache resident, and most teams discover this after the first invoice.
Part Four: What this means for builders
Three practical rules.
First, measure cache occupancy, not context length. A request that uses 200,000 tokens but only reads the first 2,000 still allocates cache for all 200,000 unless the serving stack is smart. Most are not.
Second, price long-context features per session, not per token. Token pricing assumes the marginal cost of a token is roughly constant. At long context it is not: the ten-thousandth token costs far more to attend to than the tenth, because it must attend to everything before it.
Third, treat the window as a budget you spend, not a limit you approach. The teams that ship long-context products profitably are the ones that retrieve aggressively and keep the resident cache small, then fall back to the full window only when the question demands it.
What is actually transferable
The pattern repeats across computing: a resource becomes cheap at the margin, so products are designed as if it were free at the aggregate. Memory bandwidth, then storage, then network, now attention cache. In each case the marginal price fell and the aggregate bill rose, because usage grew faster than the unit cost.
The teams that survive the transition are the ones that instrument the aggregate early — before the feature ships, not after the bill arrives. A context window is a promise about capability; the cache is the invoice that comes with it, and only one of the two appears on the pricing page.
The practical conclusion is unglamorous. Compress the cache, retrieve instead of stuffing, and measure occupancy per session. The models with the biggest windows will keep winning benchmarks, and the products with the smallest resident caches will keep winning margins.
SOL: HCqPmPJM3NVtS6YrmVaTZYnuP8XZ2PFgQZKT94VK1LkS
Creating this takes real effort — thank you for the support.
Top comments (0)