Anyone who has run a long-context model in production has met the same wall. The GPU has enough compute to generate tokens quickly, and not enough memory to hold the conversation state that makes those tokens coherent. The state is the KV cache, and at long context it grows faster than the model weights it is supposed to serve.
There are two ways out. You can move the cache to cheaper memory and read it back when needed, or you can throw it away and recompute it. Both are in production at scale, both are defended loudly by their proponents, and both have a failure mode that only shows up under real traffic.
Part One: How big the cache actually gets
For a dense transformer, the KV cache per token is:
2 x layers x kv_heads x head_dim x bytes_per_element
Take a 70-billion-parameter model with 80 layers, 8 key-value heads (grouped-query attention), a head dimension of 128, and FP16 storage. Per token that is 2 x 80 x 8 x 128 x 2 bytes, or 327,680 bytes, about 0.31 megabytes. At 8,000 tokens of context that is 2.5 gigabytes per conversation. At 128,000 tokens it is 40 gigabytes, which is more than the weights of many models people are actually serving.
That number explains the entire architecture of modern inference serving. Continuous batching works because it interleaves sequences; prefix caching works because it avoids recomputing the same system prompt; paged attention works because it stops fragmentation from wasting the cache. Every one of those techniques exists to manage a data structure that grows linearly with context and does not shrink.
Part Two: The offload path
The offload argument is straightforward. NVMe drives deliver 5 to 14 gigabytes per second of sequential read on a modern enterprise part, and a PCIe 5.0 drive can do it at a cost per gigabyte two orders of magnitude below HBM. If the cache is written once and read once per decode step, and the decode step is already taking tens of milliseconds, the transfer can hide inside the compute.
That last clause is doing a lot of work. Hiding a transfer inside compute requires that the transfer and the compute overlap, which requires that the cache access pattern is predictable, which requires that the scheduler knows which sequence will be decoded next. In a batching system with dynamic arrival, that prediction is exactly the thing you cannot make reliably. The result is a tail: most steps hide the transfer, and a minority wait for it, and the p99 latency is set by the minority.
The second problem is write amplification. Every generated token appends to the cache. A naive implementation writes the whole block again; a good implementation appends. The difference between those two designs is a factor of a hundred in write bandwidth at long context, and it is the difference between an offload path that works and one that melts the drive.
Part Three: The recompute path
The recompute argument is equally straightforward. If the cache is too expensive to keep, do not keep it. When a conversation resumes, run the prefill again over the history. A prefill pass over 8,000 tokens on a modern accelerator is a few hundred milliseconds of compute, and compute is the resource you bought the GPU for.
The failure mode here is different and, in some ways, worse. Recompute turns a memory problem into a compute problem that competes with the work you are already doing. Under load, the GPU is saturated with decode steps for active users, and a resumed conversation arrives as a burst of prefill that has to be scheduled ahead of them or behind them. Ahead means existing users see a latency spike; behind means the resuming user waits.
There is a subtle economic point buried here. Prefill and decode have very different arithmetic intensity. Prefill is compute-bound and efficient; decode is memory-bandwidth-bound and inefficient. A serving system that recomputes is choosing to spend its most efficient resource on work it has already done once. A serving system that offloads is choosing to spend its cheapest resource on a transfer that might not overlap. Neither is obviously right; the answer depends on the ratio of resumed sessions to new sessions, which is a workload property nobody publishes.
Part Four: The hybrid that actually ships
In practice, production systems do both, with a policy. The cache stays in HBM for the active set, spills to host DRAM for the warm set, and is dropped for the cold set, where dropping means recompute on return. The policy is usually an LRU variant with a cost model attached: keep a sequence if the expected reuse probability times the recompute cost exceeds the storage cost.
The interesting engineering problem is that the cost model needs to know the future. A user who comes back in thirty seconds and a user who never comes back look identical at eviction time. Every production system solves this with a heuristic, and every heuristic is wrong in a way that shows up as either wasted memory or a latency spike. This is why the serving frameworks that win are the ones that expose the policy as configuration rather than hard-coding it.
Part Five: What this means outside the datacenter
The long-context assistant on your phone is running the same trade at a smaller scale. Mobile memory is shared with the operating system, the camera pipeline and every other app, and the model cannot assume it will get two gigabytes for a cache. That is why on-device assistants quietly truncate context, and why the experience of "it forgot what I said five minutes ago" is not a model quality problem but a memory budget problem.
The same arithmetic explains why long-context API pricing is not linear in tokens. Providers charge more for input than the raw compute suggests because the cache has to live somewhere, and somewhere costs money every second, not just when it is used.
Market sizing
Inference serving is the fastest-growing part of the AI infrastructure market, with analysts putting 2026 spend somewhere between 60 and 90 billion dollars, of which memory and storage for serving is a meaningful slice. If context lengths keep doubling every eighteen months, as the model releases suggest, the cache becomes the dominant memory consumer in the fleet, and the offload market becomes a storage market. That is a strange sentence to write about GPUs, and it is where the numbers point.
The comparison that matters
Offload optimises for compute efficiency and pays in tail latency and drive endurance. Recompute optimises for memory simplicity and pays in wasted compute and scheduling contention. The right answer is a policy, not a technology, and the policy depends on a number most teams never measure: the fraction of requests that are resumptions rather than new conversations.
Measure that number first. Everything else is a guess about a workload you have not characterised.
SOL: HCqPmPJM3NVtS6YrmVaTZYnuP8XZ2PFgQZKT94VK1LkS
Creating this takes real effort — thank you for the support.
Top comments (0)