Most production prompts begin with the same few thousand tokens: a system instruction, a policy block, a schema, a handful of examples. recomputed prompt prefixes pays the full prefill cost for that block on every request. shared prefix caches keeps the resulting key-value tensors in accelerator memory and reuses them. On a slide the saving is a straight percentage of prefill FLOPs. In a serving fleet the saving is governed by a hit rate, by how much HBM a cache is allowed to occupy, and by who is allowed to see whose prefix.
With recomputed prompt prefixes, every request is self-contained. The prefill pass reads the whole prompt, produces one KV block per layer, and then decoding begins. There is no cross-request state, no eviction policy and no isolation boundary to enforce. With shared prefix caches, a hash of the token prefix is used to look up stored KV blocks; a hit skips the corresponding attention and feed-forward work. That lookup is cheap, but the stored blocks are large: a few thousand tokens across dozens of layers can consume hundreds of megabytes of HBM for a single shared prefix.
A prefix cache is a memory trade, not a free FLOP reduction: every retained KV block is HBM the decode scheduler can no longer use.
Part One: What is actually being compared
These two options are usually presented as alternative implementations of one capability. They are better read as two different answers to the question of where uncertainty is allowed to live. shared prefix caches absorbs it in one place, recomputed prompt prefixes in another, and a comparison that measures only the average outcome misses precisely the reason a team would accept the extra complexity of either one in multi-tenant LLM serving.
There is an accounting asymmetry worth making explicit. shared prefix caches tends to put expenditure into capital, where it is spread over years and needs a approval process; recomputed prompt prefixes keeps it in operations, where it is easy to approve and difficult to cap. Finance and engineering usually prefer opposite sides of that line, so the decision is often settled by the budgeting calendar rather than by a measurement in multi-tenant LLM serving.
Part Two: The arithmetic that decides it
The break-even is a memory accounting problem. Suppose a 4,096-token prefix occupies about 0.8 GB of KV state on a given accelerator, and the device has 80 GB of usable HBM after weights and activations. Holding fifty live prefixes consumes roughly half the remaining memory, which reduces the batch size the scheduler can admit for decode. If the prefill saving is 30 percent of step time but the smaller batch raises decode cost per token by 20 percent, the net gain is thin. The cache only wins while its hit rate stays above the point where the admitted-batch penalty cancels the prefill saving.
What actually decides the outcome is the slowest link in the chain, not the fastest. shared prefix caches improves one step, but if the surrounding steps still dominate the total, the improvement barely moves the wall clock. This is the recurring disappointment in multi-tenant LLM serving: a component-level win that is real, reproducible and almost invisible to the user because it was applied to the wrong term in the sum.
Deployment changes what gets measured. After shared prefix caches goes in, the metric people optimise tends to shift toward predictability; after recomputed prompt prefixes, toward unit price. That shift is why two teams can look at the same production data and reach opposite conclusions about which option won, and why an evaluation should record the metric before the trial rather than after it.
Part Three: Where the simple model breaks
Tenant mix decides that hit rate. A single enterprise customer with one system prompt gets near-perfect reuse; a public endpoint with thousands of distinct instruction blocks gets almost none, yet still pays the memory reservation and the hashing overhead. Worse, eviction is bursty: a sudden influx of new prefixes pushes out the hot ones, and the fleet then pays both the recomputation and the cache-thrash memory traffic at once. This is why a cache that benchmarks beautifully on one workload can regress a mixed workload.
Holding roughly fifty 4,096-token prefixes can consume about half the HBM left after weights, and the resulting smaller batch can erase a 30 percent prefill saving.
Part Four: The cost that arrives later
There is a privacy boundary that is easy to underprice. KV tensors are not irreversible hashes of text; under some conditions they retain enough structure to leak information about the source tokens. A shared cache therefore needs per-tenant namespaces, and a hit must never cross that boundary. Namespacing fragments the cache into smaller pools, which lowers the effective hit rate precisely in the multi-tenant setting where caching was supposed to pay for itself.
Adoption is unlikely to be a single flip. The first movers are the cases where the deciding variable is most extreme, the middle of the market follows once tooling and reference designs exist, and the long tail stays with recomputed prompt prefixes until the cost of switching falls below the cost of maintaining an exception. A reasonable reading of the next several years is coexistence with a slowly moving boundary rather than a replacement event.
Part Five: What this changes outside the engineering team
For an ordinary user the visible effect is the wait before the first word appears. A well-hit cache turns a two-second pause into a few hundred milliseconds on a long system prompt; a badly hit cache makes that pause unpredictable, which is more annoying than a consistently slow one. Product teams end up designing around the variance with loading states and timeout copy, a cost that never shows up in a throughput benchmark.
There is a mundane public-facing consequence as well: reliability and predictability are what people actually notice. A household, a commuter or a small shop does not experience an architecture, only whether a screen updates, a bill arrives on time or a service works after two years. Those are the terms in which the eventual judgement will be made.
A fair trial measures time to first token and total tokens per second at the same tail-latency target, on the real tenant mix, with the cache memory reservation included in the admitted batch. Anything measured on a single replayed prompt will overstate the win.
A market sizing sanity check
If inference keeps absorbing the majority of accelerator spend, the serving-layer market for prefix-aware schedulers, KV tiering and cache-aware routers is plausibly $2B to $7B annually by 2030. The low end assumes these features stay inside a few dominant serving frameworks; the high end assumes a distinct tiering and routing layer is sold across clouds. The driver is not model size but the ratio of shared instruction tokens to unique user tokens, which rises as agents and tool schemas become standard.
Treat the numbers as a scale model. They are built to show where the curves cross and which term dominates, and they deliberately round the inputs that vary most between deployments. The ranking is robust to those rounds; the exact threshold is not.
The comparison that matters
The comparison that matters is between prefill arithmetic avoided and HBM capacity surrendered. Shared prefixes win when a small number of stable system prompts dominate traffic; recomputation wins when prompts are diverse, memory is tight or isolation rules fragment the pool. The same endpoint may need both policies at different hours.
A change in regulation, in the supply of a single component, or in the tariff a site faces could invalidate the ranking overnight. The comparison is a snapshot of prices and constraints in multi-tenant LLM serving, not a property of the technologies, and anyone reusing it in a different year should re-run the threshold rather than reuse the conclusion.
Judge a cache on time to first token under the real tenant mix and with per-tenant isolation enabled, never on a single replayed prompt.
SOL: HCqPmPJM3NVtS6YrmVaTZYnuP8XZ2PFgQZKT94VK1LkS
Creating this takes real effort — thank you for the support.
Top comments (0)