TL;DR — Most teams size LLM serving around FLOPs and tokens-per-second, but the actual constraint is KV-cache memory. Once you model inference as a memory-allocation problem instead of a compute problem, batching cliffs, quantization tradeoffs, and prefix caching all make sense as the same lever pulled in different directions.
Ask most engineers what limits LLM inference throughput and you'll get an answer about FLOPs, GPU utilization, or maybe tokens-per-second benchmarks. That answer is wrong for the overwhelming majority of production serving workloads. The actual constraint is memory — specifically, the KV cache — and treating it as a compute problem is why so many serving stacks fall off a latency cliff instead of degrading gracefully.
This matters because the fix for a compute-bound system and the fix for a memory-bound system are opposite. If you think you're compute-bound, you buy more GPUs or optimize kernels. If you're actually memory-bound on KV cache, more GPUs without a memory strategy just moves the cliff further out — it doesn't remove it.
Why FLOPs Are the Wrong Mental Model
During autoregressive decoding, each new token requires attending to every previous token's key and value vectors. Those vectors have to live somewhere for the duration of the sequence — that's the KV cache. Unlike model weights, which are fixed and shared across every request, KV cache grows linearly with both sequence length and batch size, and it's allocated per-request.
The size of that cache per token is roughly: 2 (K and V) × layers × attention heads × head_dim × dtype_bytes. For a mid-size dense model with a few dozen layers, this comes out to tens of kilobytes per token, per sequence. Multiply that by a long context window and a large batch, and you can burn through GPU memory faster than you burn through compute.
The consequence: your GPU can often do far more matrix multiplication than it can afford to feed with cached context. Compute utilization looks fine on paper while memory is the thing actually gating how many concurrent requests you can serve.
The Batching Cliff, Not a Batching Slope
Continuous batching — the technique that made modern serving engines fast — works by packing new requests into in-flight decode steps as slots free up. It's a genuinely great idea, and it's why throughput-oriented serving stacks outperform naive request-per-batch designs by a wide margin. But it has a hidden failure mode: batch size isn't just a scheduling decision, it's a memory commitment.
Every additional concurrent sequence adds its own KV cache footprint, growing with every generated token. As long as aggregate cache fits in memory, the system behaves smoothly. The moment it doesn't, the engine has to make an ugly choice: evict a sequence, block admission of new ones, or spill to slower memory. Any of those options shows up to the caller as a sudden, sharp increase in tail latency — not a gradual slowdown.
This is why production LLM serving so often exhibits a cliff-shaped latency curve rather than a smooth one: the system looks fine until KV-cache pressure crosses a threshold, and then p99 latency jumps by an order of magnitude in what looks like a small load increase. The overwhelming instinct is to blame "load," but the actual root cause is memory admission control, or the lack of it.
Quantize the Cache, Not Just the Weights
Most quantization discussions focus on model weights — shrinking a model from a higher-precision format to INT8, FP8, or lower to fit on fewer or smaller GPUs. That's valuable, but it addresses the wrong side of the memory budget for long-context, high-concurrency serving. Weights are fixed and shared; KV cache is per-request and grows without bound as context lengthens.
Quantizing the KV cache itself — storing keys and values in FP8 or INT8 instead of a higher-precision format — directly attacks the actual variable cost of serving. It roughly halves or quarters the memory footprint per token, which translates directly into either more concurrent sequences at the same latency, or the same concurrency at longer context lengths. The accuracy cost is real but usually much smaller than people expect, because attention scores are somewhat forgiving of reduced precision in the value vectors especially. Engines like vLLM and SGLang treat KV-cache quantization as a first-class serving lever precisely because it changes the shape of the cliff described above — it pushes the threshold outward rather than eliminating it.
The strategic point: if your serving cost model doesn't separately account for weight memory versus KV-cache memory, you'll systematically under-invest in the lever that actually scales with your traffic pattern.
Prefix Caching Is a Financial Instrument, Not Just a Speed Trick
Prefix caching — reusing the KV cache for a shared prompt prefix across multiple requests — gets pitched as a latency optimization for chat-style workloads with repeated system prompts. That's true, but it undersells what's happening economically. Every cache hit on a shared prefix is prefill compute you don't pay for and memory you don't have to regenerate. In workloads with long, stable system prompts, few-shot examples, or retrieved context that repeats across many requests, prefix caching effectively amortizes the most expensive part of inference — prefill — across an entire population of requests instead of paying for it every time.
This changes how you should think about prompt design. A prompt structure that maximizes shared prefixes across requests — even at the cost of a slightly longer prompt — can be cheaper in aggregate than a shorter, more request-specific prompt that never hits cache. Prompt engineering for cost is, in part, cache-topology engineering.
Thinking in Bytes per Dollar, Not Tokens per Second
The practical shift this implies: stop sizing LLM serving capacity primarily around tokens-per-second, and start sizing it around KV-cache bytes per dollar of GPU memory, at your actual context-length and concurrency distribution. Tokens-per-second is a useful headline benchmark, but it's measured under conditions — short contexts, modest batch sizes — that rarely match production traffic, where context length is the variable that actually determines whether you're compute-bound or memory-bound.
Concretely, that means instrumenting KV-cache occupancy the way you'd instrument disk or memory pressure on any other stateful system: track it per-request, alert on it before it becomes an eviction event, and treat cache quantization and prefix-sharing design as capacity-planning decisions, not micro-optimizations. The teams that get LLM serving costs under control aren't the ones with the fastest kernels — they're the ones who stopped asking "how many tokens per second" and started asking "how many bytes of context can we afford to hold, for how many users, at once."
Top comments (1)
The memory framing is the useful part here. Treating KV cache as a first class capacity constraint makes batching decisions much easier to reason about, especially when p99 latency matters more than peak throughput. A simple cache budget per request seems like a practical starting point for admission control.