A 70B-class model processing 8K context with batch size 32 requires roughly 640GB of KV cache memory alone, often exceeding the model weights themselves. That single statistic reframes how you should think about LLM inference economics in 2026: the key-value cache, not the model, is the thing that breaks your budget.
The KV cache stores the key and value tensors of previously processed tokens so they can be reused during autoregressive generation, avoiding recomputation of past tokens. It's the mechanism that makes transformers tractable at inference time. But its memory consumption scales linearly with both context length and batch size, and it is often the largest consumer of GPU memory during inference after model weights. If you're serving long-context agents or multi-tenant chat, you're already fighting this bottleneck — whether you know it or not.
What I call the Cache-First Serving pattern has emerged as the central framing for 2026 LLM infrastructure. Every major cost-leverage technique — provider prompt caching, FP8 quantization, offload tiers — reduces to managing prefix stability and cache lifetime rather than model selection. The teams winning on cost-per-token aren't the ones picking the cheapest model. They're the ones who treat the KV cache as the primary engineering object and optimize around its constraints.
What Is the KV Cache and Why Does It Dominate GPU Memory?
Transformers generate text one token at a time. Each new token must attend to every token before it, which means the model needs the key and value vectors for all prior tokens on every single decode step. Without caching, generating token 1,000 would require recomputing attention for all 999 previous tokens — quadratic work that's almost entirely redundant. The KV cache eliminates that by storing those vectors once and reading them back on subsequent steps.
Here's where it gets painful. The cache grows linearly with both sequence length and batch size, and it's per-layer — every transformer layer maintains its own set of cached keys and values. For a 70B-class model at 128K sequence length across 256 concurrent sessions, KV cache alone demands over 1 TB of HBM, illustrating that KV cache is the primary serving bottleneck. That's not a model weight problem. That's a cache problem.
The numbers get stark at smaller scales too. A 70B-class model processing 8K context with batch size 32 requires roughly 640GB of KV cache memory, often exceeding model weights. At 128,000-token contexts, the KV cache can claim the majority of an NVIDIA H100 GPU's 80GB HBM, leaving inadequate room for model weights and concurrent sessions. You load a model that fits comfortably, start a few conversations, and then hit an out-of-memory error because the cache silently consumed every spare byte.
Traditional inference systems waste 60-80% of allocated KV cache memory through fragmentation and over-allocation. They reserve contiguous memory blocks for the maximum possible sequence length upfront — a 4K max context allocates 4K worth of cache even for 100-token requests, wasting 97.5% of reserved memory. PagedAttention, introduced by Berkeley researchers and popularized by vLLM, mitigates this by paging the cache into small fixed-size blocks (typically 16 tokens) that can be stored anywhere in memory, much like an operating system's virtual memory. This eliminates internal and external fragmentation and lets a server handle 2 to 4× more concurrent requests on the same VRAM.
How Does FP8 KV Cache Quantization Change the Economics?
FP8 (e4m3) quantization of the KV cache halves its size compared to BF16, roughly doubling available cache capacity on NVIDIA H100 GPUs with minimal accuracy impact. That's the headline. But the interesting part is what it actually changes — and it's not what most people assume.
Here's the contrarian finding: FP8-quantizing the KV cache slightly slows per-token generation yet raises overall throughput ~41% and cuts cost per token ~30% because the binding constraint was concurrent session count from memory capacity, not raw compute. Compression that loses micro-benchmarks wins macro-economics.
Cloudflare's production data on Moonshot Kimi K2.6 makes this concrete. Storing the KV cache in FP8 instead of BF16 raises resident context from roughly 686,000 tokens to about 1.37 million tokens (2x) and enables ~41% higher throughput at ~30% lower cost per token. At any single concurrency level, BF16 is a few percent faster per token — the FP8 attention kernel has to convert values as it reads them. But BF16 runs out of cache at 32 concurrent requests and can't admit a 33rd, while FP8 keeps going to 64. The per-token slowdown is real but irrelevant; the throughput gain comes from fitting twice as many requests in the same memory.
| Optimization | Mechanism | Tradeoff | Target Audience |
|---|---|---|---|
| FP8 KV Quantization | Halves cache size vs BF16 | Slight per-token slowdown; ~0.5% accuracy loss | High-concurrency serving (Cloudflare, vLLM users) |
| PagedAttention | Pages cache in fixed-size blocks | No quality cost; requires compatible serving framework | Multi-tenant inference (vLLM deployments) |
| KV Cache Offload | Moves cache to external storage tiers | Added latency; infrastructure complexity | Long-context agentic workloads (GKE + Lustre) |
In vLLM specifically, FP8 KV-cache quantization can reduce the per-token cost of the KV cache to 54% of its BF16 counterpart in best cases for memory-bound decoding. The vLLM team's comprehensive validation across decoder-only and MoE models on both Hopper and Blackwell architectures found that for head dimensions 64 and 128, FP8 offers speedups on both prefill and decoding. The main caveats are hybrid-attention models with small sliding-window layers (where skipping those layers is often better) and large-head-dimension models where prefill can still regress.
The quality picture has converged. By 2026, both the research literature and internal benchmarks have landed on a consensus: with the right quantization scheme, quality loss stays within 0.5%. E4M3 (4 exponent, 3 mantissa bits) offers more precision and works well for code generation and mathematical reasoning. E5M2 (5 exponent, 2 mantissa bits) has wider dynamic range and is stronger on long-context and multilingual workloads. Per-channel scale plus per-token shift — an activation-aware quantization that absorbs outliers — is implemented in both SGLang 0.4.x and vLLM 0.7.x.
When Should You Offload KV Cache to External Storage?
You offload when the cache outgrows local GPU HBM and the cost of adding more GPUs exceeds the latency penalty of reading from a remote tier. This is becoming the default architecture for long-context agentic workloads, not an edge case.
Google's implementation is the most thoroughly documented. Offloading shared prefilled KV caches to external storage tiers — specifically Google Cloud Managed Lustre — can yield over 50% TCO savings and reduce GPU-hour requirements by nearly 60% for Llama-3.3-70B inference with a 95% cache hit rate. The benchmark configuration used a six-node A3 Mega cluster with 50,000-token prompts, 256-token questions, and 512-token outputs. Extending the Lustre offload with CPU RAM integration delivered approximately 40% improvement in Time to First Token and a 30% reduction in end-to-end latency.
The hardware ecosystem is racing to support this. NVIDIA formalized its Context Memory Storage platform (CMX) in January 2026, managed by the BlueField-4 DPU, with partners including VAST Data, DDN, IBM, Nutanix, WEKA, and Cloudian. XCENA launched its MX1 CXL-based memory expansion platform targeting KV cache offload on Intel Xeon 6, demonstrating a 20 TB CXL memory pool at FMS 2026. Samsung Electronics showed that a 1TB CXL memory pool sustained KV cache demand where a 512GB DRAM configuration had performance degraded.
The tradeoff is real. Offloading expands context capacity and reduces GPU-hour use, but keeping KV on GPU HBM preserves low latency and strictly limits batch size and context length. You're trading latency for capacity, and the right answer depends on your workload's tolerance for added time-to-first-token. For agentic workloads that maintain growing task histories across many reasoning steps, the capacity trade almost always wins — the alternative is buying more GPUs purely for HBM capacity, which is the most expensive way to solve a memory problem.
How Do Provider Prompt Caching and KV Cache Relate?
Provider prompt caching is the billing-surface manifestation of KV cache management. When a provider stores the computed KV cache of a prompt's leading tokens, subsequent requests with an identical prefix skip recomputation and are billed at a discounted rate. It requires exact byte-for-byte prefix match, not fuzzy matching — that's semantic caching, a separate technique with separate economics.
As of June-July 2026, Anthropic Claude and OpenAI GPT-5.x offer prompt cache reads at 0.10x base input (90% discount), while write costs differ. The mechanics split into two camps:
-
Explicit cache control (Anthropic): You place
cache_controlmarkers in your API request to define where the cached prefix ends. Maximum control, up to 4 breakpoints per request, but requires code changes and charges a write surcharge — 1.25x base input for the 5-minute TTL, 2x for the 1-hour TTL. - Automatic caching (OpenAI, DeepSeek): The provider auto-detects matching prefixes. Zero code changes, no way to force a hit. OpenAI's GPT-5.6 bills cache writes at 1.25x base input and reads at 0.1x (90% off), per MixRoute's cross-provider guide and CometAPI's pricing analysis. DeepSeek V4 Flash drops from $0.14 to $0.0028 per million tokens on a hit — a 50x cut, automatic since 2024, with no write premium and no storage fee.
Here's the tension worth flagging: sources disagree on OpenAI's write costs. MixRoute (July 2026) and CometAPI (August 2026) report GPT-5.6 cache writes billed at 1.25x base input with reads at 0.1x (90% off). But tokenkarma (2026) and swfte (May/July 2026) report OpenAI writes as free (1.0x) with reads at 0.25-0.5x (50-75% off). The discount depth you actually receive depends on the model tier and possibly the gateway route — verify against your actual API response metadata before forecasting savings from published rates.
Google's Gemini adds a third model: implicit caching delivers a 75% discount (0.25x base) per DigitalApplied's engineering guide, while MixRoute lists Gemini reads at 0.1x base input (90% off) with implicit writes free. Explicit caching adds a per-hour storage fee ($4.50 per 1M token-hour for Gemini 2.5 Pro), which changes the math for low-traffic workloads where the cache may expire before it's read enough times to break even.
The highest-ROI action here is enforcing strict prefix stability. Put your system prompt, tool definitions, and few-shot examples at the top in a byte-stable order. Put volatile content — user messages, retrieved RAG chunks, timestamps — last, after the cache boundary. One changed character near the top invalidates the entire prefix. A workload that moves from 7% to 84% hit rate through prompt restructuring sees a 12x improvement in cache effectiveness without touching the model or the infrastructure.
What Are the Real Tradeoffs of Aggressive KV Compression?
Aggressive KV compression — eviction, sub-8-bit quantization, cross-layer sharing — maximizes concurrency and cuts hardware cost but risks silent accuracy loss on rare retrieved context. Full-precision KV cache guarantees lossless output but caps concurrent users and inflates GPU memory spend. The choice isn't abstract; it's a specific bet on whether your workload will ever need the tokens you're about to throw away.
Eviction is the most tempting and most dangerous family. Dropping cache entries frees memory immediately, and heuristics based on attention scores usually work well on benchmarks. The failure mode is that "usually" hides the case you care about: a detail mentioned once early in a long document is exactly what a low attention score would evict and exactly what a user asking about that document wants retrieved. One documented case saw a Go code model's pass rate fall from 57.9% to 37.8% — twenty points gone — because an eviction policy dropped prompt tokens once context exceeded the budget. The 15-problem smoke test showed identical pass rates. The 164-problem benchmark revealed the catastrophe.
Quantization is safer. 4-bit quantization of the cache turned out to be genuinely lossless in the same case study — the precision was never load-bearing. FP8 is now the production default across major serving frameworks, with quality loss staying within 0.5% on aggregate benchmarks. Going below 8 bits is where artifacts start to appear on generation benchmarks, though research like TurboQuant (ICLR 2026) and STAR-KV (ICML 2026) is pushing the frontier with chained mechanisms that combine low-rank compression with mixed-precision quantization.
Microsoft's VeriCache offers a different path entirely: use the compressed KV cache to draft tokens, then verify them against the full KV cache kept out of GPU memory. The compressed cache is HBM-bandwidth-bound; the full-KV swap is PCIe/network-bound. The two can run in parallel. VeriCache achieves up to 4x higher throughput than full-KV inference while producing identical outputs — lossless compression through speculative verification. It's the most elegant resolution of the compression-versus-accuracy tension I've seen, though it adds system complexity.
Which KV Cache Strategy Should You Adopt First?
The highest-ROI action for any production LLM system in 2026 is enforcing strict prefix stability for prompt caching and adopting FP8 KV quantization with tiered offload — not model upgrades. Realized savings only appear above ~60% cache hit rate, and the KV cache is the true scaling bottleneck.
Here's the decision framework, ordered by effort-to-impact ratio:
- Restructure prompts for prefix stability. Zero infrastructure cost. Move static content to the front, volatile content to the back. This alone can move your cache hit rate from single digits to 80%+. If you're using provider prompt caching and not doing this, you're leaving the entire discount on the table.
-
Enable FP8 KV cache quantization. One flag in vLLM (
--kv-cache-dtype fp8). Halves cache memory, doubles concurrent capacity, ~0.5% accuracy loss. The per-token slowdown is real but irrelevant when your binding constraint is memory capacity, not compute speed. - Deploy PagedAttention. Already default in vLLM. If you're on a serving framework that doesn't page the cache, you're wasting 60-80% of allocated memory to fragmentation. Switch frameworks.
- Add KV cache offload for long-context workloads. When HBM is insufficient — which happens at 128K+ contexts with meaningful concurrency — offload to CPU RAM, NVMe, or a dedicated tier like Managed Lustre. Google's data shows 50%+ TCO savings at 95% hit rates. The latency penalty is real but the alternative is buying GPUs purely for HBM capacity.
- Evaluate eviction only with your own retrieval patterns. Never trust a benchmark average for eviction policies. Test against the specific long-tail queries your users actually ask. If a detail mentioned once early in context is something a user might ask about later, eviction will burn you.
The teams that win on cost-per-token in 2026 aren't the ones chasing the cheapest model or the latest architecture. They're the ones who recognized that the KV cache is the central object in LLM serving and engineered around its constraints. The question isn't whether to optimize your cache strategy — it's whether you can afford not to before your GPU bill forces the issue.
Originally published at SaaS with Alex
Top comments (0)