KV Cache Wars: How Compaction, Quantization, and a Rust Rewrite Are Redefining LLM Serving Economics

This edition of the LLM Inference Digest covers new research and engineering published between 2026-10-21 and 2026-10-28.

🔥 Highlights

1. Serving DeepSeek-V4.1-Flash at Scale: vLLM Matches DeepInfer and Scales to GB300 NVL72 — vLLM now rivals DeepSeek's own serving stack.
   

2. How Anthropic ships Claude with Claude: Separating the Brain from the Hands — a Rust rewrite cuts production CPU cost 40-50%.
   

3. How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real Applications — one lever, no retraining, massive savings.
   

4. GLM-5.3 Optimizations, Part 2: PCP, DCP, and Hybrid HiSparse on 16 H200s — breaks the 1M-token KV memory wall.
   

5. Folding Attention Biases into Keys Makes Compacted KV Caches Run on Standard Kernels — removes the custom-kernel tax on KV compaction.

arXiv (cs.LG, cs.DC)

• Speculative Decoding for Mixture-of-Experts Models: Expert Bytes, Kernel Bandwidth and the Best Draft Length — 2026-10-25. Derives a step-time model for MoE speculative decoding based on bytes-per-kernel-request over effective bandwidth, since consecutive drafted tokens on an MoE model often hit the same experts. Gives a concrete formula for when speculation pays off and what draft length is optimal on MoE serving, directly actionable for anyone tuning speculative decoding on DeepSeek/Mixtral-style deployments.
• Distributed LLM Inference via Communication-Efficient Speculative Decoding — 2026-10-25. Addresses the vocabulary-scale communication bottleneck in device-server collaborative speculative decoding, where exact rejection sampling normally requires shipping full token distributions over the network. Relevant for anyone deploying split inference across bandwidth-constrained links such as mobile-to-cloud.
• ZipFusion: Fusing Lossless KV Decompression with Attention for Efficient LLM Decoding — 2026-10-24. Points out that prior lossless KV-cache compression only reduces inter-device transfer but still restores the full KV cache in GPU memory before attention. Fuses decompression directly into the attention kernel to cut both memory footprint and bandwidth pressure during decode, a concrete throughput/concurrency win for long-context serving.
• Folding Attention Biases into Keys Makes Compacted KV Caches Run on Standard Kernels — 2026-10-27. KV-cache compaction methods that add a per-key bias term normally can't run on FlashAttention/vLLM's paged-cache kernels since there's no input slot for the bias. This folds the bias into the keys via a closed-form per-head direction, removing a "custom kernel required" barrier to adopting KV compaction in production stacks.
• RAQ-MLA: Role-Aware Quantization for Multi-Head Latent Attention Caches — 2026-10-24. Existing KV-cache quantization schemes were designed for standard per-head K/V and don't account for what Multi-Head Latent Attention (used in DeepSeek-style models) actually caches, causing accuracy loss at low bitwidths. Proposes a role-aware scheme tailored to MLA's cache structure, useful for teams quantizing MLA-based models for memory savings.
• ResCom: Activation Residual Compensation for Large Language Model Quantization — 2026-10-25. Targets accuracy loss from activation rounding in low-bit weight-activation quantization, a gap calibration-based reconstruction can't fix. Introduces a low-rank correction using the rounding residual itself, a technique that could improve accuracy retention at INT4/INT8-class quantization without extra inference cost.
• DualPace: Decoupled Scheduling and Cooperative GPU–CPU Decoding for LLM Serving — 2026-10-24. Current CPU-offload serving systems are "GPU-gated" — CPU-resident requests only progress when the GPU scheduler exposes overlap opportunities, causing bursty CPU throughput. Proposes decoupling scheduling so CPU and GPU act as independently-paced resources, relevant to anyone running hybrid GPU/CPU-offload serving to extend effective KV capacity.
• InferScale: Kubernetes-Native Predictive Autoscaling for SLO-Aware AI Inference — 2026-10-25. Argues Kubernetes HPA's reactive scaling is too slow for inference pods with tens-of-seconds cold-start, worsened in multi-tenant clusters where independent HPAs fight over resources. Presents a K8s-native autoscaler combining inference-aware signals and short-horizon demand prediction, directly applicable to teams fighting cold-start-driven SLO violations.
• TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving — 2026-10-27. For agentic workloads alternating between GPU-side inference and CPU-side tool calls, shows existing serving systems ignore tool-return readiness and waste GPU resources waiting. Proposes a joint scheduler accounting for near-term tool-return timing, timely given the shift toward agentic serving infrastructure.
• KVStream: Adaptive Parallel Storage Management for KV Caching in Large-Scale LLM Serving — 2026-10-22. For long-context/agentic workloads where reused-prefix KV caches outgrow GPU+host memory, a single node-local NVMe absorbs all prefix retrieval traffic while parallel storage nodes sit idle. Casts prefix retrieval as aggregate-bandwidth scheduling across storage tiers, a practical fix for storage I/O becoming the bottleneck in prefix-caching-heavy deployments.
• Gandalf: Flash Shall Not Die! Increasing Flash Storage Lifetime for KV Cache Offloading in LLM Serving — 2026-10-22. Flags a real production hazard: SSD-based KV-cache offloading under multi-round-conversation/agent workloads writes so much that SSD lifetime drops from years to months. A hardware-cost/ops concern worth knowing before deploying SSD-backed KV offload.
• Plenty of Room in the Middle: Revealing Opportunities on a GPU Cluster via Fine-Grained Monitoring — 2026-10-26. A university research GPU cluster study finds prior minute-level monitoring missed GPU-level (not just job-level) utilization opportunities that only fine-grained monitoring reveals. Useful empirical evidence for anyone doing GPU-cluster capacity planning or utilization audits.
• Context Is Not Free: Carbon-Aware LLM Routing for Multi-Turn LLM Web Services across Geo-Distributed Data Centers — 2026-10-26. Multi-turn routing decisions that send a query to a cleaner-grid region or smaller model are complicated because each turn's KV cache lives only where earlier turns ran, so naive carbon/cost-aware routing can break cache locality. Relevant for anyone operating multi-region LLM serving who needs to balance cost/carbon against KV-cache locality.

vLLM blog

• Day-0 Support for Flint on vLLM — 2026-10-21. vLLM shipped day-0 support for Recursive's Flint, a 40B-param MoE (~2.5B active/token), in both NVFP4 and BF16. Flint-specific kernels cut NVFP4 decode latency from 4.81ms to 1.78ms per forward pass (2.7x) and reduced kernel count per forward from 1,096 to 346, enabling ~500 tok/s single-request decode at 100K context on one GB200.
• Serving DeepSeek-V4.1-Flash at Scale: vLLM Matches DeepInfer and Scales to GB300 NVL72 — 2026-10-23. A full-rack GB300 NVL72 deployment using prefill/decode disaggregation plus Wide-EP reaches ~954K tok/s/GPU on AgentX, matching DeepSeek's own DeepInfer engine within 6% on decode throughput; a specialized low-latency decode path hits 826 tok/s for a single user. Directly actionable for teams sizing large-scale MoE serving on Blackwell racks.
• GLM-5.3 Optimizations, Part 2: PCP, DCP, and Hybrid HiSparse on 16 H200s — 2026-10-23. Introduces Prefill/Decode Context Parallelism combined with Hybrid HiSparse decode on a 2-node/16x H200 setup, delivering 1.7–3.2x per-user speed over a DP8 baseline at equal throughput and serving 1M-token contexts. Relevant to anyone hitting KV-cache memory walls on long-context deployments.
• vLLM Delivers Day-0 Support for MiniMax-M3.1-Flash — 2026-10-23. Day-0 support across Blackwell, Rubin, and AMD MI350-series, using NVFP4 KV cache and NVFP4-quantized MoE weights that shrink the checkpoint from 451.5GB to 250.1GB. Reports up to 418.9 tok/s/user (GB300), 724.9 tok/s/user (Rubin), and 133.3M total tokens/$ on MI355X — directly useful cost/throughput reference points across three hardware targets.

Hugging Face blog

• How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real Applications — 2026-10-21. Dharma-AI's VocabX rebuilds a small model's output vocabulary around per-token generation cost rather than frequency, keeping the base model, data, and recipe unchanged. On a production Portuguese essay-grader built on Qwen3-4B it cut measured inference cost 77% and average response length from 350 to 101 tokens while the bench score rose, exploiting the fact that generating tokens costs roughly 7.4x more than reading them.

No other posts in the window had real serving/throughput/cost substance — the rest were model launches, operator-level profiling tutorials, or non-inference research.

Together AI blog

• Auto repair for Together GPU Clusters: from GPU fault to healthy node — 2026-10-27. Explains Together's fault-detection system, which recommends the least-disruptive fix with supporting evidence, applies per-fault-type/per-Xid-code policy, escalates repairs on recurrence, and auto-revalidates repaired nodes before they rejoin the pool. The post ties this explicitly to inference reliability: GPU faults can take a serving replica down, making this relevant to anyone operating GPU fleets for inference.

Modal blog

Nothing relevant this week — the only two posts in the window ("Introducing our Training API" and "Get started with LLM training on Modal") are about post-training/fine-tuning infrastructure, not inference or serving.

Latent Space

• Why AI Compute Comes in Rectangles — 2026-10-22. Modal's CEO explains why fixed GPU contracts don't fit inference's spiky demand and how Modal pools demand across customers to cut idle capacity, covering fast-replica-startup tricks (GPU memory snapshots, process checkpointing) and interest in disaggregated inference. Direct operator insight into why Kubernetes/Docker-style autoscaling falls short for GPU-serving workloads.
• How Anthropic ships Claude with Claude: Separating the Brain from the Hands — 2026-10-27. Anthropic engineers describe rewriting the Messages API in Rust, shadow-tested against the Python system, reporting ~40-50% CPU savings and ~10-20% better time-to-first-token, with bigger gains on long-running workloads. Also covers CPU as a capacity-planning bottleneck and a "brain/hands" architectural split to cut latency.
• Kimi 3.1 Pro 104BA-2.8T: Fable-5.1-Class Open Weights... Highspeed API with 2-3x speed — 2026-10-24. New Kimi K3.1 Pro MoE release with a "Highspeed" variant claiming 2-3x faster generation at 2x cost; the digest also bundles a 12% DeepSeek V4.1 Flash decode-throughput gain from small-message all-reduce kernels and a KV-cache "read-ahead" trick cutting wait time from 12.2s to 3.2s on a 6GB GPU.
• Black October — 2026-10-23. Dense inference-systems roundup: AMPD routes multi-turn prefill/decode by latency headroom (67% SLO-attainment gain over Dynamo); AKV lets agents reposition their own KV cache (-27.7% cost); Rubin NVL72 hits 862 tok/s/user on GLM 5.3 after kernel work; MiniMax M3.1 Flash gets day-0 vLLM support with NVFP4 weights/KV-cache.
• OpenAI's Patch the Planet program... — 2026-10-22. Deep dive on DeepSeek's DeepInfer serving stack combining speculative decoding, dynamic prefill sizing, and CPU/GPU overlap scheduling, reporting 1.5M input tok/s per prefill accelerator with a 96% cache-hit rate and 14K output tok/s/accelerator on decode. Also covers SGLang-TorchTPU reaching 770 decode tok/s at batch-1 and EAGLE-4 speculative decoding at 3.59x speedup.
• Anthropic files for $2T IPO in 2 weeks — 2026-10-27. Digest reports vLLM-on-DeepSeek-V4.1-Flash achieving $38.96 revenue/GPU-hour at 954K tokens/s/GPU aggregate throughput, DCOCO speculative decoding restoring context between parallel drafts for up to 6.6x speedup, and KV-cache-sharing across loops cutting GPU memory 66.2%.
• Anthropic ends Coding Interviews: "Hunger beats tenure" — 2026-10-28. Reports CoreWeave-side serving gains (+63% traffic on GLM-5.3, 2x per-user throughput on Qwen3.8-27B) and TokenGate claiming up to 11.3x faster long-context prefill, plus Anthropic's S-1 showing serving compute cost per revenue dollar falling from $0.88 to $0.15.

Through-line

Nearly every independent source this week converges on the KV cache as the real site of competition in LLM serving: arXiv papers attack it from quantization (RAQ-MLA), compaction-kernel compatibility (the folded-bias trick), storage tiering (KVStream), and SSD wear (Gandalf), while vLLM's GLM-5.3 work and several Latent Space items attack it from the context-parallelism and repositioning angle. At the same time, production engineering payoffs are standing shoulder to shoulder with new algorithms — Anthropic's Rust rewrite and Dharma-AI's vocabulary specialization both show that rethinking an existing system, not just adopting a new technique, can move cost and latency by double digits. Vendor-neutral serving stacks are also closing the gap with model vendors' own infrastructure, as vLLM's DeepSeek-V4.1-Flash benchmarks landing within 6% of DeepInfer illustrate.

What's catching your eye this week — KV-cache tricks, the Rust rewrite, or something else? Drop a comment below.

Comments

[Max Quimby]

Great roundup — the throughline I'd pull out is that the KV cache, not the weights, is the memory wall now for long-context and agent workloads, and the moves that'll actually get adopted are the ones that don't demand a custom kernel. The "folding attention biases into keys so compacted caches run on standard kernels" paper is the sleeper here for exactly that reason: a technique that imposes a custom-kernel tax tends to die in the gap between a great benchmark and something an ops team will run in production.

The practitioner angle that ties it to serving cost: in multi-agent setups the KV pressure is self-inflicted — every turn re-sends a long, mostly-stable prefix — which is why prefix caching and compaction compound so well there rather than competing. You're paying to keep the same 20k tokens resident across dozens of calls.

Of the five highlights, which do you think lands in vLLM mainline first? My bet's on the standard-kernel compaction work, precisely because it asks the least of the deployment stack.