Speculative Decoding Grows Up, and Prefix Caches Learn to Collapse

This digest covers LLM inference and serving news from 2026-11-04 to 2026-11-11: new arXiv research on KV-cache reuse, speculative decoding, and MoE communication, plus fresh posts from the vLLM, Hugging Face, and Together AI blogs, and relevant coverage from Latent Space. Modal had nothing new on this theme in the window.

🔥 Highlights

1. ForeSpark: Joint Draft and Verification Budget Scheduling for Parallel Speculative Decoding — stops spec decoding from wasting 40% of step time.
2. Introducing Kinference: An Open Source Inference Control Plane for vLLM and Kubernetes — SLO-aware routing cuts median TTFT 95%.
3. Bursts That Outlast Themselves: Persistent Degradation in Prefix-Cached Agent Serving — a traffic spike can wreck your cache for hours.
4. Encoder Disaggregation in vLLM: Scaling Multimodal Serving with EPD — decouples encoder scaling from prefill/decode.
5. Together Compiler: Continual Compilation for AI inference — turns kernel tuning into a live, continuous loop.

arXiv (cs.LG, cs.DC)

• Can KV Cache Be Reused Across Model Updates? A DeepSeek-V4-Flash Case Study (2026-11-10) — Introduces "Functional Cache Divergence" to check whether KV caches built under an old checkpoint stay valid after a model update. Reusing caches across DeepSeek-V4-Flash versions gives 2.1x–5.08x median TTFT speedups on long-prefix QA, but teams doing frequent rollouts need answer-level agreement checks, not just first-token agreement, before trusting reused caches.
• ForeSpark: Joint Draft and Verification Budget Scheduling for Parallel Speculative Decoding (2026-11-10) — Under high load, parallel-drafter spec decoding wastes up to 40% of step time on draft tokens that never get verified; ForeSpark jointly tunes draft/verification length per request via a global sort. On vLLM it reports up to 1.77x throughput over the prior SOTA and 1.51x more request rate at the same latency SLO.
• Bursts That Outlast Themselves: Persistent Degradation in Prefix-Cached Agent Serving (2026-11-10) — A short request burst on a prefix-caching vLLM/SGLang server can tip it into a collapsed state (cache hit ratio craters, throughput halves) that outlasts the burst itself, because slower service keeps more agent sessions alive and overflows the KV pool. Simple throttling doesn't fix it — admission control that caps resident KV demand against pool capacity does.
• Decode Token Parallelism: Memory-Efficient Distributed Attention Parallelism for Scalable LLM Decoding (2026-11-09) — TKNP decouples weight placement (tensor-parallel within a node) from attention/KV ownership (local per-node), so only current-token activations cross the network. On Llama 3.1 405B and Nemotron-4 340B with up to 64 H100s it beats decode context parallelism by up to 1.7x and data parallelism by 2.06x at equal GPU count for long-context, high-concurrency serving.
• SinkState: On-the-Fly SVD of Recurrent States for Efficient Hybrid LLM Serving (2026-11-10) — For hybrid attention/linear-attention models (Mamba-2, Gated DeltaNet), exploits the fact that the recurrent state is effectively low-rank, cutting peak recurrent-state memory by up to 2.6x and speeding decoding up to 1.58x versus Mamba-SSM/FLA with no accuracy loss.
• One-Sided Expert-Parallel All-to-All for Mixture-of-Experts Inference over NVLink (2026-11-09) — NVLinkOneSided writes MoE dispatch/combine tokens directly into target-GPU buffers, skipping metadata exchange. On Blackwell GPUs it's 1.2–3.0x faster than DeepEP for dispatch/combine and up to 2.4x faster end-to-end, with the biggest wins at small batch sizes.
• Ostinato: Fast CPU-GPU MoE Inference for Local Agents via Resident Experts in Every Forward Pass (2026-11-08) — Exploits the fact that 84% of expert routes for short prefills hit a stable set that fits in GPU memory, letting large MoE models run on a handful of consumer GPUs plus CPU offload. On 4x RTX 4090D it beats llama.cpp/KTransformers by 22–55% lower post-tool-call latency.
• REAL Loss: Optimizing Real Expected Acceptance Length for Speculative Decoding (2026-11-10) — Argues common draft-model training losses (cross-entropy, KL) don't actually optimize expected acceptance length, and proposes REAL, a drop-in training-loss change (no architecture or inference changes) that beats baselines across 9 task subsets.

vLLM blog

• Encoder Disaggregation in vLLM: Scaling Multimodal Serving with EPD (2026-11-11) — Splits the multimodal encoder into its own vLLM instances with remote embedding transfer over NIXL/Mooncake, scaling encoder capacity independently of prefill/decode. Audio workloads saw throughput rise from 62.21 to 73.13 req/s, and DeepSeek V4.1 Flash on GB200 saw 44–47% lower P90 TTFT at concurrency 64–128.
• Migrating vLLM to the LibTorch Stable ABI (2026-11-06) — vLLM's CUDA build now targets PyTorch's LibTorch Stable ABI, so one wheel works across newer PyTorch releases without rebuilding. Faster PyTorch adoption and fewer rebuilds for operators, at the cost of custom pybind11 kernels needing migration to TORCH_LIBRARY.
• Introducing Kinference: An Open Source Inference Control Plane for vLLM and Kubernetes (2026-11-04) — An Apache 2.0 control plane in front of vLLM on Kubernetes adding KV-cache-aware load balancing, token rate limiting, and per-tenant cost tracking. Buoyant's own benchmarks claim a 95% cut in median TTFT and 7.4x goodput on shared-prefix traffic vs. default k8s round-robin (vendor figures, not independently verified).
• vLLM Delivers Day-0 Support for Kimi K3.1 Pro (2026-11-04) — Day-0 support for Moonshot's 2.8T-param, 1M-context Kimi K3.1 Pro with sparse attention, giving ~4x more usable KV-cache per GPU at long context. Also ships token-sharded prefill (up to 1.69x faster TTFT at 340K tokens) and DSpark speculative decoding (3.0x decode speedup at batch size 1).

Hugging Face blog

• BotPsy: On-Device Personal Assistant with a Sub-Billion-Parameter Agentic Language Model (2026-11-11) — A 0.8B agentic model built on Qwen3.5-0.8B, shipped in bf16/Q8/Q4/GGUF for fully on-device operation. Its reasoning traces use ~3x fewer tokens than the base model on GPQA Diamond, a lever for cheap local agentic inference instead of cloud API calls.
• Your Experts Have Partners. There's HOPE for Keeping Them Together (2026-11-09) — HOPE is a second-order MoE expert-pruning method accounting for pairwise expert interactions, needing no retraining. At 50% expert pruning it matched unpruned Qwen3.5-35B-A3B on SWE-Bench Pro, freeing memory that can go toward longer context or more concurrent tool calls.
• MOSS-Live: A Full-Duplex Real-Time Interaction System (2026-11-07) — A full-duplex speech system that listens and talks simultaneously while running background tool calls without pausing conversation. Reports 1.14s turn-taking latency and 2.36s for tool-use, vs. 1.61s for GPT Realtime 2.1 — a useful latency benchmark for low-latency voice/duplex agents.
• Decision Models in Hugging Face (2026-11-10) — Introduces "decision models" that answer typed questions (choice/score/yes-no) as probability distributions instead of generating free-form text, served via Transformers/llama.cpp with zero output tokens on a sample response. A real architectural lever to cut latency and cost on classification/routing workloads, since autoregressive decoding is usually the costliest part of serving.
• A Closer Look at Phonon TTS (2026-11-05) — A 31M-param open-weight on-device TTS model reporting ~28ms P95 time-to-first-audio on CPU streaming vs. 165ms for Pocket TTS. Ships a custom ptts inference framework targeting 10+ backends plus a Transformers path tuned with torch.compile for high-throughput batched generation.

Together AI blog

• From Static to Polymorphic Inference: A new era for production open-source inference systems (2026-11-04) — Argues static, hand-tuned serving configs degrade as traffic, models, and hardware shift, and proposes treating kernel/engine optimization as a continuous agentic search problem against live production endpoints. A framing piece for teams rethinking how serving configs should evolve with traffic; the hard numbers live in the companion Together Compiler post.
• Together Compiler: Continual Compilation for AI inference (2026-11-04) — A continual-compilation loop where agents recompile/hot-reload serving configs into long-running engines instead of restarting them, cutting iteration time from ~467s to ~9s per candidate and reporting up to 2.6x higher tokens/sec within 24 hours. Also reports 1.7x throughput over megakernel baselines on Llama 1B and concrete low-latency numbers on MiniMax M3.1-Flash and DeepSeek V4.1-Flash.

Modal blog

Nothing qualifying this window — the only dated post (a Snorkel agent-simulation case study) had no throughput/latency/serving-framework substance for this theme.

Latent Space

• NaiveAI Promises 2,100 Tokens per Second. How Well Does It Code? (2026-11-10) — A hands-on review of Naive-N0.5-Flash (a MiMo-V2.5-derived MoE, 310B total/15B active) on the proprietary NaiveRT engine, using speculative decoding with an FP8 drafter on an 8-GPU Blackwell node. Concrete numbers: ~984 tok/s "thinking", ~1,816 tok/s response, peak 2,100 tok/s, and a spec-decoding round in 3.4ms vs. 12.3ms on SGLang — useful for anyone comparing inference engines on the speed/verbosity/cost trade-off.
• It's Swapping Season (2026-11-05) — A news roundup with a strong serving angle: NVIDIA's context compaction cuts peak context from 32K to 8K tokens on SWE-bench Verified while keeping accuracy, llama.cpp v0.9.0 adds native FP8 and multi-node tensor parallelism via NCCL, and a real-world GLM-5.3 self-host on 56 B200s with NVFP4 and disaggregated prefill/decode cuts ~90% of inference spend; a Baseten router also cut coding costs 72%.
• Inkling 2 Small: 17A-594B trained in 68 hours on 12T tokens, beating Muse Spark (2026-11-06) — Covers a new 594B-total/17B-active MoE model trained end-to-end in NVFP4, with cost-per-task comparisons against competing models. The same roundup reports Inferact serving 7T+ tokens/day on optimized vLLM and Zyphra FP8 kernels on AMD MI355X delivering 1.16–1.76x more prefill throughput.
• Anthropic IPO: Whither Now? (2026-11-07) — Despite the IPO-focused title, reports that Artificial Analysis found nearly 1 in 4 serverless endpoints performing below self-hosted baseline (attributed to output limits and quantization) — directly relevant to self-host vs. serverless decisions. Also covers DeepSeek V4.1 Flash throughput gains on GB300 NVL72 (596→848 tok/s per agent) and a ~21% rise in 1-year H100 contract pricing.
• Fable 5.2 retakes World #1 LLM spot as $ANT IPO roadshow begins (2026-11-10) — Covers Fable 5.2's API price cut ($8/$40 per million tokens, down from $10/$50), plus a speculative-decoding benchmark showing a draft model reaching 2.9–3.8x throughput on SGLang (with mixed community results), and debate over BF16 vs. Q4 quantization for local models.
• Mercor $750m Series D at $20B valuation, $3B run rate (2026-11-11) — The headline is a funding round, but the roundup includes TokenSpeed serving Kimi K3 in NVFP4 at 331 tok/s per GPU on 32 GB300s (1.92x a DCP8 baseline), and notes vLLM's move to the LibTorch Stable ABI — a direct operational win, eliminating rebuilds on every PyTorch release.

Through-line

Two threads run across sources this week. First, speculative decoding is maturing from a single-model trick into a scheduling problem: ForeSpark's joint budget allocation, Kimi K3.1 Pro's DSpark integration, and the draft-model benchmarks covered on Latent Space all point to the same realization — acceptance rate and verification cost need to be tuned jointly with load, not fixed at deploy time. Second, "serving config" is increasingly treated as a live, continuously-optimized system rather than a static deployment artifact — Together's polymorphic inference and continual compiler, vLLM's Kinference control plane, and the arXiv post-mortem on prefix-cache collapse all argue for runtime-adaptive admission control, routing, and compilation instead of one-time tuning.

What's catching your eye this week — the prefix-cache collapse post-mortem, or the push toward continuously-recompiled serving? Drop a comment below.