This digest covers LLM inference and serving developments from 2026-10-28 to 2026-11-04. The pattern this week is hard to miss: almost every serious throughput or cost win came from rethinking how the KV cache and expert weights are managed in memory, not from new attention math. Speculative decoding also got a round of re-tuning specifically for mixture-of-experts and hybrid-attention models, which behave differently from the dense models it was originally designed around.
🔥 Highlights
Rimuru: Unified Memory Management for Modern LLM Serving via Elastic Expert Offloading — up to 19x throughput by merging KV cache and expert-weight memory into one pool.
Day 0 support for Mistral Large 4 on vLLM — speculative decoding, NVFP4, and KV offload shipped for a 1T-param model on release day.
HybridSpec: Throughput-Aware Speculative Decoding for Hybrid GDN-MoE Models — speculative decoding re-tuned for hybrid MoE architectures.
Sliding window beats linear attention, but only on short answers — quantifies the real KV-cache tradeoff for long-context serving.
"Weight Space Programming" cuts Autoresearch out of the RSI loop — a 1M-token-context model squeezed onto a 16GB consumer GPU.
arXiv (cs.LG, cs.DC)
HybridSpec: Throughput-Aware Speculative Decoding for Hybrid GDN-MoE Models (2026-11-02) — Instead of a fixed verification length, HybridSpec picks it dynamically per step using draft-acceptance estimates and measured target latency, because hybrid Gated-DeltaNet+MoE models see verification cost explode with length far faster than dense models do (71% growth vs. 4-6% from length 4 to 16). It also adds block-parallel GDN kernels that reconstruct recurrent state in one launch instead of per token. Reports 1.75-1.88x over plain autoregressive decoding and 1.3-1.33x over the prior best speculative method — directly relevant if you're serving Qwen3.6-class hybrid-attention MoE models and tuning speculative-decoding parameters.
Why Speculative Decoding Works Better Than Predicted on Sparse MoE Models (2026-10-28) — An empirical study across 10 sparse MoE models (30B-1T params) finds that fine-grained MoE breaks the textbook rule that speculative-decoding speedup peaks at mid-batch-size; it instead peaks at low batch sizes, traced to expert-routing overlap across draft tokens plus H100 memory-bandwidth and L2-cache reuse effects. The authors' max-entropy model predicts unique-expert counts within 4% error. Useful if your production speculative-decoding speedup curve doesn't match what the dense-model literature predicts — this explains why, and gives a way to model it for capacity planning.
PDScope: Breaking the Prefill-Decode Seesaw for Memory-Bound MoE Inference (2026-10-29) — For CPU-offloaded MoE inference, PDScope decouples prefill (compute-bound, wants aggressive expert prefetch) from decode (I/O-bound, wants conservative PCIe transfer) using phase-specific expert-activation prediction and a compiled, zero-copy async execution pipeline. Reports 2.5-10.3x prefill throughput and 2.1-6.1x decode throughput gains over existing systems — a concrete recipe for running large MoE models without keeping every expert GPU-resident.
Rimuru: Unified Memory Management for Modern LLM Serving via Elastic Expert Offloading (2026-10-29) — Built on top of vLLM, Rimuru replaces fixed memory boundaries between model weights/experts and KV cache with a shared page pool, per-tensor-type "memory views," and a page-table-aware fused MoE kernel that reads non-contiguous expert weights directly. Reports up to 19.03x throughput and a 98.1% average TTFT reduction versus baseline — a serving-framework-level change applicable to any vLLM-based MoE deployment under fluctuating load.
ARK: Avoiding Routing Collisions for KV Cache Transfer in Disaggregated LLM Inference (2026-10-30) — Identifies that KV-cache transfers in prefill/decode-disaggregated serving act as long-lived "elephant flows" that stateless ECMP routing can collide onto the same network spine, stretching transfer time by seconds. ARK is a sender-coordinated path-reservation scheme that spreads concurrent transfers across spines with no switch changes. Cuts mean/P95 flow-completion time by up to 27.3%/34.0% and mean TTFT by 12.9% in RDMA simulation — a network-layer fix easy to miss when optimizing disaggregated serving purely at the scheduler level.
ROCm InferaSim: Cut LLM Serving Trial Costs and Deploy the Best Configuration (2026-11-02) — A benchmarking tool that predicts serving performance from a handful of "anchor" measurements (one per distinct kernel set) and analytically scales to other node counts and configurations, instead of exhaustive trial-and-error. On AMD MI355X with vLLM/SGLang/ATOM, 11 anchors (9.1 GPU-hours) replaced testing 226 configurations (204 GPU-hours) with 8.1% median latency-prediction error, landing a top-5 shortlist within 12% of the best feasible config — a direct cut to the cost of deployment config search.
vLLM blog
- Day 0 support for Mistral Large 4 on vLLM (2026-10-29) — Covers vLLM's stack for serving Mistral Large 4 (1T params, ~49B active, MoE with latent attention and cross-layer KV sharing): FP8/NVFP4 quantized checkpoints with FP8 latent KV cache (NVFP4 ~1.1-1.2x faster decode at TP8, within 2 points of FP8 on evals), a built-in DSpark speculative decoder (~2x throughput at TP8), decode/prefill context parallelism (~7.9x KV capacity, up to 1.6x lower TTFT), NIXL disaggregated prefill/decode on GB300, and HiSparse host-memory KV offload (4.5-5.7x token capacity). Gives concrete, measured levers for anyone sizing a Mistral Large 4 deployment on day one.
Hugging Face blog
Welcome Le Chonk: Mistral Large 4 on Hugging Face (2026-10-29) — HF's companion piece to the vLLM post, with vLLM serve flags (tensor/context/expert parallelism, batching limits, prefix caching), GGUF quantization levels (Q2_K through BF16) with llama.cpp memory/throughput figures on Apple Silicon and multi-GPU, and Inference Endpoints benchmarks on 8xH200. A concrete menu of deployment paths with hardware-specific cost/performance figures for teams deciding how to run this model.
A Low-Cost Decision Model on CPU: Optimizing DiffusionGemma with vLLM - Part I (2026-10-29) — Serves a 26B-A4B decision model on CPU (Intel AMX, BF16) via vLLM instead of GPU, including a new
single_pass_readsflag (upstream PR in review) that merges prompt and answer-scoring into one forward pass, cutting latency 18.8% at concurrency 1 and raising throughput 5.5-15% at concurrency 2-20. A 24-vCPU node (~$996/mo) cuts always-on cost 79% versus an A100 replica (~$4,754/mo) with 5-13% lower cost-per-1000-decisions under moderate load — useful for teams deciding CPU vs. GPU for latency-tolerant classification/decision workloads.Sliding window beats linear attention, but only on short answers (2026-11-02) — Compares sliding-window attention against a hybrid linear-attention mixer ("Bifocal") at equal KV-cache memory budget, distilled from Qwen3-4B. At matched cache size, sliding-window wins on short answers but Bifocal pulls ahead on long math answers (15.5% vs 13.8% pass@1 on AIME/HMMT) thanks to retained long-range state; on H100, both hold ~195 tok/s decode from 1K-128K context while full attention drops to 92 tok/s at 128K. Concrete guidance for choosing attention/KV-cache architecture for long-context serving — sliding window is a cheap strong default, but the hybrid can buy better long-generation quality at a small prefill-speed cost and no extra decode cost.
Together AI blog
Nothing new in this window — the latest post predates it and was already covered.
Modal blog
Nothing new in this window — checked the blog index and sitemap directly; the most recent post predates the window.
Latent Space
"LLMs can read & write neural network weights": "Weight Space Programming" cuts Autoresearch out of the RSI loop (2026-11-03) — Buried in the daily roundup: a patched Strata inference engine runs DeepSeek V4.1 Flash with 1M-token context on a consumer RX 9070 XT (16GB VRAM) via KV-cache splicing of stable prompt segments plus resident-only expert routing, dropping cold-start from ~100s to ~5s and SSD reads from 58GB to 1.3GB per prompt. Also notes a 763B MoE quantized to ~1.05 bits/param (72.1GB) losing only 1-4 points on LiveCodeBench/SWE-bench. Concrete evidence that aggressive KV-cache tricks can make huge-context inference viable on consumer hardware, and a data point on how far sub-2-bit quantization can go before quality degrades.
"Google admits AI Mode hurts Search revenue; -15% $500B drop worst since 2004 IPO. AI Bear Market coming?" (2026-10-31) — Inference-relevant items inside the roundup: prompt caching cuts a typical agent task's cost 3.6x ($27 to $7.44), with each cache invalidation adding ~5% to the bill; Nemotron 3.5 Super hits 4,856 output tok/s/GPU on GB200 at NVFP4 while keeping 99.62% of BF16 accuracy; and turbopuffer's switch from FIFO to LFU chunk caching roughly doubled byte hit ratio and halved p99 latency. Hard numbers for justifying prompt-caching investment and NVFP4 as a near-free accuracy/throughput trade.
"Building AIE CODE: The top Coding Agent Conference" (2026-10-30) — Inference-relevant items inside the roundup: vLLM's day-0 Mistral Large 4 support (see above) with speculative decoding yielding 1.92-2.23x output throughput; SGLang kernel fusion on GB300 NVL72 nearing 1,000 tok/s/user; a vLLM tuning pass nearly doubling Nemotron 3.5 Super VL throughput (6,058 to 11,840 tok/s on one GB300); and SkyPilot's in-rack spare-node scheme recovering ~27% extra usable capacity on a 128-rack cluster. Multiple concrete throughput-tuning levers with before/after numbers for serving cost decisions.
"Gemini 4 Carbon tops leaderboards, launches for cyber defenders." (2026-10-29) — Inference-relevant items inside the roundup: the AKV serving engine's cache "drop/repos" operations cut agent cost 27.7% and improve P95 latency 1.51x; SGLang's HiCache CPU-DRAM KV offload adds 20-40% more cacheable tokens under concurrent agent load on B200; MLX kernel updates make batched decoding 2-3x faster; and Modular open-sourced its full MAX serving stack (compiler, runtime, hardware backends) royalty-free. KV-cache offloading and engine-level cache ops are now showing double-digit cost/latency wins, and a major serving stack going fully open-source changes the framework-selection calculus.
Through-line
Three independent threads converge this week: the KV cache has become the primary lever for both cost and latency (elastic expert/cache memory pooling in Rimuru, network-aware transfer scheduling in ARK, host-memory offload in the Mistral Large 4 launches, cache "drop/repos" ops and HiCache offload surfacing in Latent Space's roundups, and consumer-GPU KV splicing for million-token context); speculative decoding is being actively re-derived for MoE and hybrid-attention models rather than assumed to behave like it does on dense transformers (HybridSpec, the sparse-MoE speedup study, DSpark on Mistral Large 4); and quantization keeps pushing further than expected, from NVFP4 holding near-BF16 accuracy at scale to sub-2-bit MoE checkpoints still clearing benchmarks within a few points. None of these are algorithmic breakthroughs in attention itself — they're memory-and-scheduling engineering, and that's where the throughput is coming from right now.
What's catching your eye this week — the KV-cache offloading trend, or the MoE-specific speculative decoding retuning? Drop a comment below.
Top comments (0)