Elastic MoE scaling, free batch-sorting wins, and DeepSeek's serving stack goes open

This digest covers LLM inference and serving developments from 2026-10-14 to 2026-10-21: arXiv (cs.LG/cs.DC), the vLLM and Hugging Face blogs, Together AI, Modal, and Latent Space.

🔥 Highlights

1. Push-Based KV Transfer to Improve TTFT and TPOT for Disaggregated Inference in vLLM — up to 3x better TTFT for prefill/decode-disaggregated setups.
2. Achieving near-zero-downtime scaling in Elastic Expert Parallelism — MoE autoscaling without SLO breaches.
3. Sort your prompts: a free speed-up for MoE batch inference — a zero-code, zero-cost batching win.
4. KVCompass: A Discovery Plane for Internet-Scale Prefix Cache Sharing in LLM Serving — cross-region cache reuse to kill duplicate recompute.
5. [AINews] DeepSeek DeepInfer: Open Source SOTA Inference running 155T toks/day/cluster making $22m/day at ~80% margin for a 81 day payback — DeepSeek open-sources its production serving stack.

arXiv (cs.LG, cs.DC)

KV cache management

• When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill (2026-10-15) — In a real GLM-5.3 deployment on 8xB300, a misconfigured remote KV cache tier made p99 latency worse (3.96s → 17.3s at 16k context) because a remote cache hit became slower than just recomputing the prompt. The proposed fixes (rank-based sharding, async loading) cut p99 from 83.32s to 0.67s in a hit-heavy scenario — a concrete cautionary tale for anyone running distributed KV cache in production.
• Efficient KV Cache Migration for Geo-Distributed LLM Inference in Collaborative Edge Computing (2026-10-15) — EdgeFlow speeds up KV cache migration between edge servers by sending part of the state and regenerating the rest in parallel, cutting migration time up to 45% on a real testbed. Useful for failover/load-balancing between edge nodes without killing active sessions.
• Understanding KV Residency in Production-Scale Agentic LLM Workloads (2026-10-16) — A production study shows a 1-hour idle cap cuts HBM needed for KV cache by ~3.6x with minimal extra recompute; naive LRU eviction forces up to 63% recompute on "Deep Research"-style workloads, but int4 compression on cold sessions drops that to 13.6%. Hard capacity-planning numbers for anyone sizing a GPU fleet for agentic traffic.
• KVCompass: A Discovery Plane for Internet-Scale Prefix Cache Sharing in LLM Serving (2026-10-16) — A discovery layer that finds, across distant regions, which worker already holds a reusable KV prefix, avoiding duplicate global recompute. Reports 1.3x–14x lower p99 TTFT and up to 4.7x more throughput — relevant for multi-region serving operations.
• TideKV: Delayed KV Eviction via Layer-Wise Chunked Prefill for Memory-Efficient Long-Context LLM Inference (2026-10-17) — Processes prefill in per-layer chunks and only evicts cache at the end of each layer, cutting ~84% of transient memory at 64K context on an A100 with only ~1% TTFT overhead. Lets you run 128K context where post-prefill compression approaches run out of memory — including on edge hardware (tested on Jetson).
• The KV-Surgeon: Compensation-Aware Offline KV-Cache Compaction (2026-10-19) — Offline-compacts the KV cache of a fixed, heavily-reused context, adjusting remaining entries to compensate for the removal's effect on attention. At 50x compaction it gains up to 4 accuracy points over prior attention-matching methods at comparable cost — good for RAG/fixed-context setups with high reuse.

Serving, scheduling, and batching

• Koi: Evidence-Guided, Self-Calibrating Planning for Multi-Cloud, Heterogeneous LLM Inference Fleets (2026-10-17) — A fleet planner combining an LLM reasoning layer with deterministic components to decide where/how to run models across heterogeneous hardware and multiple clouds. On a real cluster it delivered 1.18x the goodput of the best baseline and cut cost-per-token ~20% — a direct FinOps lever for multi-cloud inference.
• CEMA-Router: Infrastructure-Aware Batch Routing for LLM Inference in Cloud-Edge Systems (2026-10-19) — Routes request batches between cloud and edge weighing cost, energy, and batch completion time alongside answer quality. On GSM8K/MMLU it reports up to 33%/19% lower energy use and up to 27%/30% shorter batch completion time vs. state of the art.
• JointServe: Serving Multi-Model Agents on Memory-Constrained GPUs (2026-10-14) — Manages weight shards and KV cache blocks as a single pool under GPU memory limits, with predictive prefetch of the next model an agent is likely to call. Cuts session completion time by 21.7% (up to 49% in replay traces) and raises SLO attainment from 2.8–24.5% to 76.8–100% — relevant for anyone serving multiple models/agents on the same GPU.
• Exact Makespan Bounds and the GPipe Homogeneity Bias for Heterogeneous LLM Inference Pipelines (2026-10-15) — Formally shows the standard GPipe formula overestimates pipeline time on heterogeneous GPU clusters, and concludes that balancing compute across stages matters more for stable throughput than reducing inter-stage communication. Useful for anyone running pipeline parallelism across mixed GPU generations.
• Accelerated Dataflow: scaling inference in streaming data analytics with efficiency (2026-10-15) — A system built on Google Cloud Dataflow combining autoscaling across heterogeneous worker groups with prefix-aware batching, aimed at streaming pipelines with heavy GPU-based LLM inference. Reports 1.5–2x more throughput at the same latency, up to 75% less peak resource use, and 84% lower peak tail latency.

Quantization

• Fractional Power-of-Two Quantization for Efficient and Effective Multiplier-Free LLM Inference (2026-10-15) — Proposes a half-octave weight grid (FPoT) enabling multiplier-free 4-bit shift-add datapaths, staying within ~0.4pp accuracy of uniform integer baselines on Llama-3-8B. Gains are projected via hardware gate counts, not measured latency/throughput — more relevant to accelerator designers than to anyone just swapping GPU kernels.
• SpectrumQuant: Spectrum Alignment for LLM Quantization (2026-10-14) — A data-aware orthogonal transform for post-training quantization of Llama/Mistral, matching leading rotation methods' accuracy while using under 1/3 the GPU memory and much less prep time than learned-rotation methods like SpinQuant/DartQuant. The practical value is in quantization prep cost, not inference itself — no latency/throughput numbers reported.
• PN-Mix: Pair-Norm Guided Mixed-Precision Key Cache Quantization for Efficient Long-Output LLM Inference (2026-10-14) — Channel-pair-guided mixed-precision key-cache quantization reaching ~2.26 average bits (vs. 2.32–2.39 for prior methods) while matching or beating accuracy, with smoother degradation down to 1.5 bits. Directly cuts cache memory/bandwidth footprint for long generations — accuracy-only results, no latency/throughput numbers.

Speculative decoding

• FOLD: Position-Free Far Memory with Local Order for Long-Context Speculative Decoding (2026-10-14) — A training-free method pluggable into existing drafting setups with no retraining. On target-drafter pairs tested up to 256K context, it roughly doubled best-case speedup over autoregressive decoding (2.08x → 4.22x) — a plug-and-play option for anyone already running speculative decoding at long context.
• HyCo: Hybrid Drafting for Cost-Efficient Mobile-Cloud Speculative Decoding (2026-10-15) — Runs the server model's multi-token-prediction head on the phone's NPU combined with a small local model, cutting server GPU time per generated token by 38.6% (Spec-Bench) and 29.2% (32K summarization) vs. server-only self-speculation. Relevant for anyone trying to cut server cost in hybrid mobile-cloud setups.

vLLM blog

• Announcing Humming: Fast, Flexible Kernels for Quantized Inference (2026-10-21) — Ant Group and the vLLM community open-sourced Humming, a kernel library for quantized inference supporting many weight/activation formats on NVIDIA GPUs from Turing onward. Dense W4A16 hits ~3-4x cuBLAS speed at small batch and ~2x Marlin at large batch on H200; an end-to-end DeepSeek-V4.1-Flash test on 8xH20 saw 14-43% higher throughput switching Marlin W4A16 → Humming W4A8 — a real win for quantized MoE inference on older or mixed NVIDIA fleets.
• Introducing vLLM TorchTPU (2026-10-20) — A new open-source backend runs vLLM on Google Cloud TPUs via Google's TorchTPU stack, reusing vLLM's model code, scheduler, and API server with TPU-specific Pallas kernels underneath. No throughput/cost figures in the post itself, but it widens the hardware menu for teams comparing TPU vs. GPU cost/perf for vLLM serving.
• Achieving near-zero-downtime scaling in Elastic Expert Parallelism (2026-10-19) — Elastic EP can now add/remove GPUs from a live MoE deployment with near-zero pause, pre-warming joining workers in the background and swapping ranks via NIXL EP without invalidating CUDA graphs. On DeepSeek V4 Pro/GB200, downtime dropped from ~100s to 0.82s scaling 16→20 GPUs, with p95 TTFT staying under a 5s SLO across 11,391 replayed requests — removes the "scaling event = SLA breach" problem for MoE autoscaling.
• Hyper Optimizing Kimi K3 in vLLM: Moving Fast and Responsibly (2026-10-16) — A new decode path fuses most ops into custom CuTe DSL "megakernels" for Kimi K3 on GB300, cutting per-layer kernel count from 23 to 13. Batch-1 decode with speculation reaches 842 tok/s vs. 488 on stock vLLM (1.73x), and AgentX P90 interactivity roughly doubles — a large, concrete decode-latency win for Kimi K3 on NVIDIA.
• Scaling Kimi K3 on AMD Instinct MI355X: Matching NVIDIA HGX B300 Throughput per GPU on AgentX (2026-10-16) — At a fixed 50 tok/s/user P90 SLO, an 8xMI355X node delivers ~7,567 tok/s/GPU vs. 6,083 for HGX B300 (+24.4%), and using modeled TCO ($1.50 vs. $2.26/GPU-hr) that's +87.4% tokens-per-dollar — a direct GPU-vendor cost comparison for provisioning large-MoE capacity.
• Push-Based KV Transfer to Improve TTFT and TPOT for Disaggregated Inference in vLLM (2026-10-14) — A new push-based NIXL connector moves post-prefill KV-transfer coordination off the critical path: the decode instance pre-registers destination blocks and the prefill instance writes KV cache in directly on a background thread. On a 1P1D Qwen3-32B benchmark, push mode gave up to 3x better TTFT and up to 30% better TPOT vs. pull mode — directly applicable to any prefill/decode-disaggregated vLLM deployment.
• Adaptive Verification Without a Confidence Head (2026-10-14) — Introduces an online-fitted logistic-regression estimator over drafter top-token log-odds that lets Adaptive Verification trim low-acceptance speculative-decoding draft tokens without a separately trained confidence head. Metrics shown are calibration-only (AUC 0.8336, ECE 0.0100); useful for teams running speculative decoding who want adaptive verification without a separate training pipeline, though no hard throughput number is quoted yet.

Hugging Face blog

• Hugging Face ❤️ TorchTPU: TPUs, Natively (2026-10-20) — Transformers/Diffusers/Kernels-Hub models now run natively on TPUs via TorchTPU. A Hub ragged-dot kernel reaches up to 21.5x the default implementation at batch 64 on Qwen3-30B-A3B — a big batch-decode speedup for anyone evaluating TPU as an MoE serving target, though no cost figures are given.
• TorchNeuron: running HF models on AWS Trainium is easier than ever (2026-10-19) — A new PyTorch-native backend for AWS Trainium, integrated into Transformers/Diffusers/Kernels. 89% of the top 100 text-generation architectures now work out of the box (vs. 46% under Optimum Neuron v0.4.5) — mainly a signal that Trainium just got much easier to adopt for cost-efficient inference.
• Sort your prompts: a free speed-up for MoE batch inference (2026-10-19) — Shows that mixing cross-domain prompts in one MoE batch forces the GPU to load more expert weights per decode step; simply sorting/grouping prompts by domain before llm.generate sped up unmodified vLLM by 12-17% on Qwen3-30B-A3B and 7-12% on gpt-oss-120b. A genuinely free, no-code-change throughput win for anyone batching heterogeneous prompts against MoE models.
• Tokenmaxxing BERT: 50x faster tokenization (2026-10-15) — HF's tokenizers team made BERT tokenization ~50x faster by skipping Unicode normalization on ASCII text and caching character-class lookups, with identical output token IDs. Mainly relevant to encoder-model (classification/retrieval) serving pipelines where tokenization is a real fraction of latency.

Together AI blog

• Together Post-Training: frontier-grade model shaping, the way that fits you (2026-10-19) — Launches a managed SFT/RL service with dedicated GPU allocation and multi-LoRA adapter serving on the same dedicated instance. The serving-relevant pieces: FP4-merged checkpoint export for production inference, and multi-adapter serving that cuts the cost of hosting several fine-tuned variants on one GPU.

Modal blog

• Serverless routing made sticky (2026-10-20) — Modal's new "Sticky Sessions" pin each request of a session to the same container via a session token, with the autoscaler counting active sessions rather than raw request volume. Not LLM-specific, but directly useful for stateful inference workloads (voice agents, shared chat) that need to keep decoder/KV-cache state on the same container, cutting cold-start latency and avoiding lost state on long sessions.

Latent Space

• [AINews] DeepSeek DeepInfer: Open Source SOTA Inference running 155T toks/day/cluster making $22m/day at ~80% margin for a 81 day payback (2026-10-21) — DeepSeek open-sourced DeepInfer, a vLLM-derived serving stack behind V4.1 Flash, reporting ~155T tokens/day per cluster, 96% input cache-hit rate, ~400ms TTFT, multi-drafter adaptive speculative-decoding verification (+6%), and SSD-backed shared KV cache with multi-day retention. The densest item of the week for anyone operating inference at scale — self-reported numbers, but vLLM independently confirms decode throughput at 6% of DeepInfer's on GB200, giving a real benchmark to size cache/scheduling decisions against.
• [AINews] not much happened today (2026-10-20) — Covers the vLLM v0.32.0 release (814 commits): a structured-decoding endpoint, low-precision KV cache (NVFP4, 4-bit on ROCm), a Model Runner V2 with a GPU-side n-gram speculator, and multi-API workers giving +43-85% requests/s; precompiled-bytecode images cut container startup 14-18%. A direct throughput/startup-latency win for anyone already on vLLM deciding whether to upgrade.
• [AINews] High Taste Testers Ep. 1 (2026-10-17) — Covers day-0 vLLM serving benchmarks for Reflection Beam 1.0 (1,769 tok/s output on 4xB300, NVFP4, TP4 at concurrency 16) and a +62% decode-throughput gain from LoRA via SkyRL (276→447 tok/s/GPU, decode only). Concrete capacity-planning numbers for anyone evaluating B300 + NVFP4 for serving new models.
• [AINews] Agents disprove Polysemanticity; greatly improve Interpretability & Alignment (2026-10-16) — Cites Artificial Analysis data showing the NVIDIA Vera Rubin NVL72 rack delivering up to ~5.7x the throughput-per-megawatt of GB300 on DeepSeek V4 Pro in high-speed-per-user mode, narrowing to ~1.14-1.45x at peak throughput. A direct cost/energy-per-throughput data point for teams choosing between NVIDIA rack generations for inference at scale.
• [AINews] Modal's $1B ARR, $16B Series D makes it NYC's newest AI Infra Decacorn (2026-10-15) — Notes vLLM's "adaptive verification" now works across multiple speculative-decoding drafters (headless MTP, EAGLE, DFlash, DSpark) without retraining, calibrating verification via top-token probability plus online logistic calibration. Lets you swap speculative-decoding drafters without retraining, cutting the operational cost of maintaining multiple decode-acceleration configs.
• The Programmable Data Center — Evan Conrad, SF Compute (2026-10-20) — An interview on SF Compute's physical GPU compute market, noting a B300 running inference at 50% utilization nets ~$25/hr but can be rented for ~$5-6/hr, and explaining why batch vs. live inference get priced differently based on interruption tolerance. A direct unit-economics data point for buy-vs-rent decisions on inference capacity.

Through-line

The week's clearest pattern is that elasticity and cache locality are becoming first-class serving concerns, not afterthoughts: vLLM's near-zero-downtime Elastic EP, KVCompass's internet-scale prefix discovery, and Modal's sticky sessions all attack the same problem — keeping state (KV cache, session affinity, expert placement) coherent while infrastructure scales up and down underneath it. A second thread is that some of the biggest wins this week cost nothing extra: sorting prompts by domain before batching, or swapping speculative-decoding drafters without retraining, both squeeze real throughput out of existing deployments with zero new hardware or training runs. And DeepSeek's DeepInfer release, alongside vLLM's own v0.32.0 feature set, shows production-grade serving optimizations (SSD-backed KV cache, adaptive speculative verification, low-precision caches) converging across both open-source infra and in-house frontier labs.

What's catching your attention this week — elastic MoE scaling, the free batching trick, or DeepSeek opening up its serving stack? Drop a comment below.