Multi-Turn Memory, Mixed Precision, and Safer Rollouts — Serving Catches Up to Agentic Workloads

This edition of the LLM Inference Digest covers new developments from 2026-10-07 to 2026-10-14 across arXiv, the vLLM and Hugging Face blogs, Together AI, Modal, and Latent Space.

🔥 Highlights

1. Improve TTFT for Multi-Turn Agentic Workloads with Bi-Directional KV Transfer in vLLM — lets decode nodes hand KV blocks back to prefill
2. DeepSeek-V4.1-Flash on vLLM: 5x Agentic Throughput Since Day 0 — three kernel tricks, 5.3x throughput under TPS limits
3. Endpoint Candidates: evaluate and promote inference endpoints — built-in traffic-mirrored canarying for serving changes
4. Vesta: Queueing-Aware LLM Scheduling for Real-World Workloads — balances cache locality vs. queueing, cuts TTFT sharply
5. Run fast, private agents on-device with LFM2.5-24B-A2B — sparse MoE hits 21K tok/s on one GPU

arXiv (cs.LG, cs.DC)

• MultiAlloc: Budget-Aware Precision Allocation for KV Cache Quantization (2026-10-13): splits a fixed KV quantization budget between layer-wise precision boosts and token-wise preservation, using lightweight offline profiling instead of uniform compression. Outperforms single-granularity baselines on Mistral/Llama/QwQ at matched memory budgets — useful when tuning KV memory limits per model rather than applying one blanket bit-width.
• E8 Lattice Key Quantization for Low-Bit KV-Cache Compression (2026-10-11): replaces scalar KV quantization with vector quantization on an E8 lattice that exploits cross-dimension correlation, reaching a 2-bit keys/2-bit values config (previously needing 3-bit keys) for roughly 6x compression with decode throughput matching scalar baselines. Relevant for anyone pushing long-context serving memory limits.
• The Price of Precision: A Calibrated Cost Model for Mixed-Precision LLM Serving (2026-10-11): builds a cost model that identifies whether compute, weight loading, or KV-cache access is the real bottleneck at a given batch size/context length, then picks mixed-precision configs via dynamic programming instead of guessing from byte-width alone. Practical for choosing precision per-layer or per-component.
• MLAfication: Efficiently Enabling High-Fidelity MLA for Modern LLMs (2026-10-12): converts standard multi-head attention into Multi-Head Latent Attention (DeepSeek-style), also handling QKNorm and MoE architectures that earlier conversion methods couldn't. Claims up to 5.35x serving throughput and ~95% memory reduction combined with quantization — a retrofit path for existing checkpoints instead of retraining with MLA from scratch.
• FORGE: Procedural Weight Generation for Sparse MoE Inference (2026-10-12): regenerates MoE expert weights on-the-fly from compact rotor parameters instead of storing full expert tensors, cutting the memory footprint that forces multi-node parallelism with all-to-all overhead. Reports 1.3-2.8x higher throughput at matched GPU count and 10-15% lower decode latency.
• Quota-Based Aging for Prediction-Guided Scheduling in LLM Serving (2026-10-13): a scheduling policy that keeps the mean-latency benefit of predicted-short-first admission while bounding tail latency for long requests via quota-based promotion, addressing the classic "shortest-job-first starves long jobs" problem. A solid read for teams tuning their scheduler's fairness/latency trade-off.
• Vesta: Queueing-Aware LLM Scheduling for Real-World Workloads (2026-10-10): decouples per-session KV-cache "ownership" from per-request execution placement, balancing prefix-cache locality against queue congestion using queueing theory instead of always routing to the cache-affine node. Reports a 14% throughput gain and 57.8% mean TTFT reduction colocated, plus up to 29% P99 TTFT reduction disaggregated.
• SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference (2026-10-08): notes that Hessian-guided pruning calibrated on natural text mismatches the self-generated token distributions seen during actual decoding, and fixes this with decode-time calibration plus custom sparse matrix-vector kernels aimed at the memory-bound decode phase rather than prefill. Up to 1.48x wall-clock decode speedup on A100s.
• Adaptive DHiSparse: Speculative Decoding over a Load-Adaptive KV Hierarchy (2026-10-09): combines speculative decoding with a hierarchical sparse KV cache that dynamically shifts entries between GPU memory and overflow based on load instead of a fixed resident/evicted split. At 500K context and concurrency 512, reports 1.85x throughput over resident serving — relevant when long-context, speculative decoding, and memory pressure collide.
• CAST: Cost-Aware Speculative Trees from One-Pass Block Drafters (updated 2026-10-10): packs alternate candidates from a block drafter's single forward pass into a verification tree whose width adapts at runtime based on measured hardware latency, rather than a fixed, manually-tuned tree shape. Up to 43% speedup with no changes to model/drafter/decoding algorithm, provably preserving the output distribution — a drop-in upgrade for teams already running block speculative decoding.
• Reproducible LLM Inference Benchmarking: A Sequential Isolation Protocol for Regression Testing (2026-10-07): a measurement methodology, not a new serving technique — a "sequential isolation" benchmarking protocol that cut coefficient of variation from 15.2% to 2.2% on vLLM/A100 across context sizes and concurrency levels. Useful for anyone building CI-style regression gates for inference latency/throughput, where noisy benchmarks currently make such gates unreliable.
• Dissecting the Carbon-SLO Tension in Geo-Distributed LLM Serving (2026-10-10): four schedulers (from a greedy heuristic to a facility-level MILP) for carbon-aware request routing across geo-distributed datacenters, built on a simulator that models actual inference-engine queueing/batching dynamics. Reports up to 60% carbon reduction without violating SLOs — relevant for infra teams with multi-region deployments and sustainability targets.

vLLM Blog

• Improve TTFT for Multi-Turn Agentic Workloads with Bi-Directional KV Transfer in vLLM (2026-10-12): until now KV cache only flowed prefill → decode; this adds the reverse path, letting the decode node hand already-computed KV blocks back to the prefill node on later turns of a conversation so the prefix doesn't get recomputed, coordinated by a KV-cache-aware router with peer-to-peer tensor transfer over EFA on P5en. In multi-turn agentic workloads, this cut TTFT up to 1.6x on Qwen3-32B (778ms→460ms at 24k tokens) and up to 3.3x on GLM-5-FP8 (2334ms→702ms at 20k tokens) — a real latency/cost win for serving agents with long conversation history, no extra hardware required.
• DeepSeek-V4.1-Flash on vLLM: 5x Agentic Throughput Since Day 0 (2026-10-07): three optimizations landed in three weeks — "SWA bounded replay" (recomputing only the last 128 tokens instead of caching the full sliding-window KV, +30-40% prefill), DeepSeek's own MegaAttention (1.14-1.51x) and Mega-Gate (1.18-1.31x) kernels, plus operation fusion and CUDA graphs to cut kernel-launch overhead. Net result: 1.9x speedup at low concurrency and 5.3x throughput under a 150 TPS constraint, with ~70% lower TTFT at 100K throughput — a direct cost-per-token decision for anyone serving DeepSeek-V4.1-Flash in production.
• vLLM Support for NVIDIA Vera Rubin NVL72: 7.8x Throughput over GB200 NVL72 (2026-10-09): day-0 support for NVIDIA's new Vera Rubin NVL72 cluster, with "locality domains" tuned for MoE routing and FlashInfer 0.7.0 kernels adapted for Rubin while staying compatible with existing Blackwell kernels; the platform offers 5x more NVFP4 FLOPS and 2.4x more memory bandwidth than GB200. Reports 7.8x throughput per GPU over GB200 NVL72 on AgentX and 3.7x VLM throughput over GB300 on MLPerf v6.1 — relevant to next-generation hardware planning, though it's more a preview of future hardware than something actionable today.

Hugging Face Blog

• Run fast, private agents on-device with LFM2.5-24B-A2B (2026-10-13): a 24B MoE model with only 2B active parameters per token (40 layers, 64 experts, top-4 routing) and context expanded from 32K to 131K, built to run on-device. Concrete throughput numbers — 184 tok/s on an Apple M5 Max and up to 21K tok/s aggregate on a single NVIDIA B300 — give a direct data point for choosing a sparse MoE over a larger dense model when serving low-cost, low-latency agents locally or on one GPU.
• Other posts this week (Falcon ASR, Nemotron fine-tuning for IOI/IMO competition math, a retrospective on a model release) are model/training content without serving-performance relevance, so they're skipped here.

Together AI Blog

Nothing new in the window — the latest post ("Expanding our enterprise inference capacity with IBM Cloud and NVIDIA") predates this edition's cutoff and was already covered previously.

Modal Blog

• Endpoint Candidates: evaluate and promote inference endpoints (2026-10-12): a feature for staging alternative deployments of a Dedicated Endpoint — a new checkpoint or a different model entirely — that mirrors a sample of live production traffic to measure quality, latency, throughput, and cost before promotion. The underlying engineering extends Modal's Pingora-based proxy so each mirrored candidate reads from its own bounded, deduplicated buffer with per-process memory budgets, so a slow candidate can't add latency to production traffic. A built-in, low-risk canary/A-B mechanism for quantization tweaks, model upgrades, or engine config changes, without building custom traffic-mirroring infrastructure.
• Toward RL on mixed-precision, mixed-accelerator samplers (2026-10-13): describes training an open-weight model (Qwen3.6-35B-A3B) where the RL "samplers" — the inference/rollout side, which consumes 80-85% of GPUs in large RL runs — run heterogeneously across B200/B300 (NVFP4), H100/H200 (FP8/BF16), and A100/RTX PRO 6000 (BF16), each with an FP8 KV cache, while a single BF16 trainer learns from all of them via off-policy correction. Matches homogeneous-sampler quality at lower cost — a concrete recipe for exploiting idle or cheaper heterogeneous GPU capacity and low-precision serving kernels for inference-heavy workloads.

Latent Space

Nothing in the window met the inference-engineering bar — this week's posts covered agent orchestration patterns, AI news roundups, and robotics, none touching quantization, KV-cache, scheduling, or serving throughput/cost directly.

Through-line

The common thread this week is serving built around conversations, not single requests: vLLM's bidirectional KV transfer and arXiv's Vesta scheduler both attack the same problem — multi-turn, agentic sessions that keep re-paying prefill costs or get stuck behind naive queueing — from the serving-engine side and the scheduling-theory side respectively. Meanwhile Modal's Endpoint Candidates and the DeepSeek-V4.1-Flash kernel work point at the other half of the same story: once you've picked a faster serving path, you need a safe way to test it against real traffic before flipping it on for everyone.

What are you seeing in your own serving stack this week? Drop a comment below.

Comments

[Manoj Kumar Kagitha]

The Modal endpoint candidates item is the one that caught me. Traffic-mirrored canarying for serving changes is exactly the safe rollout pattern most teams hand-roll badly.