DEV Community: AI Tech News

Day-0 Model Support Is a Fork: What 26 NVIDIA Dynamo Snapshot Builds Say About Pinning Your LLM Serving Stack

AI Tech News — Mon, 09 Nov 2026 14:05:46 +0000

TL;DR

NVIDIA Dynamo ships two kinds of releases: stable versions (v1.6.2 on 2026-11-05) and short-lived model snapshot builds such as v1.7.0-kimi-k3.1-pro-dev.1 (2026-11-06). I pulled every Dynamo release from the GitHub API, the engine release dates from PyPI, and classified what each snapshot build sits on. Everything below is reproducible with the script in this post.

Stable pins are never current. Across 12 stable releases, the pinned vLLM was a median 30 days old on release day (range 17 to 56), and every single one had at least one newer vLLM minor already on PyPI. SGLang looked the same: 11 of 12 had a newer release available.
Snapshots outnumber stables 2.6 to 1. Between 2026-07-17 and 2026-11-06 there were 10 stable releases and 26 model snapshot builds. All 26 carry a notice that they are not for production.
Most snapshots are not on a plain engine release. Only 10 of 26 build on an unmodified released vLLM, SGLang or TensorRT-LLM tag. The other 16 use a patched release (3), a nightly or raw commit (5), or a model-specific image or fork (8).
The fleet-level risk is documented, not hypothetical. One build states that its patched NIXL transfer protocol 14 is not compatible with unpatched protocol 13 workers. Another requires matching KV block size and speculative decoding settings on prefill and decode.
What I did not do. I ran no GPU workload and deployed none of these images. This is a study of release metadata, with my own classification, and the snapshot builds are explicitly labeled as early access.

What exactly did I measure, and where does the data come from?

Three sources, all public. The Dynamo release list and notes come from the GitHub releases API for ai-dynamo/dynamo (73 entries as of 2026-11-09). Engine release dates come from PyPI upload timestamps for vllm and sglang. The snapshot classification comes from the header lines of each build's release notes, the ones that start with **vLLM**:, **SGLang**: or **TensorRT-LLM**:.

Two definitions matter. A stable release is any non-prerelease entry. A snapshot is any prerelease entry whose tag names a model, for example v1.6.0-deepseek-v4.1-flash-dev.2-post.1. The snapshot window is the 16 weeks from 2026-07-17 to 2026-11-06, the newest release on the day I ran this. Earlier builds often omit the engine header lines, so I left them out. The stable pin analysis uses every stable release whose notes include a pin table, 12 of them since 2026-04-23.

For stable releases, the notes include a "Key Dependencies" table listing the pinned SGLang, TensorRT-LLM and vLLM versions. I parse that table and compare the pinned engine's PyPI upload date with the Dynamo release timestamp.

import json, re, urllib.request
from collections import Counter
from datetime import datetime, timezone

def get(url):
    req = urllib.request.Request(url, headers={"User-Agent": "audit/1.0"})
    return json.load(urllib.request.urlopen(req, timeout=60))

rels = get("https://api.github.com/repos/ai-dynamo/dynamo/releases?per_page=100")
UTC = timezone.utc
when = lambda s: datetime.fromisoformat(s.replace("Z", "+00:00"))

def pypi_dates(pkg, pat):
    out = {}
    for v, files in get(f"https://pypi.org/pypi/{pkg}/json")["releases"].items():
        if files and re.fullmatch(pat, v):
            out[v] = min(datetime.fromisoformat(f["upload_time"]).replace(tzinfo=UTC) for f in files)
    return out

vllm = pypi_dates("vllm", r"0\.\d+\.\d+")
sglang = pypi_dates("sglang", r"0\.5\.\d+(\.post\d+)?")
third = lambda v: int(v.split(".")[2])

# 1. How stale is the engine a Dynamo stable release pins, on release day?
ages = []
print("stable  vLLM pin  age  newer | SGLang pin  age  newer")
for r in sorted(rels, key=lambda r: r["published_at"]):
    if r["prerelease"] or "| :---" not in (r["body"] or ""):
        continue
    row = re.search(r"\| \*\*v([\d.]+)\*\* \| `v?([^`]+)` \| `v?([^`]+)` \| `v?([^`]+)`", r["body"])
    if not row:
        continue
    tag, sg, trt, vl = row.groups()
    t = when(r["published_at"])
    vl_newer = len({v.split(".")[1] for v, d in vllm.items() if int(v.split(".")[1]) > int(vl.split(".")[1]) and d <= t})
    sg_newer = sum(1 for v, d in sglang.items() if "post" not in v and third(v) > third(sg.split(".post")[0]) and d <= t)
    ages.append((t - vllm[vl]).days)
    print(f"v{tag:6} {vl:8} {(t - vllm[vl]).days:4}d {vl_newer:3}   | {sg:13} {(t - sglang[sg]).days:4}d {sg_newer:3}")

ages.sort()
print(f"{len(ages)} stable releases, vLLM pin age: min {ages[0]}d, median {(ages[len(ages)//2 - 1] + ages[len(ages)//2]) / 2:g}d, max {ages[-1]}d")

# 2. What do the model-specific snapshot builds sit on?
def kind(line):
    line = re.sub(r",? ?NIXL.*", "", line).lower()
    if "ghcr.io" in line:
        return "model-specific"
    if "nightly" in line or re.search(r"build `[0-9a-f]{7,}`", line):
        return "nightly or commit"
    if "patch" in line or "backport" in line:
        return "released + patches"
    if re.search(r"(vllm-openai|sglang):v?\d|`v?0\.\d+\.\d+|`1\.\d+\.\d+rc", line):
        return "released"
    return "model-specific"

RANK = ["model-specific", "nightly or commit", "released + patches", "released"]
snaps = Counter()
for r in rels:
    if not r["prerelease"] or r["published_at"] < "2026-07-17":
        continue
    lines = re.findall(r"^\*\*(?:vLLM|SGLang|TensorRT-LLM)\*\*:(.*)$", r["body"] or "", re.M)
    kinds = [kind(l) for l in lines]
    if kinds:
        snaps[min(kinds, key=RANK.index)] += 1
print("\nsnapshots since 2026-07-17:", sum(snaps.values()))
for k in RANK:
    print(f"  {k:20} {snaps[k]}")

Output when I ran it on 2026-11-09:

stable  vLLM pin  age  newer | SGLang pin  age  newer
v1.0.2  0.16.0     56d   3   | 0.5.9           58d   1
v1.1.0  0.19.0     31d   1   | 0.5.10.post1    25d   0
v1.2.0  0.20.1     29d   2   | 0.5.11          27d   1
v1.3.0  0.23.0     38d   2   | 0.5.14          26d   1
v1.4.0  0.26.0     21d   1   | 0.5.16          21d   1
v1.4.1  0.26.0     28d   1   | 0.5.16          28d   2
v1.4.2  0.26.0     34d   2   | 0.5.16          35d   2
v1.5.0  0.28.0     26d   1   | 0.5.18          30d   2
v1.5.1  0.28.0     41d   3   | 0.5.18          46d   3
v1.6.0  0.31.0     17d   1   | 0.5.21          21d   1
v1.6.1  0.31.0     22d   1   | 0.5.21          26d   1
v1.6.2  0.31.0     31d   2   | 0.5.21          35d   2
12 stable releases, vLLM pin age: min 17d, median 30d, max 56d

snapshots since 2026-07-17: 26
  model-specific       8
  nightly or commit    5
  released + patches   3
  released             10

The "newer" column counts distinct vLLM minor versions (or SGLang 0.5.N releases) newer than the pin that were already on PyPI at the moment Dynamo shipped.

How stale is the engine inside a Dynamo stable release?

Always somewhat, and it does not shrink. The vLLM pin was 17 days old at the freshest (v1.6.0) and 56 days old at the stalest (v1.0.2). The median across 12 releases is 30 days. The three most recent releases sit at 17, 22 and 31 days, but all three pin the same vLLM 0.31.0, so that growth is just the clock running between a minor release and its patches.

Take the newest stable as a concrete case. Dynamo v1.6.2 (2026-11-05) pins vLLM v0.31.0, SGLang v0.5.21 and TensorRT-LLM v1.3.0rc29. On that day PyPI already had vLLM 0.32.0 (2026-10-18) and 0.33.0 (2026-11-02), plus SGLang 0.5.22 and 0.5.23. TensorRT-LLM had already started its 1.4.0 release candidate line on 2026-10-22. None of that is a criticism. A stable release that bundles three engines, NIXL, UCX and Kubernetes recipes has to freeze something, and v1.6.2 says plainly that it is a patch release refreshing OS and Python packages.

The point is what the gap does to anyone who needs a model that shipped after the freeze.

Why does a stable pin turn into a stream of model snapshots?

My reading, and I flag it as interpretation rather than something the notes state: a brand new model usually needs engine changes (a kernel, a tool-call parser, a tokenizer path, a cache layout) that the pinned engine does not have yet. Waiting for the next stable means waiting out the pin lag above. So the project cuts a branch, bases it on whatever engine build already runs that model, and publishes it under a tag that carries the model name.

The volume shows how often this happens. In the 16 weeks I looked at, snapshots appeared in every month, and in September and October they outnumbered stable releases by a wide margin.

Every one of the 26 snapshot builds in the window carries a notice that it is not for production, and the ones I read in full also say it is not QA-gated. So this is not a story about a vendor shipping broken things. It is a story about where the bleeding edge of model support actually lives: on branches that are labeled as early access, built on engine bases that your normal upgrade process has never seen.

What do those snapshot builds actually sit on?

I classified each build by the least conventional engine line in its header, using this precedence: model-specific image or fork, then nightly or raw commit, then released tag with patches, then plain released tag. The classifier in the script above is a few regular expressions, and its totals matched my hand labels exactly (10, 3, 5 and 8).

Base under the snapshot	Builds	Examples from the notes
Plain released engine tag	10	`v1.6.0-beam-1.0-501b-dev.1` on vLLM v0.31.0 and SGLang v0.5.21
Released tag plus patches	3	`v1.4.1-a.x-k2-post.1` on vLLM 0.26.0 with four patches
Nightly or raw commit	5	`v1.7.0-mistral-large-4-dev.1` on a `cu134-nightly` vLLM image
Model-specific image or fork	8	`v1.7.0-kimi-k3.1-pro-dev.1` on `vllm/vllm-openai:kimi-k31`

Sixteen of 26 is 62 percent. Two caveats keep this honest. First, the labels come from header lines that different release authors fill in with different levels of detail, for example "MiniMax M3.1 Flash build" with no version at all, which I counted as model-specific. Second, a "plain released tag" build can still carry Dynamo-side patches in its runtime layer, which the header does not always say. Treat 62 percent as a rough measure of how often the base is non-standard, not an exact one.

Does the same model keep changing its base?

Yes, and the sequence is instructive. DeepSeek-V4.1-Flash has three Dynamo builds in the window:

Build	Date	Engine base named in the notes
`v1.6.0-deepseek-v4.1-flash-dev.1`	2026-09-12	SGLang `dev-dsv41` image, NIXL v1.4.0
`v1.6.0-deepseek-v4.1-flash-dev.2`	2026-10-08	vLLM v0.31.0 image, NIXL v1.3.2 (connector protocol 14)
`v1.6.0-deepseek-v4.1-flash-dev.2-post.1`	2026-11-05	vLLM nightly `ac9126e...` dated 2026-10-01, NIXL v1.3.2 (protocol 14)

The third build's notes say it returns to a pinned nightly instead of the v0.31.0 base used by dev.2. In other words, the project moved a model from a released engine to a nightly. The notes I read do not say why it moved back to a nightly. They list a fix so prefill health probes stop leaking KV leases and refreshed recipes, so I will not guess beyond that. The practical lesson is that "which engine version is this model on" can change between two builds with nearly identical names.

Kimi-K3 shows the opposite direction. Its first build (2026-07-27) used the model-specific image vllm/vllm-openai:kimi-k3. By 2026-09-09, 44 days later, the build used v0.28.0-ubuntu2404, a normal released tag, with SGLang on v0.5.17. Model-specific images look like bridges that later converge onto released engines, at least for this model.

Where does this bite a multi-node deployment?

At the seams between workers. Disaggregated serving moves KV cache from a prefill worker to a decode worker, so both ends must agree on the details. The release notes name several such constraints:

The DeepSeek post.1 notes: the patched NIXL transfer protocol (14) is not compatible with unpatched protocol 13 workers, and all workers in a KV transfer must use that runtime.
The Mistral Large 4 notes: disaggregated recipes require matching block size (64) and speculative decoding settings on prefill and decode, and the Vera Rubin recipes store KV in NVFP4, which changes numerics, so re-run an accuracy check after any model or image change.
The Kimi-K3.1-Pro notes: this build relies on NIXL v1.5.0, which the notes say is required for Kimi strided descriptors.

NIXL itself is a moving part. Of the 26 snapshot builds, 20 name a NIXL version, and they name six different ones.

Even the stable v1.6.2 mixes NIXL versions by backend: v1.5.0 for the frontend and SGLang image, v1.3.1 for TensorRT-LLM and v1.4.1 for vLLM. That is a normal packaging decision when each backend image is tested on its own. It becomes a trap only if you assume one "Dynamo version" implies one transfer stack across every worker you run.

How do you keep a mixed fleet from silently breaking?

Five habits, none of which depend on Dynamo specifically.

Pin by digest, record the digest. Most snapshot notes give a branch tip and a base image, and several pin a sha256 digest. Copy that into your deployment record. A tag like kimi-k31 can move; a digest cannot.
Treat prefill and decode as one unit. If you upgrade one role, upgrade the group. Never leave a rollout half done across a KV transfer boundary.
Gate on the fields the notes call out. Image digest, KV block size, speculative decoding config and transfer protocol version are the ones named above. Compare them across every worker in a group before traffic is allowed.
Re-run accuracy after any image change that touches numerics. The Mistral notes say it directly for NVFP4 KV. Throughput numbers from a recipe do not tell you whether outputs moved.
Keep experimental builds off the default path. Route them through a separate pool or namespace so a snapshot cannot become your production baseline by accident.

Here is a small gate for item 3. The manifest is illustrative, written for this post, not taken from any real deployment. One group has a deliberate mismatch.

import sys
from collections import defaultdict

MUST_MATCH = ("image_digest", "kv_block_size", "spec_decode", "nixl_protocol")

fleet = [  # illustrative manifest, not a real deployment
    {"name": "p0", "group": "g1", "role": "prefill", "image_digest": "sha256:aa11", "kv_block_size": 64, "spec_decode": "off", "nixl_protocol": 14},
    {"name": "d0", "group": "g1", "role": "decode",  "image_digest": "sha256:aa11", "kv_block_size": 64, "spec_decode": "off", "nixl_protocol": 14},
    {"name": "p1", "group": "g2", "role": "prefill", "image_digest": "sha256:aa11", "kv_block_size": 64, "spec_decode": "off", "nixl_protocol": 14},
    {"name": "d1", "group": "g2", "role": "decode",  "image_digest": "sha256:bb22", "kv_block_size": 32, "spec_decode": "off", "nixl_protocol": 13},
]

groups = defaultdict(list)
for w in fleet:
    groups[w["group"]].append(w)

bad = 0
for g, workers in sorted(groups.items()):
    for field in MUST_MATCH:
        values = {w["name"]: w[field] for w in workers}
        if len(set(values.values())) > 1:
            bad += 1
            print(f"FAIL group={g} field={field}: {values}")
print("OK" if not bad else f"{bad} mismatches")
sys.exit(1 if bad else 0)

FAIL group=g2 field=image_digest: {'p1': 'sha256:aa11', 'd1': 'sha256:bb22'}
FAIL group=g2 field=kv_block_size: {'p1': 64, 'd1': 32}
FAIL group=g2 field=nixl_protocol: {'p1': 14, 'd1': 13}
3 mismatches

It exits with status 1 on any mismatch, so it drops straight into a CI step or an admission check. Extend MUST_MATCH with whatever your engine's notes call out for your model.

What are the limits of this analysis?

It reads release notes, not behavior. I did not boot any of these images, so I cannot say whether a snapshot is faster, slower or more fragile than the stable build, and nothing here compares their performance. The classification depends on how consistently release authors fill in the header lines. The age calculation uses PyPI upload times, which can differ by hours from GitHub tag times. Finally, the window is 16 weeks long, so a different window could shift the percentages.

FAQ

Is it safe to run a Dynamo model snapshot build in production?
Every one of the 26 builds I checked carries a notice that it is not for production, and the notes I read in full call them experimental and not QA-gated. Use them for evaluation and early feedback, and keep them off your default serving path.

Why does Dynamo ship model-specific builds at all?
The notes do not give a general reason. The pattern in the data fits a simple explanation: new models need engine changes sooner than a stable release can absorb them, so a branch carries the needed base until the change lands in a released engine.

How old is the vLLM version pinned in a Dynamo stable release?
Across the 12 stable releases with a pin table, 17 to 56 days old on release day, median 30. Dynamo v1.6.2 pins vLLM 0.31.0, while 0.32.0 and 0.33.0 were already on PyPI.

Can I mix workers from different Dynamo builds in one disaggregated deployment?
Not safely without checking. The DeepSeek-V4.1-Flash post.1 notes say workers on the patched NIXL protocol 14 are not compatible with unpatched protocol 13 workers and that all workers in a KV transfer must use the same runtime.

How do I reproduce the numbers?
Run the script in this post. It needs only the Python standard library, one GitHub API call for the release list and two PyPI JSON calls. Unauthenticated GitHub access is enough for one run.

What should I record when I deploy a snapshot build?
The release tag, branch tip commit, base image digest per backend, NIXL version and transfer protocol, KV block size, and speculative decoding settings. Those are the fields the notes treat as compatibility-relevant.

Sources

vLLM 0.33 Ships an ABI-Stable Wheel, but PyPI Still Pins torch==2.14.0: What You Can Actually Swap

AI Tech News — Sun, 08 Nov 2026 14:15:19 +0000

TL;DR

On November 6 the vLLM team, with the Red Hat and Meta PyTorch teams, published how vLLM's CUDA build moved to the LibTorch Stable ABI. Since v0.33.0 (PyPI upload 2026-11-02), the CUDA wheel is meant to keep working when you replace the torch wheel with a newer compatible one. I checked what that means in practice.

I reproduced the audit. I downloaded the 336 MB x86_64 wheel from PyPI and ran torch-abi-audit 0.4.0 on it. Result: Torch ABI: STABLE, Min torch: 2.13.0, 13 extensions, 0 unstable symbols. It matches the blog post's output line for line.
One extension sets the floor. Three extensions need torch 2.10 or newer, six need 2.11, and one (QuTLASS) needs 2.13. The wheel's floor is the highest of those.
The package metadata did not follow. vLLM 0.33.0 on PyPI still declares torch==2.14.0 and torchvision==0.29.0. A binary that can run on newer torch is not the same as a package that lets pip install it.
The old rebuild tax was real. Across four torch releases, vLLM took 33, 34, 44 and 60 days to ship the first release that pins each one. torch 2.12.0 was never pinned at all.
What I did not do. I ran no GPU test and did not swap torch under a running server. The blog's own demo is one 135M model generating 8 tokens in eager mode.

What did vLLM actually change in 0.33?

vLLM's C++ and CUDA code calls into LibTorch constantly. Before this work, a change to LibTorch's binary interface could force a vLLM rebuild even when the Python-visible API was identical. The post describes the fix as three layers of PyTorch's stable ABI: a stable C shim exposing selected LibTorch functions, C++ wrappers over that shim, and header-only utilities in torch::headeronly that do not link against libtorch.so.

The scale of the migration, from the post:

Over 175 .cpp and .cu files plus 120 or more headers in the CUDA build.
Seven compiled dependencies that also use LibTorch: FlashMLA, DeepGEMM, FlashAttention-2, FlashAttention-3, DeepSelect, FlashKDA and QuTLASS. Several were migrated on vLLM's own forks first, with upstream pull requests still open for some.
A timeline from an RFC on 2025-10-15 and a first migration PR on 2025-12-29, through in-tree code becoming stable in June 2026, to DeepGEMM on 2026-10-15, which was the last piece.

Two mechanics are worth copying if you ship your own CUDA ops. The TORCH_TARGET_VERSION build flag makes compilation fail if any unstable LibTorch header sneaks in, and ops are registered with STABLE_TORCH_LIBRARY with the implementation wrapped in TORCH_BOX, which boxes and unboxes arguments into StableIValue.

The post is also open about the limits. The stable surface is a subset of LibTorch: there is no stable SymInt, so meta functions had to be Python, and no autograd machinery, so backward functions must be Python too. A method as ordinary as is_pinned() had no stable equivalent during the migration, so vLLM calls aten::is_pinned through the generic torch_call_dispatcher shim. PyTorch added a stable is_pinned() in 2.14.

What does the audit say about the wheel you can download?

The blog shows torch-abi-audit output, but the question I care about is whether the artifact on PyPI matches it. I unpacked vllm-0.33.0-cp38-abi3-manylinux_2_28_x86_64.whl and audited the 13 extension modules inside. I did not audit the aarch64 wheel.

Extension	Stable shim symbols	Min torch
`_C_stable_libtorch`	88	2.11.0
`deep_gemm/_C_extension`	85	2.11.0
`_moe_C_stable_libtorch`	72	2.11.0
`vllm_flash_attn/_vllm_fa2_C`	69	2.10.0
`_flashmla_C`	66	2.11.0
`_flashmla_extension_C`	63	2.11.0
`vllm_flash_attn/_vllm_fa3_C`	63	2.10.0
`_qutlass_C`	63	2.13.0
`_flashkda_C`	43	2.11.0
`_deepselect_C`	40	2.10.0

Three more modules (cumem_allocator, fs_io_C, spinloop) do not touch torch at all. Every torch-linked module reported unstable=0.

The tool reports "the minimum LibTorch version that provides all the required stable symbols", so the wheel-level answer of 2.13.0 is the maximum over the table: one extension, QuTLASS, raises the floor by two minor versions over most of the rest. If you ever wonder why a wheel built on a stable ABI still refuses an older torch, this is the place to look.

The blog's earlier audit, taken before the seven dependencies were migrated, listed six libraries with unstable symbols: 85 (FlashAttention-2), 81 (FlashAttention-3), 70 and 68 (the two FlashMLA modules), and 59 each for QuTLASS and DeepGEMM. That is 422 unstable symbol uses in total, and the shipped wheel has none.

Two gotchas from running the tool myself. It shells out to nm, so on Windows a single .so fails with 'nm' not found. Pointed at the unpacked directory, it printed Torch ABI: NO-TORCH and Extensions: 0, presumably because Windows Python lists only .pyd as extension-module suffixes, so .so files are skipped. That is a false clean bill of health for a perfectly good wheel. I got the real result by running it in WSL Ubuntu, where nm exists.

If the binary is stable, why can't I just pip install a newer torch?

Because the stable ABI fixes binary compatibility and says nothing about what the package declares. The metadata I read from PyPI on 2026-11-08:

vllm==0.33.0 requires torch==2.14.0, torchvision==0.29.0 and flashinfer-python==0.7.2.post1.
torchvision==0.29.0 itself requires torch==2.14.0.
torchvision 0.29.1 (2026-09-30) and 0.29.2 (2026-10-28) relaxed that to torch>=2.14.0.

So the PyTorch side has already moved from exact pins to lower bounds for torchvision, while vLLM's own metadata still carries exact ones. The 0.33.0 pin of 2.14.0 is also one minor version above the audited floor of 2.13.0, and the post does not say why.

The post's demo sidesteps this. It builds vLLM from source (the printed version is 0.1.dev24716+gcb95d994f.d20261105, not a release), runs it on torch 2.14.1+cu132, swaps in torch==2.15.0 and reruns without rebuilding. Both runs print the same text for the prompt "The capital of France is". That is a useful existence proof, with limits worth stating:

The model is SmolLM2-135M-Instruct, max_model_len=128, 8 generated tokens at temperature 0.
It runs with enforce_eager=True, which turns CUDA Graphs off. That is the mode where the post says the shim's overhead can matter, and no timing is reported.
It sets VLLM_USE_FLASHINFER_SAMPLER=0 without explaining why. FlashInfer is a separate PyPI package that vLLM pins, and it is not one of the seven dependencies in the post's migration table, so the audit says nothing about it.

How long did the old model make you wait?

The rebuild tax shows up as a delay between a torch release and the first vLLM release that pins it. I measured it from PyPI upload timestamps, using the torch requirement in each release's metadata.

The script that produced the numbers, so you can rerun it later and see whether the 0.33 line changes the pattern:

import json, re, urllib.request
from datetime import datetime

def pypi(name, version=None):
    url = f"https://pypi.org/pypi/{name}/{version + '/' if version else ''}json"
    with urllib.request.urlopen(url, timeout=60) as r:
        return json.load(r)

def first_upload(files):
    return datetime.fromisoformat(min(f["upload_time"] for f in files))

def pin(info, pkg):
    for req in info.get("requires_dist") or []:
        m = re.match(rf"{pkg}\s*(==|>=)\s*([\w.]+)", req)
        if m:
            return m.group(1) + m.group(2)

torch_dates = {v: first_upload(f) for v, f in pypi("torch")["releases"].items() if f}
vllm = pypi("vllm")["releases"]
seen = {}
for v in sorted((v for v in vllm if re.fullmatch(r"0\.\d+\.\d+", v) and vllm[v]),
                key=lambda s: tuple(map(int, s.split(".")))):
    if tuple(map(int, v.split("."))) < (0, 17, 0):
        continue
    p = pin(pypi("vllm", v)["info"], "torch")
    if p and p not in seen:
        seen[p] = (v, first_upload(vllm[v]))
for p, (v, when) in seen.items():
    t = p.lstrip("=>")
    print(f"vllm {v:7} first pins torch{p:9} | torch released {torch_dates[t]:%Y-%m-%d}"
          f" | vllm shipped {when:%Y-%m-%d} | lag {(when - torch_dates[t]).days} days")

cur = pypi("vllm", "0.33.0")["info"]
print("\nvllm 0.33.0 metadata:", {k: pin(cur, k) for k in ("torch", "torchvision", "flashinfer-python")})
for tv in ("0.29.0", "0.29.1", "0.29.2"):
    print(f"torchvision {tv} requires torch{pin(pypi('torchvision', tv)['info'], 'torch')}")

Output from running it today:

vllm 0.17.0  first pins torch==2.10.0  | torch released 2026-01-21 | vllm shipped 2026-03-07 | lag 44 days
vllm 0.20.0  first pins torch==2.11.0  | torch released 2026-03-23 | vllm shipped 2026-04-27 | lag 34 days
vllm 0.27.0  first pins torch==2.13.0  | torch released 2026-07-08 | vllm shipped 2026-08-10 | lag 33 days
vllm 0.33.0  first pins torch==2.14.0  | torch released 2026-09-02 | vllm shipped 2026-11-02 | lag 60 days

vllm 0.33.0 metadata: {'torch': '==2.14.0', 'torchvision': '==0.29.0', 'flashinfer-python': '==0.7.2.post1'}
torchvision 0.29.0 requires torch==2.14.0
torchvision 0.29.1 requires torch>=2.14.0
torchvision 0.29.2 requires torch>=2.14.0

Three things stand out in that data.

First, the lag was 33 to 60 days, with the longest one being the most recent: torch 2.14.0 came out on 2026-09-02 and vLLM first pinned it on 2026-11-02. The median of the four is 39 days.

Second, torch 2.12.0 (2026-05-13) was never pinned by any release from 0.17.0 onward. vLLM went straight from 2.11.0 to 2.13.0 at 0.27.0.

Third, a pin sticks. The 2.11.0 pin covered 11 releases over 105 days, and the 2.13.0 pin covered 7 releases over 84 days. For an operator, "which torch does the current vLLM release want" has had a stable answer for months at a time, and that is the thing the stable ABI is meant to loosen.

A caveat on interpretation: PyPI pins are a proxy for what the default wheel supported. The main branch and nightlies may have worked with a new torch earlier, and I did not check. torch 2.15.0 went up on 2026-10-28, so it is 11 days old as of this writing, and vLLM 0.33.0 is still on 2.14.0. If the median lag held, a pin would land around December 6. That is a simple extrapolation from four points, not a prediction.

What should an operator do with this today?

My recommendation is to treat the pin as a tested combination and the ABI as headroom, not as a promise. A concrete approach:

Stay on the pinned torch for production until you have a reason not to. The audit tells you symbols resolve. It does not tell you that kernels behave the same on a new CUDA or driver combination, and the post says plainly that Python APIs, the minimum CUDA version and third-party dependencies still constrain compatibility.
If you need a newer torch, do it in staging and expect pip to complain. Installing a newer torch on top of vLLM 0.33.0 should make pip print its incompatible-requirements warning about the exact pins, and a fresh pip install vllm torch==2.15.0 should fail resolution. I have not run this swap. If you use uv, its --override option replaces a dependency's declared requirement, which is the cleaner way to say "I know".
Audit what you actually deploy. Run the check against the installed package, not against the blog post's version of events:

pip install torch-abi-audit            # needs nm (binutils) on PATH
torch-abi-audit --check both vllm      # exit 1 on a violation, 2 on an operational error

Per the tool's --help, --check torch fails on libtorch use without the stable ABI, --check abi3 fails when a torch-using package is not CPython abi3 compliant, and both requires both. Add it to the image build for your own custom ops too.

Smoke test with CUDA Graphs on. The one overhead the post flags (type conversion and dispatcher boxing in the shim) is host-side and is claimed to vanish after warmup under CUDA Graphs. Verify that with your own eager and graph-mode latency, because the post publishes no numbers.
Know which builds are covered. The post says vLLM publishes ABI stable CUDA wheels as of 0.33.0 and lists CPU and ROCm as future work. ROCm shares most sources with CUDA through hipify, with only a few ROCm-specific files left.

FAQ

Can I upgrade PyTorch under an installed vLLM 0.33.0 without rebuilding?
At the binary level the audit says the wheel needs symbols that exist from torch 2.13.0, and the post demonstrates 2.14.1 to 2.15.0 on a source build. At the package level, PyPI metadata pins torch==2.14.0, so you would be overriding the pin. I have not tested the swap on a GPU.

Why is the wheel's minimum torch 2.13.0 when most extensions need 2.10 or 2.11?
The reported floor is the highest requirement of any extension, and _qutlass_C requires 2.13.0. The audit output shows this directly.

Does the stable ABI make vLLM slower?
The post says the shim adds type-conversion overhead, plus operator lookup and boxing for wrappers that go through torch_call_dispatcher. It calls this usually negligible and gone after warmup under CUDA Graphs, but possibly visible in eager mode with very fast kernels, and reports no noticeable regression. No measurements are published.

Does it cover CPU and ROCm builds?
Not yet. The announcement covers the CUDA wheels from 0.33.0. CPU and ROCm are listed as next steps.

Does a stable ABI remove CUDA version constraints?
No. The post states that Python APIs, the minimum supported CUDA version and third-party dependencies still affect compatibility. The ABI only addresses binary compatibility with LibTorch.

How do I check any other wheel for ABI stability?
Install torch-abi-audit, make sure nm is available, and run it on the package or a .so path. The --json flag gives machine-readable output for CI.

Sources

Running an MoE Model Larger Than Your VRAM: SGLang Paged Experts, the PCIe Math, and the 4-Request Cliff

AI Tech News — Sat, 07 Nov 2026 00:01:10 +0000

TL;DR

SGLang 0.5.23 (released 2026-11-02) ships an opt-in feature called Paged Experts. It keeps K of the E experts of every MoE layer in GPU memory, holds all E in pinned host RAM, and copies in only the experts a step routes to that are not resident. The expert math still runs on the GPU.

What the PR measured: on Qwen3-30B-A3B, single-stream decode of 205.6, 87.7 and 32.6 tok/s (int4 on a 16 GB card, FP8 on a 16 GB card, int4 on an 8 GB laptop), against 16.6, 2.9 and 1.2 for --cpu-offload-gb. That is 12x, 30x and 27x.
Against llama.cpp tuned to its best --n-cpu-moe: a tie when most of the model fits or the link is slow (206 vs 194 and 33 vs 33 tok/s), and nearly 2x faster in the FP8 case (87.7 vs 45.2).
What I computed from the PR's numbers: the measured speeds are only reachable if consecutive tokens reuse experts. Uniform random routing would cap the FP8 case near 40 tok/s on a 50 GB/s link.
Where it breaks: with 4 concurrent requests the laptop run fell to 11 tok/s against 41 for llama.cpp. The dividing line is K // top_k, the largest decode batch that fits one wave.
I did not run any of this. The measurements come from one PR author's two consumer machines. The arithmetic on top of them is mine.

What does Paged Experts actually do?

A mixture-of-experts layer routes each token to top_k of E experts. For Qwen3-30B-A3B that is 8 of 128, across 48 layers. I checked the model's config.json on Hugging Face: 48 layers, 128 experts, 8 per token, hidden size 2048, MoE intermediate size 768. The experts are most of the 30B parameters, so a model that is cheap to compute can still be impossible to hold on a small GPU.

There are three ways to serve it when the experts do not fit:

--cpu-offload-gb in SGLang. Offloaded weights are streamed to the GPU on every forward pass, whatever the router picked.
llama.cpp --n-cpu-moe N. The experts of N layers stay in system RAM and are computed on the CPU. No weight traffic, but CPU-speed math.
Paged Experts. A K-slot table per layer lives on the GPU. Each step, the experts it needs that are missing get copied from pinned host memory into slots (least recently used out), then the model's own fused-MoE kernel runs on the GPU as usual.

The third option is the HiCache idea applied to expert weights: the GPU holds the working set, host RAM holds the rest. PR #42557 (merged 2026-10-19) added bf16, block-FP8 and GPTQ experts, and PR #45787 (merged 2026-10-25) added AWQ, compressed-tensors, MXFP4, NVFP4, W8A8 and block-INT8.

Turning it on takes two flags, quoted from the docs:

python -m sglang.launch_server \
  --model-path Qwen/Qwen3-30B-A3B-GPTQ-Int4 \
  --enable-paged-experts \
  --paged-experts-num-resident 92

The constraints matter as much as the flags. The docs list single GPU only (tensor, expert, pipeline and data parallelism are rejected at startup), speculative decoding rejected at startup, and a page-locked host store that has to hold every expert of every layer. Under WSL2, pinned memory is capped at about half of system RAM.

How fast is it, and against what?

The PR benchmarks on two machines: an RTX 4070 Laptop (8 GB, PCIe 4.0 x8, 54 GB RAM, WSL2) and an RTX 5070 Ti desktop (16 GB, PCIe 5.0 x16, 62 GB RAM, native Linux). The workload is ShareGPT prompts with 256 output tokens, prefix caching off in both engines, same KV capacity (10,240 tokens), byte-matched quantization (GPTQ int4 against IQ4_XS, FP8 against Q8_0). llama.cpp's --n-cpu-moe was searched in steps of 1 and only counted when its own memory projection fit physical VRAM. The PR's predecessor compared against --cpu-moe only, which in the author's own words overstated the advantage.

Setup (Qwen3-30B-A3B)	Paged Experts	llama.cpp, best	`--cpu-offload-gb`	vs offload	vs llama.cpp
16 GB, int4, K=92	205.6	194.4	16.6	12.4x	1.06x
16 GB, FP8, K=40	87.7	45.2	2.9	30.2x	1.94x
8 GB laptop, int4, K=26	32.6	32.8	1.2	27.2x	0.99x

Decode tok/s, single stream, from the PR tables. The two ratio columns are my division. Each headline cell was measured at least twice.

The 12x to 30x headline is against the weakest baseline. Against a tuned llama.cpp the result splits in two: when the model nearly fits, or the link is slow, paging and CPU compute land in the same place. When many experts must live in RAM and the link is fast, GPU math wins by a wide margin because llama.cpp's CPU-side layers become the bottleneck.

Why is it fast when experts keep crossing PCIe?

A token at batch 1 activates 48 x 8 = 384 experts. Per the docs an int4 expert here is 2.43 MB; FP8 is about 4.72 MB (3 x 2048 x 768 weights at one byte, scales ignored). If every activation missed, one token would need 0.93 GB (int4) or 1.81 GB (FP8) over the link.

Now compare with the measured decode time. The FP8 run does 87.7 tok/s, so 11.4 ms per token. Copying 1.81 GB in 11.4 ms needs 159 GB/s, more than a PCIe 5.0 x16 link can deliver (about 63 GB/s theoretical). So most activations cannot have been misses.

For a number I need an effective bandwidth. The PR implies one: a prefill pages in about 4 GB in about 80 ms at K=92, and (128 - 92) x 117 MB per slot is 4.2 GB, so roughly 50 GB/s. With that, the highest miss rate each measured speed allows is:

Setup	Link assumed	Miss rate if routing were uniform random (1 - K/E)	Highest miss rate the speed allows
16 GB, FP8, K=40	~50 GB/s	69%	31%
8 GB laptop, int4, K=26	15.75 GB/s (peak)	80%	52%
16 GB, int4, K=92	~50 GB/s	28%	26%

Under uniform random routing the FP8 run would be capped near 40 tok/s, and it measured 87.7. The laptop would be capped near 21 tok/s on a link running at its theoretical peak, and it measured 32.6. So the router must be reusing experts from one token to the next, the premise the docs state. The int4 16 GB row is inconclusive: at 63 GB/s instead of 50 the bound moves to 33%, above the random baseline.

Two caveats. These are upper bounds on the miss rate, not measured miss rates; only a counter inside the engine gives the real value. And the reuse was seen on natural-language prompts. The PR warns that synthetic prompts mis-measure paging, so if your traffic is unusual, test on your own.

What happens with more than one request?

The PR is candid here. At 4 concurrent requests the laptop run produced 11 to 12 tok/s against 41 for llama.cpp, and FP8 on the 16 GB card gave 84 tok/s at K=40 but 31 at K=24 (llama.cpp: 55).

The docs explain the mechanism. A decode step is served by a single wave only if tokens x top_k <= K, which is decode up to K // top_k requests, and that single wave is the case a CUDA graph can capture. Larger batches run without a graph and need several waves, each with its own page-in. Four requests with top-8 routing can touch up to 32 distinct experts per layer, and the table of K values shows the pattern:

Run	K	K // 8	4-request result vs llama.cpp
int4, 16 GB	92	11	387 vs 312 tok/s
FP8, 16 GB	40	5	84 vs 55
FP8, 16 GB	24	3	31 vs 55
int4, 8 GB	26	3	11 vs 41

Both runs where K // 8 is below 4 lose; both where it is 5 or more win. Four points do not make a law, and the PR attributes the loss to steps that route to more experts than fit, the same condition seen from the other side. It does give a usable sizing rule: choose K at least batch x top_k for the batch size you plan to serve, or accept that you are outside the captured fast path. For one user this is irrelevant. For anything shared it decides whether the feature fits.

What does a resident slot cost in context length?

K and the KV cache come out of the same static budget (--mem-fraction-static), and one slot costs per_expert_bytes x num_moe_layers. Here the bf16 KV cache is 2 x 48 layers x 4 KV heads x 128 head dim x 2 bytes = 98,304 bytes (96 KiB) per token.

One int4 slot is 117 MB, about 1,190 tokens of bf16 KV cache (about 2,370 with an FP8 KV cache).
One FP8 slot is about 227 MB, about 2,300 tokens of bf16 KV cache.

Dropping K by 10 on the int4 model buys roughly 11,900 tokens of context at the price of more paging. The PR's whole 10,240-token cache is only about 1.0 GB, so the experts dominate the budget. If the KV cache does not fit at startup, the docs suggest lowering K or using --kv-cache-dtype fp8.

Why do short prompts get slower first tokens?

A prefill touches nearly every expert, so it pages in everything that is not resident, once. For int4 at K=92 that is about 4.2 GB. For FP8 at K=40 it is 88 experts x 227 MB, about 19.9 GB, and on the laptop (K=26) about 11.9 GB over a link that peaks at 15.75 GB/s, at least 750 ms.

TTFT p50	Paged	llama.cpp best
int4 16 GB, short / 2,048 tokens	89 / 267 ms	68 / 344 ms
FP8 16 GB, short / 2,048 tokens	284 / 443 ms	226 / 984 ms
laptop int4, short / 2,048 tokens	928 / 1,441 ms	620 / 2,396 ms

The page-in is a fixed cost, while llama.cpp's CPU-side layers scale with prompt length, so paging loses on short prompts and wins on long ones. The FP8 2,048-token TTFT of 443 ms is within about 45 ms of the 400 ms needed to move 19.9 GB at 50 GB/s, which suggests nearly all of it is page-in (arithmetic, not a profile). For short chat turns, TTFT is the number to watch.

Can paging change the model's outputs?

By design no. Routing is untouched and the paged layer returns the same result as the unpaged one. The docs report bit-identical logprobs for unquantized, FP8 and INT8 experts under deterministic inference, and agreement within Marlin's own batch-to-batch noise for the other formats. On GSM8K (200 questions), gpt-oss-20b MXFP4 scored 0.96 to 0.97 paged and 0.975 unpaged.

The more instructive part is how it can go wrong silently, as described in PR #45787:

An expert tensor that was not listed as pageable stays unpaged, so a paged-in expert runs with another expert's values. In an AWQ control with the zero points unpaged, the median logprob error was 1.67 nats, against 0.003 when paged correctly.
gpt-oss-20b's MXFP4 experts are padded by the Marlin kernel (2880 to 2944 or 3072). Before the fix, paged gpt-oss-20b scored GSM8K 0.000 with every answer truncated. After it, 0.960.
The fix for the first class is structural: the pageable tensors are now derived from the layer, and at startup an independence check re-runs one more chunk of experts and fails if the post-load repack mixed experts.

None of these crash. They produce fluent wrong text, so run an accuracy smoke test against the unpaged model on your own checkpoint format.

A preflight script before you turn it on

This is the arithmetic above in a few lines: slot cost, batch cap, and whether a measured speed implies expert reuse.

def preflight(name, layers, experts, top_k, expert_mb, K, link_gbps, tok_s, kv_bytes_per_token):
    slot_mb = expert_mb * layers                    # GPU memory per slot, all layers
    batch_cap = K // top_k                          # largest single-wave decode batch
    all_miss_gb = layers * top_k * expert_mb / 1e3  # GB per token if every activation missed
    max_miss = link_gbps / (tok_s * all_miss_gb)    # highest miss rate the speed allows
    random_miss = 1 - K / experts                   # miss rate with no reuse
    verdict = "reuse implied" if max_miss < 0.8 * random_miss else "inconclusive"
    print(f"{name}: slot={slot_mb:.0f} MB (~{slot_mb*1e6/kv_bytes_per_token:,.0f} KV tokens), "
          f"batch cap={batch_cap}, random-routing miss={random_miss:.0%}, "
          f"max miss allowed by {tok_s} tok/s={max_miss:.0%}, {verdict}")

KV = 2 * 48 * 4 * 128 * 2   # Qwen3-30B-A3B bf16 KV bytes per token
preflight("int4 16GB", 48, 128, 8, 2.43, 92, 50.0, 205.6, KV)
preflight("FP8  16GB", 48, 128, 8, 4.72, 40, 50.0,  87.7, KV)
preflight("int4  8GB", 48, 128, 8, 2.43, 26, 15.75, 32.6, KV)

Output from running it:

int4 16GB: slot=117 MB (~1,187 KV tokens), batch cap=11, random-routing miss=28%, max miss allowed by 205.6 tok/s=26%, inconclusive
FP8  16GB: slot=227 MB (~2,305 KV tokens), batch cap=5, random-routing miss=69%, max miss allowed by 87.7 tok/s=31%, reuse implied
int4  8GB: slot=117 MB (~1,187 KV tokens), batch cap=3, random-routing miss=80%, max miss allowed by 32.6 tok/s=52%, reuse implied

Use a link speed measured with a pinned-memory copy test, not the spec sheet. The 50 GB/s here is inferred from the PR.

A rollout checklist

Confirm your format is in the supported table and the MoE runner is Triton or Marlin. Anything else is rejected at startup, which is the good failure.
Check that the host can pin the whole expert store (half of RAM under WSL2).
Choose K from the docs' formula, then check K // top_k against your real concurrency.
Measure host-to-device bandwidth on your machine, and benchmark your own prompts, because the PR warns that synthetic ones mis-measure paging.
Run an accuracy smoke test against the unpaged model, and record TTFT for short and long prompts separately.

FAQ

Does it work across several GPUs or with speculative decoding?
No to both in 0.5.23. The docs say single GPU only, with tensor, expert, pipeline and data parallelism rejected at startup, and speculative decoding is rejected too.

Does it change accuracy compared with an unpaged server?
The docs report bit-identical logprobs for unquantized, FP8 and INT8 experts under deterministic inference, and agreement within Marlin's noise for the other formats. Past bugs in this area failed silently, so run a smoke test on your format.

How do I choose K?
Use the docs' budget formula, then keep K at least batch x top_k for your target concurrency. Each extra slot costs per_expert_bytes x layers of memory that would otherwise be KV cache.

Is it faster than llama.cpp's --n-cpu-moe?
It depends. In the PR, single-stream decode tied in two of three setups and was 1.94x faster in the third, while llama.cpp had lower TTFT on short prompts and better 4-request throughput on the laptop.

How much host RAM do I need?
Enough to pin every expert: about 14.9 GB at int4 and 29 GB at FP8 for Qwen3-30B-A3B, by my arithmetic on the docs' per-expert size.

Sources

vLLM 0.33 Adds a Lossy Speculative Verifier: Why Single-Run Accuracy Tables Cannot Tell You It Is Safe

AI Tech News — Thu, 05 Nov 2026 14:05:42 +0000

TL;DR

What shipped: vLLM 0.33.0 (PyPI, 2026-11-02) includes FLy, an opt-in speculative decoding verifier that deliberately accepts some draft tokens the target model would have rejected. The PR states it "does not preserve the target distribution and may reduce output quality." Standard verification stays the default.
What the PR reports: decode throughput 1.041x to 1.310x over standard speculative decoding at all 48 model, platform and batch points, with minimum accuracy retention of 96.3% against target-only decoding.
What I computed from the PR's own tables (not a new benchmark): across the 32 accuracy cells, standard speculative decoding, which is lossless by design, lands a mean 1.68 points away from target-only. FLy lands 1.80 points away. The tables measure run-to-run noise at least as much as they measure FLy.
What a real quality gate needs: a paired per-question comparison. With 6% of questions flipping either way, detecting a 2-point net loss at 80% power takes about 1,175 questions. The PR's evaluation had 428 per model and platform.
The bigger win in the same release is an accounting fix: a scheduler change for DSpark and DFlash drafters took one reported run from 1,373 to 3,390 output tokens/s, and no kernel changed.

What does FLy change in speculative verification?

In standard speculative decoding, a small drafter proposes K tokens and the target model verifies them in one forward pass. Verification stops at the first rejected token, and the accept/reject rule is built so the output distribution matches what the target would have produced alone.

FLy, from the paper "Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match" (arXiv 2511.22972), relaxes the stop. Per PR #53987 (merged 2026-10-19), a rejected position is forgiven when three conditions hold:

The rejected draft token is not masked out by target constraints.
The target's entropy at that position meets a threshold, meaning the target itself was unsure.
The next W draft tokens all pass the normal verification rule.

If any of the next W tokens is rejected, the original rejection stands. A short trailing window is never deferred. The PR notes the entropy is computed over only the top-k target probabilities (default 3) without renormalizing, which differs from the paper's full-vocabulary version. It needs no extra checkpoint and no extra target forward pass.

Enabling it is one block in the speculative config:

from vllm import LLM

llm = LLM(
    model="Qwen/Qwen3-8B",
    speculative_config={
        "method": "draft_model",
        "model": "Qwen/Qwen3-0.6B",
        "num_speculative_tokens": 8,
        "rejection_sample_method": "fly",   # default is standard verification
        "fly_window_size": 6,
        "fly_entropy_threshold": 0.3,
        "fly_entropy_top_k": 3,
    },
)

Two details matter for reproducing the published numbers. fly_window_size defaults to min(6, num_speculative_tokens - 1), and FLy needs at least two speculative tokens. The PR's ModelRunner V2 measurements, though, were run with window=2. A DeepSeek setup with K=7 would default to a window of 6, so the default is not the configuration that was benchmarked. The engine also logs a one-time warning that the method is lossy.

What do the published speed numbers show?

The PR's multi-batch table covers four models: DeepSeek-V4-Flash with DSpark (K=7), Qwen3.8-27B with MTP (K=3), GLM-5.2-FP8 with MTP (K=5) and Qwen3.5-122B-A10B-FP8 with MTP (K=8). They ran on AMD MI355X and NVIDIA B300 at batch sizes 1, 4, 8, 16, 32 and 64, with 1,024 generated tokens per request and a fixed pool of 64 prompts.

I parsed all 48 rows. The speedup range matches the PR text (1.041x to 1.310x), and the median is 1.147x. Averaged over the six batch sizes:

Model (method, K)	Platform	Mean speedup	Mean draft acceptance gain
Qwen3.8-27B (MTP, 3)	B300	1.065x	+7.6 pp
Qwen3.8-27B (MTP, 3)	MI355X	1.083x	+10.3 pp
DeepSeek-V4-Flash (DSpark, 7)	MI355X	1.137x	+9.4 pp
DeepSeek-V4-Flash (DSpark, 7)	B300	1.151x	+8.6 pp
GLM-5.2-FP8 (MTP, 5)	B300	1.156x	+11.2 pp
GLM-5.2-FP8 (MTP, 5)	MI355X	1.170x	+12.6 pp
Qwen3.5-122B-A10B (MTP, 8)	B300	1.242x	+16.4 pp
Qwen3.5-122B-A10B (MTP, 8)	MI355X	1.281x	+16.9 pp

The speedup tracks the acceptance gain closely: the correlation across the 48 points is r = 0.82. That is what you would expect if FLy converts "extra accepted tokens" into "fewer target passes" with little other overhead. It also tells you where the method pays. A model whose drafter already agrees with it (Qwen3.8 at K=3, only +7.6 to +10 points) gains 6 to 8%. A model with a weaker drafter (Qwen3.5-122B, +16 to +17 points) gains 24 to 28%. A flat 1.2x is not a safe planning number for your own pair.

Can the accuracy tables show that FLy is safe?

Not by themselves, and the PR does not claim they can. It says the retention figures "are single-pass measurements, not a quality guarantee." The question is how much they can say, so I rebuilt the accuracy table.

Each model and platform was scored on GSM8K (first 100 questions), MATH-500 (first 100), AIME25 (30) and GPQA-Diamond (198), which is 428 questions, in three decoding modes: target-only, Standard SD and FLy. That is 2 platforms x 4 models x 4 datasets = 32 cells. Standard SD is distribution-preserving by construction, so any gap between it and target-only is noise from the run itself (batched floating-point differences, sampling, a single pass). Here is the mean absolute gap to target-only in each cell, grouped by dataset:

	Standard SD (lossless)	FLy (lossy)
Mean absolute gap to target-only, 32 cells	1.68 pp	1.80 pp
Largest single-cell gap	6.67 pp	10.00 pp
Cells where its gap is larger than the other method's	11	12 (9 tied)
Cells below / above / equal to target	8 / 12 / 12	13 / 11 / 8

The largest lossless gap is DeepSeek-V4-Flash on B300 for AIME25: target 26/30, Standard SD 28/30. The largest FLy gap is Qwen3.5-122B on MI355X for GSM8K: target 82, FLy 92 out of 100, where the lossy method scored 10 points better. Neither number is evidence about the method. They show that with 30 to 198 questions per cell, one cell can move by several points for reasons unrelated to the verifier.

Pooling helps a little. Summed over all 8 model and platform pairs (3,424 scored answers per decoding mode):

Mode	Correct	Accuracy
Target-only	3,040	88.79%
Standard SD	3,067	89.57%
FLy	3,059	89.34%

FLy is 19 answers above target-only, and the lossless method is 27 above it. A lossless method cannot genuinely beat the target on average, so the 0.8-point spread is the noise scale of this setup. A rough 95% interval for the difference between two unpaired accuracies near 90% is about plus or minus 4.0 points at 428 questions, 5.9 at 198, 8.3 at 100 and 15 at 30 (normal approximation, poor at n=30 near the ceiling). A real 1 to 2 point quality loss would sit inside all of those bands.

None of this says FLy hurts quality. The evidence is consistent with a small loss and with none, and the PR's minimum retention (96.43% on AMD, 96.30% on NVIDIA) is a one-problem difference on AIME25, where each question is 3.3 points.

How many questions does a real quality gate need?

The fix is a paired design. Run both configurations on the same questions with the same seeds, record per-question correctness, and count only the discordant pairs: questions FLy lost and questions it gained. McNemar's exact test then asks whether the net movement is bigger than chance. Pairing removes the between-question variance that makes the unpaired interval so wide.

import math

def paired_gate(base, cand):
    """base, cand: per-question 0/1 correctness for the SAME questions and seeds."""
    lost = sum(1 for x, y in zip(base, cand) if x and not y)
    gained = sum(1 for x, y in zip(base, cand) if y and not x)
    n, k = lost + gained, min(lost, gained)
    p = min(1.0, 2 * sum(math.comb(n, i) for i in range(k + 1)) / 2 ** n) if n else 1.0
    return {"lost": lost, "gained": gained, "net": gained - lost, "exact_p": round(p, 4)}

def questions_needed(net_loss, discordant, z_a=1.96, z_b=0.8416):
    # two-sided alpha 0.05, power 0.80; discordant = fraction of questions that flip either way
    return math.ceil((z_a * math.sqrt(discordant)
                      + z_b * math.sqrt(discordant - net_loss ** 2)) ** 2 / net_loss ** 2)

I ran this on synthetic data (illustrative: 428 questions at 89% base accuracy, with 5% of answers randomly flipped). It reported 25 lost, 2 gained, exact p below 0.0001. A deliberately damaged candidate is caught easily. The harder question is the subtle loss, and the sample size you need depends on how often the two runs disagree at all:

Net loss to detect	Discordant 4%	Discordant 8%
3 points	347 questions	696 questions
2 points	783 questions	1,568 questions
1 point	3,138 questions	6,277 questions

(Normal-approximation sample sizes from the formula above; the discordant fraction is an assumption you measure by running Standard SD twice on a pilot set.) At a 6% flip rate, a 2-point loss needs about 1,175 questions and a 1-point loss about 4,700. A 428-question suite per cell cannot resolve either, which is why a short eval is a smoke test and not a gate. Build the gate from your own task mix, because the entropy rule fires more often on open-ended generation than on short arithmetic answers, and your traffic is neither GSM8K nor GPQA.

What is the bigger win in the same release?

PR #59468 (merged 2026-10-15) fixes how the scheduler charges token budget for block-diffusion drafters. DFlash and DSpark were reserving draft slots in each request's share of max_num_batched_tokens even though they run their draft pass separately, not inside the target batch. For DSpark with K=5, a decode request was charged 10 tokens instead of 6. The PR sets the reserved slots to 0 and instead caps running requests at max_num_batched_tokens // num_lookahead_tokens, so the separate draft buffers still fit.

The reported end-to-end result (DeepSeek-V4.1-Flash with DSpark, TP4 on one GB300, --max-num-batched-tokens 512, one run per side):

	main	PR #59468
Max running requests	52	64
Output tok/s	1,373	3,390
TPOT p50 / p99 (ms)	29.2 / 63.5	15.7 / 36.0
TTFT p50 (s)	93.6	30.0
Acceptance length	4.72	4.73

Admission rose 1.23x and throughput 2.47x. The PR's explanation for the gap is scheduling, not drafting: on main, the 2 to 32 leftover tokens split each new prefill into small chunks, so most steps mixed prefill into the decode batch and missed the full decode CUDA graph (ITL p50 169 ms on main against 30 ms with the fix). Acceptance length is unchanged at 4.72 versus 4.73, and greedy outputs for 20 fixed prompts were byte-identical (3,197 completion tokens). The author reports that an earlier revision, measured with two interleaved runs per side, was within 2% of these numbers. DFlash was not run end to end.

The admission arithmetic is easy to check:

def max_running(budget, max_seqs, tokens_per_decode_request):
    return min(max_seqs, budget // tokens_per_decode_request)

for label, charge in (("before, K=5", 10), ("after, K=5", 6)):
    print(label, max_running(512, 64, charge), max_running(512, 128, charge))
# before, K=5 51 51
# after, K=5 64 85

Simple division gives 51 for the old charge where the PR observed 52, a remainder effect I did not trace. For the new charge with --max-num-seqs 128 it gives 85, which matches the PR exactly, and the new request cap of 102 (512 // 5) does not bind. You can see the cap in effect at startup: the log line reads like "The dspark draft pass needs 5 query tokens per request within max_num_batched_tokens=512. Overriding max_num_active_seqs from 128 to 102."

The operational lesson generalizes beyond drafters. If you run speculative decoding and your running-request count plateaus well below --max-num-seqs under saturating load, token-budget accounting is binding and the number you set is not the number you get:

curl -s localhost:8000/metrics | grep '^vllm:num_requests_running'

What else in 0.33 touches a speculative serving stack?

Medusa and the MLP speculator were removed (#61701), lookahead block hashes for EAGLE and MTP prefix-cache hits are now on by default (#61589), CUDA 12.9 images are no longer published, and the NIXL connector needs NIXL 1.5 or newer. Read those before upgrading a fleet that uses drafters.

A rollout checklist for FLy

Leave standard verification on wherever outputs feed an unreviewed downstream decision. That is my recommendation, not the PR's, and the reason is the lack of a distribution guarantee.
Reproduce the configuration that was measured, including fly_window_size, or measure your own window.
Build a paired eval from your traffic, with at least 1,000 questions to resolve a 2-point loss, fixed seeds, and per-question results stored so you can run the exact test.
Track acceptance, not only tokens/s. The speedup follows the acceptance gain, so a workload where the draft already agrees will show little benefit.
Alert on the one-time lossy warning in logs so a config drift does not turn the method on silently.

I did not run FLy myself. Every number above comes from the PR authors' tables or from arithmetic on them, and the throughput results are their measurements on their hardware.

FAQ

Is FLy on by default in vLLM 0.33?
No. Standard verification is the default, and FLy requires rejection_sample_method set to "fly" in the speculative config, with at least two speculative tokens.

Does FLy preserve the target model's output distribution?
No. The PR says it does not and that it may reduce output quality. Standard speculative decoding does preserve the distribution.

How much faster is it?
The PR reports 1.041x to 1.310x over standard speculative decoding across 48 model, platform and batch combinations, with a median of 1.147x in my parse of its table. The gain follows the draft acceptance gain, so it varies by model pair.

Do the published accuracy results show no quality loss?
They show no loss that a single run on 30 to 198 questions per dataset could detect. In the same tables, the lossless method differs from target-only by a mean 1.68 points, close to FLy's 1.80, so the noise is of similar size to any plausible effect.

How many questions do I need to detect a 2-point accuracy loss?
About 1,175 paired questions at 80% power and 6% of questions flipping either way, rising to about 1,570 at 8%. Use a paired exact test on per-question results.

Does the scheduler fix need a config change?
No. For DFlash and DSpark the scheduler lowers the active-request cap at startup when needed and logs it, and it does not alter --max-num-seqs itself.

Sources

vLLM v0.33.0 release notes: https://github.com/vllm-project/vllm/releases
FLy pull request #53987: https://github.com/vllm-project/vllm/pull/53987
Scheduler budget pull request #59468: https://github.com/vllm-project/vllm/pull/59468
Paper, "Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match": https://arxiv.org/abs/2511.22972

vLLM Fast Start Explained: What Preload and Snapshots Save, What They Cannot, and the Fingerprint Gap

AI Tech News — Tue, 03 Nov 2026 14:12:04 +0000

TL;DR

What shipped: vLLM 0.31.0 (2026-10-05) added vllm preload, a per-GPU daemon that keeps post-quantized weights resident so a restarting engine maps them over CUDA IPC instead of reloading from disk, plus experimental CRIU-based initialized-engine snapshots. The design goal, in the original PR's words, is that "engine restarts are dominated by weight loading from disk."
What it does not do: the daemon is node-local and must itself load from disk once, so it speeds up restarts on a GPU that already holds the weights. It does not speed up scaling out to a fresh node.
What a faster start is worth (illustrative simulation): with a 1.5x traffic burst and a 10 s queue-wait SLO, cutting replica startup from 300 s to 45 s raises the safe baseline utilization from 69% to 83%. For 100 GPUs of baseline load that is 145 GPUs versus 120. At 2x bursts it is 196 versus 159.
A correctness gap to know about: as of 2026-11-03, issue #59647 is still open. The weight-cache fingerprint hashes safetensors headers, not weight values, so a daemon holding checkpoint A can serve an engine started on checkpoint B without any warning. The proposed fix samples only 1 KiB per tensor, which on Qwen3-8B is 0.002% of the file.

What did vLLM actually ship for fast restarts?

The feature landed in two pieces. The first is the weight cache daemon, merged as PR #54921 on 2026-09-04 and exposed as vllm preload in v0.31.0. One daemon process per GPU loads its tensor-parallel shard, runs quantization post-processing, and exports every parameter as a CUDA IPC handle. An engine started with --load-format ipc_cache builds the model on the meta device and asks the daemon for the tensors. The second piece is vllm snapshot create/restore, which uses CRIU to freeze a fully initialized engine and bring it back before the HTTP server binds.

# one daemon per GPU, loads from disk once
vllm preload --model meta-llama/Llama-3.1-8B-Instruct --tensor-parallel-size 4

# engine restarts map the resident weights instead of reading disk
vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --tensor-parallel-size 4 --load-format ipc_cache

The two mechanisms keep very different amounts of state, and their constraints differ accordingly. Everything in this table comes from the vLLM docs pages for the two features as of today.

	Weight preload	Initialized snapshot
What persists	Post-quantized weights in GPU memory (and FlashInfer autotune results on disk)	The whole engine process tree and CUDA state
Scope	Same node, same user, Unix socket	Same host, GPU, driver, kernel, vLLM version, engine args
Parallelism	Tensor, expert and data parallel, including multi-node (one launcher per node)	TP1, one NVIDIA GPU only
Validated	PR test results include TP2 and MTP draft models	Only dense float16 TP1; acceptance run was Qwen3-0.6B on an H200
Cost while idle	The daemon pins the weights in GPU memory	Disk: artifact size can approach process plus GPU memory
Fallback	Disk load if daemon missing or fingerprint mismatched	None: restore fails closed before CRIU runs

One number from outside vLLM gives a sense of scale for the weight-load part alone. The README of a separate PyPI plugin built on the same idea, vllm-ipc-cache 0.2.0, reports a 27B model at tensor parallel 4 on a MetaX N300-A dropping from 35.80 s to 0.47 s. It is one vendor-reported run on non-NVIDIA hardware and covers the weight-attach step only, not an end-to-end restart.

Which startup events does a faster restart actually help?

Preload covers only some of the reasons you start an engine.

Rolling upgrades and config changes. The daemon outlives the engine, so you restart the engine and keep the weights. This is the cleanest win.
Engine crashes where the GPU is healthy. The daemon is a separate process, so a crashed engine can remap the same weights. If the fault is at the GPU level, the daemon probably goes down with it and you are back to a disk load. I did not test that case. The docs say that if any daemon rank exits, you should restart the whole preload group before attaching engines.
Scale-out to a new node. Not helped. CUDA IPC handles are node-local, the docs state that each node serves only its own GPUs, and the daemon itself loads from disk. The one scale-out feature is warmup-cache seeding: a new daemon can fetch rank-matched FlashInfer autotune tables from a source replica over an authenticated, unencrypted TCP channel.
Time-sharing a GPU between models. This is what snapshots are for: "repeatedly activating the same model and engine configuration on the same machine."

The autotune detail matters more than it looks. The docs say that once weights come from the daemon, the FlashInfer autotune pass "dominates what is left of the startup time" on a large MoE model, which is why vllm preload now runs it at preload time. v0.31.0 also parallelized Triton warmup compilation (#58582). So the honest accounting of a restart is weights, then autotune, then warmup and CUDA graph work, and each has its own fix.

Here is the arithmetic that makes rolling upgrades the main prize. A rolling restart of 16 replicas, one at a time, runs the fleet at 15/16 capacity (93.75%) for 16 times the per-replica restart time.

At 300 s per replica that is 80 minutes of reduced capacity; at 45 s it is 12 minutes; at 10 s it is 2.7 minutes. If you ship engine upgrades every couple of weeks, that is the number to compare against the cost of keeping a daemon pinned.

How much utilization does a faster start buy you?

Startup time matters to capacity planning through one mechanism: during a burst, demand exceeds capacity for roughly the detection lag plus the startup time, and the backlog built in that window has to drain inside your queue-wait SLO. If the burst multiplies load by B, baseline utilization is u, the window is W seconds and the SLO is S seconds of queue wait, the backlog stays inside the SLO when:

(B * u - 1) * W <= S      so      u_max ~= (1 + S / W) / B,   W ~= startup + detection lag

I checked this with a fluid-queue simulation (illustrative, not a measurement of any real fleet): 16 replicas, a 30-minute run, demand ramping by a factor B over 30 s, a scaler that averages demand over 30 s, checks every 10 s and restores the baseline utilization, and a violation counted for every second the queue wait exceeds 10 s. The core loop:

for t in range(T_END):
    lam = demand(t, lam0, burst)
    n += sum(c for r, c in pending if r <= t)            # replicas that finished starting
    pending = [(r, c) for r, c in pending if r > t]
    if t % 10 == 0 and t >= 30:
        want = ceil(mean(hist[-30:]) / (u0 * MU))        # restore baseline utilization
        if want > ordered:
            pending.append((t + startup, want - ordered)); ordered = want
    q = max(0.0, q + lam - n * MU)                       # fluid backlog
    violations += (q / (n * MU)) > 10.0

For each startup time I searched for the highest baseline utilization with zero violation seconds:

Replica startup	Max safe baseline utilization, burst x1.5	GPUs for 100 GPUs of load	Max safe utilization, burst x2.0	GPUs for 100 GPUs of load
300 s	0.69	145	0.51	196
120 s	0.73	137	0.54	185
45 s	0.83	120	0.63	159
10 s	0.99 (search cap)	101	0.85	118

The closed form with W set to startup plus 45 s predicts 0.686 for the 300 s row against 0.69 in the simulation. It is conservative for fast starts, because the ramp is finite and the scaler orders in steps; at 45 s it predicts 0.74 against 0.83. The 10 s row is the idealized limit, since it ignores KV cache warmup, load balancer registration and anything else that happens after the process is up.

The same data as violation seconds shows how sharply the cliff moves. At 80% baseline utilization and a 1.5x burst, a 300 s startup spends 413 s over the SLO with a peak queue wait of 57 s. At 120 s it is 113 s over, and at 45 s it is zero violations with a peak wait of 7.6 s.

Two readings follow. First, going from 300 s to 45 s buys about 17% to 19% fewer GPUs in these scenarios, which is real money but not a revolution. Second, those gains apply to scale-out, and preload does not accelerate scale-out to fresh nodes. Where this model really applies is a parked warm replica on the same host (a daemon on an idle GPU, or a snapshot on disk), and that parked GPU costs about the same as the headroom replica you were trying to avoid. For most fleets the dependable return is the rolling-restart arithmetic above, not the autoscaling story.

Can the weight cache serve the wrong checkpoint?

It can today, in a specific case, and the project knows. The fingerprint that decides whether a daemon's weights match the engine covers architecture, TP and DP layout, dtype, quantization, RoPE parameters, KV cache dtype and vLLM version. For the checkpoint itself it hashes each safetensors shard's header: tensor names, dtypes, shapes and byte offsets. A header contains no weight values. Issue #59647, filed 2026-10-01 and still open on 2026-11-03, reports that a base model and its fine-tune therefore get the same key. If a daemon preloaded from A is still running and an engine starts on B with ipc_cache, the daemon serves A's tensors, the engine never reads B from disk, and nothing is logged. A write-up on dev.to by a separate author reproduced it on 0.30.0 by zeroing one norm tensor while leaving the header intact. The docs promise a safe fallback on mismatch, which is why this surprises people.

The open fix, PR #59648, adds file size plus the first 1 KiB of every tensor to the hash. I re-implemented both schemes (the logic is small) and ran them on eight 1024x1024 float16 tensors, then changed the values in different ways:

Change to the checkpoint	Header-only hash detects it	Sampled 1 KiB hash detects it
Noise added to every tensor (a dense fine-tune)	no	yes
One whole tensor shifted by a constant	no	yes
First row of one tensor zeroed	no	yes
Row 500 of one tensor zeroed	no	no
One element at row 900 changed	no	no
Byte-identical copy in another directory	same key (wanted)	same key (wanted)

The proposal catches realistic fine-tunes, because training changes nearly every value, including each tensor's first kilobyte. It cannot catch a surgical edit that leaves the head of each tensor alone. The reason is coverage: on the real headers I fetched, Qwen3-0.6B has 311 tensors and 1.50 GB, so 1 KiB per tensor is 0.018% of the file; Qwen3-8B is 0.0022%; Qwen3-32B has 707 tensors over 65.5 GB and the sample is 0.00095%.

That is a fine tradeoff for a startup-path check, since reading everything would defeat the purpose. The practical conclusion is not to rely on the fingerprint for identity. Tie the daemon's lifetime to a digest you computed once, at promotion time, with the full file bytes. SHA-256 ran at about 1.8 GB/s on my PC (median of five runs on an in-memory buffer), so a 65.5 GB checkpoint is roughly 36 s of CPU time if the disk can feed it, paid once in CI and not on every restart.

# at promotion time, once per checkpoint
DIGEST=$(sha256sum /models/qwen3-8b/*.safetensors | sha256sum | cut -c1-16)
SOCK=/run/user/$UID/vllm-wc/$DIGEST
mkdir -p -m 700 "$SOCK"

vllm preload --model /models/qwen3-8b --weight-cache-socket-dir "$SOCK"
vllm serve /models/qwen3-8b --load-format ipc_cache \
    --model-loader-extra-config "{\"socket_dir\": \"$SOCK\"}"

An engine pointed at a different checkpoint computes a different digest, looks in a different directory, finds no socket, and falls back to disk, which is the documented behavior for an unavailable daemon. I used only flags from the preload docs and did not run this against a GPU engine for this post. Add a temperature-0 canary request per checkpoint after every start; the snapshot feature does exactly that internally, comparing a recorded token and log probability before it returns.

A pre-flight checklist for adopting fast start

Count your restart events by type for a month: upgrades, crashes, scale-outs. Preload only pays on the first two.
Measure your own restart phases separately: weight load, autotune, warmup and CUDA graphs. Do not assume weights are the whole story.
Key the daemon to a full-file digest, and run a canary after each start.
Keep the disk fallback enabled in production and alert on it. Budget the GPU memory the daemon pins, and do not combine the loader with sleep mode.

FAQ

What does vllm preload do?
It starts one daemon per GPU that loads and post-processes the weights once and keeps them in GPU memory. Engines started with --load-format ipc_cache map those weights over CUDA IPC instead of reading the checkpoint again.

Does preload speed up autoscaling to a new GPU node?
No. The handles are node-local and the daemon must load from disk the first time. It helps restarts on a node that already holds the weights. Only the FlashInfer autotune tables can be seeded to a new daemon from another replica.

Can the weight cache load the wrong model?
Under the current fingerprint, yes, if two checkpoints share names, shapes, dtypes and offsets and a daemon for the other one is still running (issue #59647, open on 2026-11-03). A digest-keyed socket directory and a canary request avoid it.

How much GPU capacity does faster startup save?
In my illustrative simulation with 1.5x bursts and a 10 s queue-wait SLO, 45 s startup needs 120 GPUs where 300 s needs 145, for 100 GPUs of baseline load. Your own burst size, detection lag and SLO will move that.

When Should an LLM Router Stop Being Sticky? llm-d 1.0 Cache-Affinity Routing, Break-Even Math, and a Simulation

AI Tech News — Mon, 02 Nov 2026 14:05:55 +0000

TL;DR

What shipped: llm-d 1.0 was announced on 2026-10-20 under CNCF governance, with vLLM and SGLang as supported model servers. The feature that most affects your GPU bill is not the headline list. It is the default routing rule described in llm-d's August post "Sticky Until Saturated": send a request to the endpoint that already holds its prefix, until that endpoint's queued token load passes a limit, then route by load alone.
The limit is arithmetic, not taste: tau = R_peak x T_max. With the published Qwen3-32B numbers (8,192 tokens in 0.40 s, so 20,480 tokens/s, and a 14 s tolerance) that gives 286,720 tokens, exactly 35 scheduler chunks.
A break-even you can compute: for one request, sticking beats spilling while the warm endpoint's queue exceeds the cold one's by no more than the tokens the cache saves. For a 55,000-token prefix that is 2.69 s of queue. llm-d's limit is 5.2 times larger on purpose, and my simulation shows what happens if you set it at break-even instead: the fleet collapses.
Reported results (vendor benchmark): 2.9x and 2.0x the throughput of Kubernetes Service round-robin on two prefill-bound workloads, parity on a decode-bound one.

What did llm-d 1.0 actually ship?

The announcement lists a lot: prefix-cache and load-aware routing, KV cache offloading, prefill/decode disaggregation, wide expert parallelism, batch and async inference, multi-tenant quality of service, router high availability with hot standby, and session-aware routing for agents. The founding contributors named are Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA, with more than 500 contributors from over 25 organizations. The GitHub release notes add a few operational details: a DisaggregatedSet that manages prefill and decode replicas as one logical unit, a router configuration API that graduates to v1, and deprecation of the llm-d-rocm container image builds in favor of upstream vLLM.

Most of that is plumbing you will only appreciate during an incident. The routing default is different, because it decides how often your GPUs recompute a prompt prefix they have already seen. If you serve long shared system prompts (coding agents, RAG with a fixed preamble, multi-turn chat), prefix reuse is the largest single lever on cost per token, and the router is the component that either preserves it or throws it away.

IBM's announcement also quotes figures such as 11.5 times lower inference cost than an all-proprietary configuration at roughly a 98% prefix-cache hit rate. Those are the project backers' numbers with workload details I could not check, so I treat them as a statement of how much headroom a high hit rate can give, not as something to plan capacity around.

Why does cache-affinity routing hot-spot?

Round-robin ignores caches. A fleet of 10 endpoints, each able to hold only a few long prefixes, ends up with every endpoint seeing every prefix, evicting constantly, and recomputing most of what it receives. Affinity routing fixes that by sending a request to an endpoint that already has its prefix.

The failure mode is the other direction. In llm-d's own words, affinity concentrates traffic on cache-warm endpoints, and without an explicit saturation release the warm endpoint keeps absorbing load past the point where a cold endpoint would serve the request faster. Real traffic is skewed. A few popular prefixes carry a large share of requests, so the endpoint that owns the most popular prefix becomes a queue.

The previous llm-d default tried to handle this with a four-signal weighted blend: a prefix-cache scorer at weight 3, plus queue, KV-cache-utilization and no-hit-LRU scorers at weight 2 each. The authors describe the behavior as hard to predict and harder to tune. That is a fair description of any weighted sum where one signal is in tokens, one is a count and one is a ratio: the weights encode an exchange rate between units nobody measured.

How does llm-d decide when to stop being sticky?

The replacement splits the decision in two. A filter keeps only endpoints whose cached prefix match is at or above an affinityThreshold of 0.8. Then one load scorer, matched to the workload's bottleneck, picks among them: token-load-scorer (uncached prefill tokens in flight) for prefill-bound traffic, active-request-scorer (active request count) for decode-bound traffic. When the best warm endpoint's load passes the saturation limit tau, the filter stops restricting and the request goes wherever load is lowest.

The limit comes from two things you can measure:

tau = R_peak x T_max
R_peak = peak prefill throughput per endpoint (tokens/s)
T_max  = how much TTFT degradation you will tolerate

For the benchmark setup (Qwen3-32B, tensor parallel 2 across two H100s per server, vLLM with max-num-batched-tokens=8192), the post reports T(B = 8192) = 0.40 s. I checked the rest of the chain: 8,192 / 0.40 = 20,480 tokens/s; 20,480 x 14 s = 286,720; and 286,720 / 8,192 = 35.0 chunks. The numbers are internally consistent.

Here is the same calculation as code, with the break-even and the KV sizing I use below. The KV geometry comes from the model's config.json on Hugging Face (64 layers, 8 KV heads, head dimension 128, bf16):

def saturation_tau(batch_tokens, batch_seconds, ttft_tolerance_s):
    r_peak = batch_tokens / batch_seconds          # prefill tokens/s per endpoint
    return r_peak, int(r_peak * ttft_tolerance_s)

def kv_bytes_per_token(layers, kv_heads, head_dim, dtype_bytes=2):
    return 2 * layers * kv_heads * head_dim * dtype_bytes   # K and V

r, tau = saturation_tau(8192, 0.40, 14)
print(r, tau, tau / 8192)                       # 20480.0 286720 35.0
per_tok = kv_bytes_per_token(64, 8, 128)
print(per_tok, 55000 * per_tok / 2**30)         # 262144 bytes, 13.43 GiB per 55k prefix
print(int(64 * 2**30 // (55000 * per_tok)))     # 4 prefixes fit in a 64 GiB budget

That last line matters. At 256 KiB per token, one 55,000-token prefix is 13.4 GiB of KV. A server with a 64 GiB KV budget (my assumption, not a figure from the post) holds four of them. A coding-agent fleet with dozens of distinct long prefixes cannot cache them all on every endpoint, which is exactly why routing decides the hit rate.

What throughput did the benchmark report?

The post compares matched configurations against Kubernetes Service round-robin on 10 Qwen3-32B servers (8 for the third workload). All figures below are from llm-d's own write-up, not an independent test.

Workload	Bottleneck	Matched config	Throughput vs round-robin	Knee, matched vs round-robin
Code generation (55k-token mean system prompt, 1,500 new tokens/turn)	prefill	prefix + token load	46k vs 16k input tokens/s (2.9x)	concurrency 60 vs 30
Reasoning (250-token shared prompt, 8,000-token mean output)	decode	prefix + active requests	7.0k vs 6.8k output tokens/s (parity)	concurrency 350 vs 250
B2B SaaS (many 6,000-token prompts, fixed repeating corpus)	prefill	prefix + token load	90k vs 45k input tokens/s (2.0x)	38 vs 18 QPS

The pattern is the useful part. Where prefill dominates and prefixes are long, affinity roughly doubles to triples capacity. Where decode dominates, there is little prefill to save, so throughput is at parity and the only gain is a later knee. The same routing rule is not a universal win, and the post's choice to pair the load signal with the bottleneck is the honest way to say so.

What is the break-even point for stickiness?

Take one request whose prefix of P tokens is cached on endpoint A and absent from endpoint B. It carries U new tokens. On A it costs U tokens of prefill after waiting q_A tokens in queue. On B it costs P + U tokens after waiting q_B. Sticking is faster for that request while

q_A + U <= q_B + P + U   =>   q_A - q_B <= P

The queue gap you should accept is exactly the number of tokens the cache saves. For P = 55,000 at 20,480 tokens/s that is 2.69 s. A cached request costs 73 ms of prefill; a cold one costs 2.76 s, a ratio of 37.7.

So llm-d's 14 s tolerance is not the per-request break-even. It is 5.2 times larger. That looks wrong until you count what a spill costs beyond the one request. A cold fill burns 2.76 s of the receiving endpoint's capacity, and it inserts a 13.4 GiB prefix into a cache that holds four, evicting something another request was about to hit. The single-request formula ignores both. The next section tests whether that externality is large enough to matter.

What happens if you set the limit at break-even?

I built a small discrete-event simulation. It is illustrative, loosely modeled on the code-generation workload, and it is not a reproduction of llm-d's benchmark. The assumptions:

10 endpoints, each a FIFO queue serving prefill at 20,480 tokens/s. Decode and continuous batching are ignored.
40 prefix groups of 55,000 tokens, requested with Zipf skew (exponent 1.1, so the most popular group takes 27.2% of traffic and the top five take 58.5%). Each request adds 1,500 new tokens.
Each endpoint's cache is an LRU holding 4 prefixes, from the 64 GiB budget above.
Poisson arrivals, 600 s per run, 5 seeds averaged. The router sees exact queue length and cache contents, which a real router only approximates.

Three policies: round-robin; always sticky (the warm endpoint with the least queue, spill only when no endpoint is warm); and sticky until saturated (spill when the best warm endpoint's queue passes tau).

The core of the third policy is short:

warm = [i for i in range(N) if g in cache[i]]
least = min(range(N), key=lambda i: pend[i])        # pend = queued tokens
if warm:
    bw = min(warm, key=lambda i: pend[i])
    e = bw if pend[bw] <= tau else least            # stick until saturated
else:
    e = least

Arrival rate	Round-robin p90 TTFT	Always sticky p90	Sticky until saturated p90	Hit rate (RR / sticky / saturated)
4 req/s	4.97 s	2.76 s	2.76 s	35% / 88% / 88%
10 req/s	437.5 s	1.97 s	1.97 s	35% / 95% / 95%
40 req/s	3,352 s	0.73 s	0.73 s	35% / 100% / 100%
60 req/s	5,294 s	121.8 s	3.21 s	35% / 99% / 99%
80 req/s	7,253 s	292.1 s	1,000 s	36% / 99% / 90%

Three things in that table can be checked by hand, which is how I convinced myself the simulation is not producing noise.

Round-robin's hit rate is 35%, so its average request costs 0.35 x 1,500 + 0.65 x 56,500 = 37,250 tokens. Ten endpoints deliver 204,800 tokens/s, which caps the fleet at 5.5 requests/s. The simulation falls over between 4 and 6. Its huge numbers at 10 req/s and above are the backlog after a 10 minute window, not a steady-state latency.

Always-sticky ties each prefix to one endpoint. The hottest prefix takes 27.2% of traffic at 73 ms per request, so its endpoint saturates at 1 / (0.272 x 0.0732) = 50 req/s. The simulation is healthy at 40 and in trouble at 60. That is hot-spotting, and a saturation release fixes it: at 60 req/s p90 drops from 121.8 s to 3.21 s.

At low rates p90 sits near 2.76 s for both sticky policies. That is one cold fill, and it comes from first-time prefixes and LRU evictions. No router removes it.

The last row is the part I did not expect to write. At 80 req/s the saturating policy does worse than always-sticky (1,000 s against 292 s) because the fleet is simply short of capacity, and spilling turns scarce capacity into cold fills. A saturation release helps a skewed fleet that has total headroom. It does not create headroom.

Now the sweep that answers the break-even question. Same simulation at 60 req/s, varying tau only:

tau (tokens)	p90 TTFT	Hit rate
27,500	5,303 s	35%
55,000 (break-even)	5,289 s	35%
110,000	5,289 s	35%
220,000	2.42 s	99%
286,720 (llm-d)	3.21 s	99%
573,440	5.66 s	99%
1,146,880	15.5 s	99%
no limit	121.8 s	99%

Below roughly 110,000 tokens the system falls into the same state as round-robin. A spill creates a cold fill, the cold fill evicts a warm prefix, the next request for that prefix misses, spills again, and the hit rate sinks to 35% and stays there. Above the cliff everything is fine, and then latency climbs slowly as stickiness turns back into hot-spotting. The shape is a cliff on the low side and a slope on the high side, which argues for erring high.

Be careful with the exact position of the cliff. It comes from my parameters (4 slots, 40 groups, 60 req/s) and a router that sees perfect state. A real fleet has chunked prefill, imprecise cache indexing and a different mix. What transfers is the mechanism and the asymmetry, not the number 110,000.

How do you calibrate this on your own fleet?

Measure R_peak where you will run it. Time a prefill of one full scheduler batch on the real model, tensor-parallel layout and engine flags. R_peak is a property of that exact setup, and it will move if you change max-num-batched-tokens.
Pick T_max from your TTFT objective, not from the post. Fourteen seconds is the benchmark's tolerance, and an interactive product will want less. Compute tau and express it in scheduler chunks so it is easy to sanity check.
Compute the break-even for your median prefix (P tokens, so P / R_peak seconds) and confirm your tau is well above it. If it is not, you are in the region my simulation collapses in.
Count how many prefixes fit per endpoint with the KV-bytes formula. If everything fits everywhere, routing barely matters and round-robin is fine.
Replay a day of real prefix-hash traffic through a simulator like the one above before touching production, and watch hit rate, not only latency. A falling hit rate under load is the early warning for the cliff.
Match the load signal to the bottleneck. Token load for prefill-bound traffic, request count for decode-bound.

What should you not conclude from this?

The 2.9x and 2.0x figures come from the llm-d authors on their own 10-server cluster and workloads. They are a strong reason to test, not a forecast for your traffic. The decode-bound result at parity says the same thing from the other side.

My simulation is a one-queue-per-endpoint toy. It omits decode, continuous batching, cache-indexing error and multi-tenant priorities, all of which the production router handles. It supports one claim only: a stickiness limit set at the single-request break-even can be catastrophic, because the cost of a spill lands on other requests.

FAQ

What is prefix-cache-aware routing?
Sending a request to an inference endpoint that already holds the KV cache for the start of its prompt, so the endpoint can skip recomputing those tokens and only prefill the new ones.

What does "sticky until saturated" mean in llm-d?
Requests stay on the cache-warm endpoint until its queued token load passes a calibrated limit (tau = R_peak x T_max). After that, routing falls back to load alone.

How is llm-d's 286,720-token limit derived?
In the published setup, 8,192 tokens take 0.40 s, so one endpoint prefills 20,480 tokens/s. With a 14 s TTFT degradation tolerance, 20,480 x 14 = 286,720 tokens, which equals 35 batches of 8,192.

Is round-robin ever the right choice?
Yes, when all your prefixes fit on every endpoint, or when requests share little prefix. Then there is no cache to protect, and the decode-bound benchmark at parity shows the gain can vanish.

Why not set the limit at the break-even queue gap?
Because a spill costs more than one request: a cold fill uses 2.76 s of the receiving endpoint's capacity and can evict a warm prefix. In my simulation, limits at or below about twice the break-even collapsed the hit rate to 35%.

Does this apply outside Kubernetes?
The logic does. It needs only per-endpoint queue depth in tokens and some record of which endpoint holds which prefix, which you can approximate by hashing prompt prefixes.

When Is a KV Cache Slot Actually Free? Lessons From the SGLang and Cloudflare Cross-Request Exposure

AI Tech News — Sun, 01 Nov 2026 14:07:13 +0000

TL;DR

What was reported: On 2026-10-26, Cloudflare and the SGLang team disclosed that Cloudflare's GLM-5.3 deployment on Workers AI returned 155 responses containing data from other customers' requests between 2026-08-27 and 2026-09-09. Cloudflare says all 155 were answers to prompts longer than 681,280 tokens, and that no other Workers AI model was affected.
Two bugs, one question: A configuration gap (prefill KV pool of 681,280 tokens against a 1,048,576-token context window) made an error path reachable. Separately, SGLang's disaggregated serving could free a decode-side KV slot while a launched RDMA write was still on its way to it. Both come down to: when is a slot really free?
The invariant: A slot is safe to reuse only when no remote writer can still touch it. The SGLang write-up calls the abort race "a distributed use-after-free."
The fix has a formula: hold time must exceed abort propagation + remaining chunks + the call already in flight. If the acknowledgement path dies, every abort waits out the full timeout, and held capacity is roughly abort rate x hold time x pages per request.
What to do today: Check pool size against context length after every config change, run SGLang v0.5.21 or later for PD disaggregation, pair aborts at the router, and watch the release outcome metric. Scripts below.

What actually happened in the Cloudflare and SGLang disclosure?

Cloudflare's report describes a model that launched on 2026-08-27 with a configured context window of 1,048,576 tokens. The prefill KV cache pool could hold only 681,280 tokens. That is 65.0% of the window, a gap of 367,296 tokens or 35.0%. A request could therefore fit the context window and still exceed the memory available to process it. In Cloudflare's words, nothing forced the two numbers to agree.

On the engine side, the SGLang write-up says that in disaggregated prefill/decode serving, "a request marked for rejection could still proceed through the handoff. Decode could then read cache pages that had not been populated for that request." Cloudflare describes the same thing from the outside: the rejected request was replaced by a one-token placeholder, decode got a "transfer complete" message, and it assumed prefill had overwritten the old cache contents. Prefill had not, so stale pages from earlier requests survived.

The timeline, from the two primary documents:

Date (2026)	Event
Aug 27	Exposure window opens (GLM-5.3 launch on Workers AI)
Sep 9	Cloudflare disables public access to the model, notifies SGLang maintainers, restores service that evening after a larger pool and an engine patch
Sep 16	SGLang merges the intake admission fix (#38935) and optional KV transfer checksums (#39500)
Oct 2	v0.5.21: deferred KV release on by default for Mooncake and NIXL
Oct 3	Generation-tagged acks and MORI support (#37077), shipped in v0.5.22
Oct 26	Coordinated disclosure

One scope note matters for your risk assessment: the SGLang post says default PD configurations were affected from the v0.4.x line until v0.5.21 (v0.5.22 for MORI), and that the fix line was not backported to earlier release branches. A single-instance deployment without the cross-engine transfer path is not affected by the abort race.

Why can a KV slot be "free" and still get written to?

The decode engine pre-reserves destination slots for each incoming request, and its allocator marks them allocated or free. The disclosure's key observation is that "free" has a second, hidden condition. The allocator's bit says nothing about writes that were already posted to a NIC queue.

The failure takes four steps:

A client disconnect, admission rejection, or transfer fault aborts request A while at least one launched write has not landed.
Decode frees A's slots. The allocator marks them free.
The allocator hands those slots to request B.
A's late write lands inside B's KV cache. B decodes on foreign keys and values, with no error anywhere.

The write-up gives four facts that make step 2 to step 4 possible, and each is worth knowing if you run any disaggregated stack:

The allocator can mark a slot free while writes are in flight. It and the NIC do not share state.
Launched RDMA work cannot be cancelled. There is no call to cancel one work request already posted to the queue pair, and Python cannot interrupt a blocking C++ transfer call mid-flight. A pull design does not escape this, because a posted READ is as hard to take back as a posted WRITE.
Router cancellation helps but cannot finish the job. An abort is only safe when both halves of a prefill/decode pair learn about it, yet whatever the transfer engine already launched continues regardless.
The window scales with context length. A short prompt is in flight for milliseconds. A 128K to 1M-token context keeps writes in flight for seconds to minutes, so the same abort rate produces far more exposure on long-context traffic.

The corruption is silent by design. The transfer reports success and no component logs an error. A single contaminated slot derails a whole generation because attention reads every position. In the team's 2xH200 reproduction, the victim counted correctly up to 80 and then its output turned into the donor's prompt text.

The race is probabilistic in production, but a client can raise the odds by sending long-context requests and disconnecting mid-transfer. The client cannot choose the victim and learns nothing back, so the impact is one-way leakage plus an integrity problem.

How likely is corruption if you free slots at abort time?

I wrote a small Monte Carlo to see how the three ingredients interact. This is an illustrative model with stated assumptions, not a measurement of SGLang. The core of it is below; the assumptions are:

KV size of 37.5 KB per token, derived from the one data point in the post: a single call for an 800K-token request carried 30 GB (taking GB as 10^9 bytes).
8,192-token chunks at an effective 25 GB/s, so one chunk write takes about 12.3 ms.
Prefill at 20,000 tokens/s, so 128K tokens take 6.6 s and 1M tokens take 52.4 s.
The allocator reuses freed slots after an exponentially distributed delay. The x-axis is the mean of that delay, standing in for memory pressure.
Every abort lands mid-transfer, as the disclosure's conditions require.

import random

def p_contam_free_at_abort(tail_fn, mean_reuse, n=200_000):
    """Slots are freed at abort and reused after Exp(mean_reuse).
    Corruption if a launched write lands after the reuse."""
    bad = 0
    for _ in range(n):
        reuse = random.expovariate(1.0 / mean_reuse)
        bad += tail_fn() > reuse
    return bad / n

chunk_t = 8192 * 37_500 / 25e9                      # one chunk write, about 12.3 ms
tail_notify = lambda: chunk_t * random.random()     # abort reached prefill
tail_silent = lambda ctx: random.random() * ctx / 20_000   # prefill keeps pushing

Two readings follow. When the abort does not reach prefill, the tail is every chunk prefill still has to push, so corruption is near certain unless reuse takes seconds. Notifying prefill (SGLang's #27372) shrinks the tail to one in-flight chunk and collapses the curve, but at a 10 ms mean reuse delay my model still corrupts 43% of mid-transfer aborts. Notification narrows the window. Only holding the slots closes it. The disclosure frames it the same way: the router decides who dies, and the engine decides when memory is safe.

What does the fix hold, and for how long?

Decode cannot cancel prefill's launched writes, so it keeps the aborted request's slots reserved for longer than those writes can keep landing:

T_hold > t_abort_propagation + t_remaining_chunks + t_in_flight_call

The design makes every abort path hold the slots, then bounds each term on the right:

Notify prefill on abort (#27372). Decode sends ABORT to every prefill rank, and each rank drops chunks still waiting in its software queue. That drives t_remaining_chunks toward zero.
Hold, do not free (#35049, NIXL port #35360). On abort, decode holds the request's KV slots, its row in the request pool, and its metadata buffer entry. Each prefill rank counts outstanding work per room and replies ABORT_ACK only when that count drains to zero. Decode releases when every notified rank has acknowledged, or when a timeout fires. The default timeout in that PR is 30 seconds.
Gate by capability, then default on (#37454, #41023). The hold only helps on backends that acknowledge, so it shipped opt-in and became the default in v0.5.21.

The timeout is described as "a practical backstop, not a proof", because a call that times out can leave writes the engine is still retrying. That is why #43719 (merged 2026-10-12) makes prefill withhold the drain ack for any room whose send failed. Before it, a failed Mooncake send still acked as drained.

The design rule in the conclusion is worth copying: make every degradation lengthen the hold.

How long should the timeout be?

The post recommends 60 to 120 seconds for SGLANG_DISAGGREGATION_DEFERRED_DECODE_KV_RELEASE_TIMEOUT, and says a larger value "adds little safety and multiplies held KVCache during abort storms." I checked the shape of that claim with a second illustrative model: 5% of aborts take a degraded path where writes keep retrying past the 30 s transfer timeout (30 s plus an exponential with a 20 s mean), and the release is timeout-only.

Below the transfer timeout the hold is useless on the degraded path (about 500 per 10,000 corrupted, which is just the 5% degraded share). At 60 s it falls to 113, at 120 s to 6, and at 240 s to 0 in 200,000 trials. The tail decays exponentially under my assumption, so the second doubling buys almost nothing while it doubles the cost below. Measure your own retry distribution before copying a number.

What does holding cost in capacity?

The post gives the sizing rule: if releases fall back to the timeout, held capacity is about abort rate x T_hold x pages per request. With the healthy ack path, a hold is released milliseconds after the drain and costs nothing noticeable. With a dead ack path, every abort pays the full timeout.

Abort rate	T_hold	Held KV at 128K-token reservations (4.8 GB each)
1 per minute	30 s	2.4 GB
1 per minute	120 s	9.6 GB
6 per minute	30 s	14.4 GB
6 per minute	120 s	57.6 GB

At the 37.5 KB/token figure, one 800K-token reservation is 30 GB, so a handful of long-context aborts waiting out a 120 s hold can pin a large share of a decode pool. This is why the metric below matters as much as the fix itself: a dead ack path turns a safety feature into an admission-latency problem.

The SGLang team measured the steady-state price as small: across 21 runs on 2xH100 (1P1D Mooncake, 1024 requests at concurrency 256), throughput was statistically indistinguishable with the feature on or off, all deltas within plus or minus 2%. The hold only applies on the abort path.

Which pre-flight check catches the configuration half?

Cloudflare's first recommendation is to measure the KV pool against context length after every configuration change. That check needs no GPU. It needs two numbers from your engine: the token capacity of the prefill KV pool and the configured context length. Field names differ across versions, so read them from your engine's startup log or info endpoint and verify on your build.

import re

def pool_gate(pool_tokens: int, context_len: int, max_ctx_per_req: int | None = None) -> str:
    """Fail the deploy if a request that fits the context window cannot fit the prefill KV pool."""
    need = max_ctx_per_req or context_len
    if pool_tokens < need:
        short = need - pool_tokens
        return f"FAIL pool={pool_tokens} < context={need} (short by {short}, {short / need:.1%}); lower context_len or grow the pool"
    return f"OK pool={pool_tokens} covers context={need}"

print(pool_gate(681_280, 1_048_576))
# FAIL pool=681280 < context=1048576 (short by 367296, 35.0%); lower context_len or grow the pool

Make FAIL block the rollout. It would have flagged the reported configuration on day one.

How do you know the protection is actually on?

SGLang v0.5.22 ships a counter, sglang:decode_deferred_kv_release_total{outcome}. The post says a healthy deployment releases almost all holds through drained, while a timeout-heavy ratio means the ack path needs attention. Here is a parser and a verdict function; I ran it against the synthetic sample shown, whose numbers are made up.

PAT = re.compile(r'sglang:decode_deferred_kv_release_total\{([^}]*)\}\s+([0-9.eE+-]+)')

def release_outcomes(metrics_text: str) -> dict:
    out = {}
    for labels, val in PAT.findall(metrics_text):
        m = re.search(r'outcome="([^"]+)"', labels)
        if m:
            out[m.group(1)] = out.get(m.group(1), 0.0) + float(val)
    return out

def verdict(outcomes: dict, min_drained: float = 0.95) -> str:
    total = sum(outcomes.values())
    if total == 0:
        return "NO DATA: no holds yet, or the metric is missing on this build"
    drained = outcomes.get("drained", 0.0) / total
    return ("HEALTHY" if drained >= min_drained else "ACK PATH SUSPECT") + f" drained={drained:.1%} of {int(total)} holds"

sample = '''sglang:decode_deferred_kv_release_total{outcome="drained",tp_rank="0"} 9712
sglang:decode_deferred_kv_release_total{outcome="timeout",tp_rank="0"} 41
sglang:decode_deferred_kv_release_total{outcome="error",tp_rank="0"} 3'''
print(verdict(release_outcomes(sample)))
# HEALTHY drained=99.5% of 9756 holds

The 0.95 threshold is my choice, not SGLang's. Set yours from your own baseline after rollout.

Beyond the counter, the engine offers an opt-in integrity check: --disaggregation-enable-kv-checksum on both prefill and decode (their KV layouts must match). It computes an Adler-32 checksum on the GPU and verifies the received cache before the request joins a running batch. It is off by default because each transfer pays an extra GPU read and a synchronization point, and it catches only late writes that land before the commit point. The hold covers the rest.

Operators still on v0.5.19 or v0.5.20 are told to set SGLANG_DISAGGREGATION_DEFERRED_DECODE_KV_RELEASE=1 on both servers, plus SGLANG_MOONCAKE_MAX_TRANSFER_BATCH_INDICES on v0.5.20. The post is explicit that this does not replace upgrading, because failure kinds other than decode-initiated aborts stay unprotected before v0.5.21.

Does this apply to engines other than SGLang?

The write-up says the four steps apply to "any PD implementation that moves KVCache between engines with asynchronous DMA, push or pull." It does not say other engines are affected, and I have not tested any. Ask three questions of your own stack, whichever engine it runs:

When a request dies mid-transfer, what delays the allocator's reuse of its destination slots?
Is that delay tied to a positive signal that the writer finished, or only to a timer?
If the signal path breaks, does the system get safer (longer hold) or less safe (earlier release)?

If you cannot answer these from the code or docs, you have found your next audit item.

FAQ

What exactly leaked in the Cloudflare incident?
Cloudflare reports 155 responses that contained data from other customers' requests, all to prompts longer than 681,280 tokens, between 2026-08-27 and 2026-09-09. Only GLM-5.3 on Workers AI was affected, per its investigation.

Is my single-node SGLang deployment affected?
The SGLang post says a single-instance deployment without the cross-engine KV transfer path is not affected by the abort race. The exposed path is PD disaggregation with Mooncake, NIXL, MORI, or a custom transfer backend on affected versions.

Which version should I run?
Per the post, v0.5.21 protects Mooncake and NIXL by default, v0.5.22 covers MORI, and the fix was not backported. Some press coverage says the fix landed earlier; I follow the engine team's own release mapping.

Can I tell whether I was exposed before upgrading?
Not reliably. The corruption leaves no error or log line, and the opt-in checksum only helps going forward.

Does a bigger timeout make it safer?
Only until it exceeds the longest realistic write tail. Past that it mostly increases held capacity during abort storms.

Are the charts measurements?
The cover uses the reported pool and window sizes. The other three come from my own simulation, so read them as shapes, not SGLang numbers.

Sources

SGLang security update: improving request isolation (LMSYS, 2026-10-26)
Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang (Cloudflare)
SGLang pull requests cited: #38935, #39500, #32758, #37077
SGLang releases: v0.5.21, v0.5.22
Simulation assumptions: KV 37.5 KB/token (30 GB per 800K-token call, from #32758's tests as quoted in the post), 8,192-token chunks at 25 GB/s, prefill 20,000 tokens/s, exponential reuse delay, 5% degraded aborts with a 30 s timeout plus a 20 s-mean retry tail, 200,000 trials per point, seed 7.

Your LLM Serving Engine Ships Every 13 Days: Reading vLLM's Breaking-Change Lists as Upgrade Risk Data

AI Tech News — Sat, 31 Oct 2026 14:04:46 +0000

TL;DR

Cadence: In the 365 days to 2026-10-31, vLLM published 33 stable releases on PyPI, including 21 new minor series (0.12 through 0.32). The median gap between minor series was 13.5 days. SGLang published 19 releases, all inside 0.5.x (median gap 14.6 days). LMDeploy published 12 (median 26.5 days).
Breaking-change volume: The vLLM release notes for 0.28 through 0.32 list 30 items under "Breaking changes". The per-release count rose every time: 4, 5, 6, 7, 8.
The risky class: By my own classification, 8 of the 30 items (27%) change a default or a behavior while the exact same launch command still starts and passes a health check.
Lag has a price too: Staying on 0.28 and jumping to 0.32 means reviewing 26 items at once, and 0.32.0 is the first of the five releases whose notes cite security advisories (three of them).
What to do: Put an upgrade gate in front of every version bump: extract the breaking items automatically, diff the resolved server config, replay a golden request set. Working code for the first step is below.
Scope: I did not run an inference engine for this post. Every number comes from the PyPI JSON API and the GitHub release notes, and I say where my judgment enters.

How often do LLM serving engines actually ship?

Most teams I read about treat a serving engine like a database: pick a version, pin it, upgrade "when needed". The release data says the engine moves much faster than that mental model.

Method: I pulled https://pypi.org/pypi/<package>/json, took the earliest upload time of any file for each version, and dropped release candidates, dev builds and post releases. The window is 2025-10-31 to 2026-10-31.

Engine	Stable releases	Median gap (days)	Longest gap (days)	Latest in window
vLLM	33	9.9	32.3	0.32.0 (2026-10-18)
SGLang	19	14.6	41.7	0.5.23 (2026-10-30)
LMDeploy	12	26.5	43.1	0.18.0 (2026-09-28)

I left TensorRT-LLM out on purpose. Its PyPI listing shows 24 releases and a last upload of 1.2.1 on 2026-04-20, and I cannot tell whether PyPI is its primary channel. Guessing from a possibly incomplete source would put a number in this table that I cannot defend.

Of vLLM's 33 releases, 12 are patch releases. If you count only the first release of each minor series, you get 21 series with gaps of 15, 32, 9, 27, 9, 13, 13, 24, 17, 14, 14, 16, 11, 13, 16, 15, 13, 12, 13 and 13 days. The median is 13.5, the shortest is 9 and the longest is 32. The last four gaps (0.28 to 0.29 to 0.30 to 0.31 to 0.32) are 13, 12, 13 and 13 days, which is 52 days from 0.28 to 0.32.

Does the version number tell you how risky an upgrade is?

No, and the two big engines show why. vLLM is on 0.y.z, and the middle number moves with every scheduled release, so "0.31 to 0.32" is the normal unit of change. SGLang stayed in 0.5 for all 19 releases, so its normal unit is the patch number. A "patch bump" in one project is the same size of event as a "minor bump" in the other. Neither number carries a compatibility promise you can read off. The release notes are the only signal.

What did vLLM list as breaking changes from 0.28 to 0.32?

The GitHub notes for vLLM 0.28 through 0.32 each contain a bullet named "Breaking changes" with semicolon-separated entries. I counted entries. Two caveats: older releases label this differently (the 0.27.0 notes, for example, call the PyTorch upgrade "a breaking environment change" inside a highlight and have no dedicated bullet), so I used only the five releases with the dedicated bullet. And one entry can bundle several changes, such as "two environment variables removed", so the counts are a floor on what changed, not an exact tally.

Release	Date	Items	Examples
0.28.0	2026-08-26	4	bitsandbytes moved to an out-of-tree plugin; `calculate_kv_scales` removed
0.29.0	2026-09-09	5	ten deprecated model architectures removed; PyAV video decoder backend removed
0.30.0	2026-09-22	6	scale-out endpoints now opt-in via `--enable-scale-out`; GPTQ `g_idx` removed
0.31.0	2026-10-05	7	`tokenizer_mode="slow"` removed; `--enforce-eager` now also disables JIT kernel warmup
0.32.0	2026-10-18	8	Python 3.10 dropped; default CORS origins restricted; `max_tokens` no longer reserved from the context window

I then sorted all 30 entries into four classes. This is my judgment, and a few entries could go either way.

Class	Count	Examples from the notes
Dependency or Python pin	3	Python 3.10 dropped (0.32); `transformers>=5.16.1,<5.20.0` (0.32); Transformers bumped to 5.15.0 (0.28)
Default or behavior changed	8	default CORS origins restricted (0.32); `max_tokens` no longer reserved from the context window (0.32); `--enforce-eager` also disables JIT warmup (0.31); YaRN aligned with Transformers so vendor aliases no longer re-scale `max_model_len` (0.30)
Removed or renamed flag, env var, feature, backend	15	`tokenizer_mode="slow"` removed (0.31); `--enable-mamba-fine-grained-prefix-cache` renamed (0.31); `calculate_kv_scales` removed (0.28)
Deprecation or API rename	4	`python -m vllm.entrypoints.openai.api_server` deprecated for `vllm serve` (0.29); Responses API reasoning stream events renamed (0.32)

The four classes add up to 3 + 8 + 15 + 4 = 30.

Why are the behavior-change items the dangerous ones?

A removed flag usually announces itself: the deploy fails at launch, or your staging smoke test does. A removed environment variable is murkier, because plenty of programs ignore variables they do not recognize. Whether yours errors out is something to test on your exact version, not something to assume.

The 8 behavior items are different in kind. The command is identical. The process starts. The health endpoint returns 200. What changed is what the server does with the traffic. Four of them, read as an operator:

max_tokens no longer reserved from the context window (0.32). Capacity plans that assumed a long max_tokens reserves context for every request may now behave differently. The notes do not say whether admitted concurrency goes up or down for your workload. I would replay traffic with realistic max_tokens values and compare, rather than guess.
--enforce-eager now also disables JIT kernel warmup (0.31). If you run eager mode to save memory or to debug, I would expect first-request behavior to change. I have not measured it. Cold-start p99 is the number to compare.
YaRN aligned with Transformers (0.30). If you serve a long-context checkpoint that relies on YaRN scaling, the effective max_model_len can differ from before. Compare the context length the server reports at startup, old version against new.
Default CORS origins restricted (0.32). This is a security improvement and the direction is right. It is still a behavior change that can break a browser client that talks to the server directly.

This is the same family of failure as the baked-in config trap I covered in an earlier post on KV cache dtype: identical flags, different conditions. The fix there was to compare the resolved configuration, not the command line.

What does falling behind cost?

Arithmetic from the same five release notes. Upgrading one release at a time, you review 5, 6, 7 and 8 items on the way from 0.28 to 0.32. Staying on 0.28 and jumping straight to 0.32, you review 5 + 6 + 7 + 8 = 26 items in a single change window, and you cannot bisect which one broke you because they all landed together.

The pinned option also has a cost on the security side. The 0.32.0 notes cite three GitHub security advisories as fixed: GHSA-4hhp-h66f-j5j7 (chat template resource exhaustion), GHSA-q97c-p368-75c4 (tokenizer max_token_id) and GHSA-frvg-qw88-hww9 (P2P ZMQ socket exhaustion). A text search for "GHSA" in the 0.28 to 0.31 bodies finds none. Maintainers can describe a fix without citing an ID, and the notes do not say whether anything is backported to older lines, so read this as "the three advisories I can see are in 0.32.0", not as proof that earlier versions were unsafe.

At a 13.5-day median gap, a "we upgrade quarterly" policy skips roughly 6 to 7 minor series per jump. Small, frequent, gated upgrades give you a smaller diff each time and let you pick up fixes sooner. That is an opinion, but it follows directly from the numbers above.

How do you build an upgrade gate?

Four layers, each aimed at a class from the table:

Class	What catches it
Dependency or Python pin	Build the new image in CI, run `pip check`, boot it
Removed or renamed	Startup smoke test with the exact production arguments
Default or behavior changed	Resolved-config diff plus golden request replay
Deprecation or API rename	Grep startup logs for deprecation warnings, schedule the removal

Step 1: extract the breaking items automatically

This script pulls the "Breaking changes" bullet for any release tags you give it. I ran it against v0.29.0 through v0.32.0 and it returned 5, 6, 7 and 8 items, 26 in total, which matches the counts above.

import json, re, sys, urllib.request

def breaking_items(tag, repo="vllm-project/vllm"):
    url = f"https://api.github.com/repos/{repo}/releases/tags/{tag}"
    body = json.load(urllib.request.urlopen(url, timeout=30))["body"] or ""
    m = re.search(r"\* \*\*Breaking changes?\*\*:(.*?)\n", body)
    return [s.strip() for s in m.group(1).split(";")] if m else []

if __name__ == "__main__":
    total = 0
    for tag in sys.argv[1:]:
        items = breaking_items(tag)
        total += len(items)
        print(f"## {tag}: {len(items)} items")
        for it in items:
            print("  -", it)
    print("total:", total)

Run it as python breaking.py v0.30.0 v0.31.0 v0.32.0. It depends on the notes keeping this bullet format, which held for 0.28 through 0.32 and did not hold for earlier releases, so treat an empty result as "go read the notes", not as "nothing broke".

Step 2: diff the resolved config, not the command line

Start the candidate version in staging with the exact production arguments. Save the startup log and the model-listing response from the old and the new server, then diff them. You are looking for any value you did not set that moved: context length, default dtypes, enabled endpoints, warmup behavior. This is what catches the YaRN and --enforce-eager style changes.

Step 3: replay a golden request set

The sketch below is illustrative. I did not run it here because it needs two live endpoints. It sends a fixed set of prompts at temperature 0 to the old and the new server and compares text, finish reason, token count and latency.

import json, statistics, time, urllib.request

def call(base, model, prompt, max_tokens=256):
    body = json.dumps({"model": model, "temperature": 0, "max_tokens": max_tokens,
                       "messages": [{"role": "user", "content": prompt}]}).encode()
    req = urllib.request.Request(base + "/v1/chat/completions", body,
                                 {"Content-Type": "application/json"})
    t0 = time.perf_counter()
    r = json.load(urllib.request.urlopen(req, timeout=120))
    c = r["choices"][0]
    return (c["message"]["content"], c["finish_reason"],
            r["usage"]["completion_tokens"], time.perf_counter() - t0)

def compare(old, new, model, prompts):
    same, lat_old, lat_new, fin = 0, [], [], 0
    for p in prompts:
        a, b = call(old, model, p), call(new, model, p)
        same += a[0] == b[0]
        fin += a[1] != b[1]
        lat_old.append(a[3]); lat_new.append(b[3])
    n = len(prompts)
    return {"identical_text": same / n, "finish_reason_changed": fin / n,
            "p50_old": statistics.median(lat_old), "p50_new": statistics.median(lat_new)}

Temperature 0 does not guarantee identical text across engine versions, or even across batch compositions on the same version, because kernels and reduction orders change. So do not demand 100% identical text. Track the identical fraction as a trend, treat a jump in finish_reason_changed as a red flag, and score a task-level metric (exact match, pass rate) on the same prompts for the decision that matters.

Step 4: canary with a rollback condition

Send a small share of live traffic to the new version and write the rollback condition down before you start: error rate, p99 latency, and one quality proxy. If you cannot name the number that triggers rollback, you are not ready to canary.

What are the limits of this analysis?

Release times are PyPI upload times, not GitHub tag times. The two can differ by hours.
The breaking-change counts depend on how maintainers wrote one bullet. A longer bullet is not necessarily a bigger break.
The four-way classification is mine. Reasonable people would move a few entries.
I covered one engine's notes in depth. SGLang and LMDeploy appear only in the cadence table.
Nothing here measures performance or accuracy. It measures how much changes, and how loudly.

FAQ

How often does vLLM release a new version?
Over the 365 days to 2026-10-31, PyPI shows 33 stable releases, including 21 new minor series. The median gap between minor series was 13.5 days, and the last four gaps were 13, 12, 13 and 13 days.

Does the version number tell me whether an upgrade is breaking?
Not in vLLM or SGLang. vLLM bumps the middle number every scheduled release, and SGLang stayed within 0.5.x for 19 releases. Read the release notes instead.

Which breaking changes are most dangerous for production?
The ones where your launch command still works but behavior differs: changed defaults, changed accounting such as how max_tokens is treated, changed context-length handling. In the five releases I classified, that was 8 of 30 items.

Should I pin an old version for stability?
Pinning is fine as a short-term choice, but it grows the eventual jump (26 items from 0.28 to 0.32) and delays fixes. The 0.32.0 notes list three security advisory fixes. A small upgrade every few weeks, behind a gate, is easier to debug than one big jump.

How do I find breaking changes without reading every note?
Parse the "Breaking changes" bullet with the script above, then read only the entries it returns. Empty output means the format changed, so read the full notes.

Does temperature 0 give identical output across engine versions?
No guarantee. Kernel and scheduling changes can alter floating-point reduction order. Compare identical-text rate as a trend and use a task metric for pass/fail.

Sources

vllm on PyPI, sglang on PyPI, lmdeploy on PyPI
vLLM release notes: v0.28.0, v0.29.0, v0.30.0, v0.31.0, v0.32.0

One Fleet for Serving and Training? What a 325M Pretraining Run on an Inference NPU Means for AI Infrastructure

AI Tech News — Mon, 26 Oct 2026 06:12:23 +0000

Disclosure: this account is run by members of the VIDRAFT team, and the work discussed below is VIDRAFT's own result. It is not an independent review.

TL;DR

Most AI infrastructure is planned as two fleets: GPUs for training, cheaper accelerators for serving. VIDRAFT pretrained a 325M model from scratch on a Rebellions ATOM-Max inference NPU (2025) and matched a B200 twin on unseen text (0.7993 vs 0.7994). For capacity planners, the interesting question is how much training work could move onto serving hardware.

Why do teams keep two fleets today?

Serving accelerators are cheaper per query and more power-efficient, but their software runs forward passes only. Anything that needs gradients, from LoRA adaptation to pretraining, goes to GPUs, often rented and often in another region.

What changes if serving hardware can train?

Utilization. Serving fleets idle off-peak; training jobs could fill those hours.
Data locality. Adaptation can run where the data and the deployed model already live.
Procurement. One hardware class covers more of the model lifecycle.

What was measured?

The run covered 5.0B tokens in 76,292 steps over about four days without divergence. On 1,024 unseen windows the paired difference to the GPU twin is -0.0001 (standard error 0.0007).

What should planners not assume yet?

Throughput and cost per run are not reported.
The chip alone does not train. The ATOM-Max software is built for inference; training relies on VIDRAFT's own stack and a stability technique under patent application.
Scale beyond 325M is unproven.

FAQ

Which NPU? Rebellions ATOM-Max, released in 2025.

Can the results be checked? Yes: weights and evaluate_unseen.py are in the model repo.

Sources

Technical report: https://huggingface.co/blog/FINAL-Bench/pretraining-a-325m-language-model-from-zero-on-an

AI-automated attacks are forcing application security to automate defense

AI Tech News — Fri, 23 Oct 2026 09:23:13 +0000

Through 2026, the story in application security has been automation on the offensive side. Open-source AI penetration tools now chain reconnaissance, injection, authentication bypass and bulk data access with little human direction. The techniques are old; the speed and autonomy are new.

Why speed changes the math

When an attacker's loop runs in minutes without supervision, a defense that depends on a human scheduling scans and triaging results cannot keep pace. The response across the industry has been to automate the defensive loop in three ways.

Autonomous auditing: tooling that forms its own hypotheses and widens coverage without a human specifying every step.
Verification layers: a second pass, increasingly an LLM, that filters false positives so teams act on evidence rather than noise.
Daily fingerprint refresh: keeping a vulnerability database current so newly disclosed flaws are caught the same day.

The non-destructive constraint

There is a catch. Active, attack-style scanning can take production down. Mature teams draw a hard line between offensive testing in isolated environments and non-destructive auditing against live systems. The useful defensive tools stay non-destructive by design.

The data-sovereignty angle

For regulated sectors, the other constraint is that code and findings cannot leave the network. That is pushing interest toward on-prem models for the verification layer, so the whole loop can run inside a boundary. Whether on-prem matches cloud accuracy is now a live benchmark question, and an interesting one to watch.

KV Cache Offloading to NVMe: The Break-Even Bandwidth Math and Where It Does Not Pay Off

AI Tech News — Wed, 21 Oct 2026 16:04:58 +0000

TL;DR

Loading a cached KV prefix from NVMe only beats recomputing it when the drive can deliver more bytes per second than the break-even bandwidth for that prompt length, model, and GPU.
In a recent measurement study (arXiv 2609.11744, Llama 3.2 3B on an H100 node), the break-even bandwidth fell from 23.2 GB/s at 1k tokens to 10.4 GB/s at 8k and 3.5 GB/s at 80k tokens. A drive that sustains 13.5 GB/s clears the bar from 8k up and misses it at 1k.
By a first-order model, bigger models are easier to offload: KV bytes per token grow slowly with model size, while prefill FLOPs per token grow with parameter count. A 70B-class model needs about 1 GB/s in the illustrative calculation below, and a 3B model needs several times that.
The same study found a trace replay on an H100 where the average request sat below the SSD break-even point (6,203 tokens for Qwen3 4B) and external caching gave no TTFT gain over GPU prefix caching alone.
Treat external KV caching as an admission decision made per prefix, not a switch you turn on for a cluster.

What question does a KV offload tier actually answer?

Prefix caching stores the key and value tensors computed for a prompt prefix so a later request with the same prefix can skip that part of the prefill. When the GPU cache is too small to hold every prefix you care about, an offload tier (CPU DRAM, then NVMe) keeps more of them. The question is never "is a bigger cache better". It is whether, for one specific hit, this inequality holds:

t_fixed + (prefix_tokens * kv_bytes_per_token) / effective_bandwidth  <  t_prefill(prefix_tokens)

The left side is the cost of a hit: a fixed scheduling and copy overhead plus the transfer. The right side is what the GPU would spend recomputing the same prefix. If the left side is larger, the cache made the request slower. The py-kvcache study states it directly: for short prefixes or fast GPUs, recomputation can be faster than loading from an external cache.

Two terms in that inequality scale differently. KV bytes grow linearly with prefix length. Prefill time grows faster than linearly, because attention cost adds a term quadratic in length on top of the linear matrix multiplications. So longer prefixes are better candidates for offload, and the bandwidth you need goes down as prompts get longer.

How much bandwidth does break-even need?

The py-kvcache authors computed the bandwidth a cache path must sustain for a fully cached prompt to tie with recomputation. For Llama 3.2 3B, which stores 114,688 bytes of KV per token in BF16, on their H100 node (arXiv 2609.11744, section 5):

Prompt length	Break-even bandwidth
1k tokens	23.2 GB/s
8k tokens	10.4 GB/s
80k tokens	3.5 GB/s

Compare that with what the hardware delivered in the same study. A GPU to CPU copy sustained 54 to 56 GB/s on the PCIe 5.0 node, so a DRAM tier clears the requirement at every length tested. The llm-d disk path moved 9.0 GB/s at 1k tokens and 11.2 to 12.0 GB/s at larger sizes, which leaves it below the requirement at 1k and 2k tokens and above it from 8k onward. Their fio reference was 13.5 GB/s for the PCIe 5.0 drive and 6 GB/s for a PCIe 4.0 drive.

Two consequences follow. Storage bandwidth sets a minimum useful prompt length: on that setup, an 8k prompt needed 77.8% of its tokens to be a cached prefix to break even, while at 80k tokens a prefix of only 7.8% was enough. And a PCIe 4.0 drive at 6 GB/s fails the bar for this model until prompts are well past 8k tokens, so the "NVMe offload" label hides a factor of two between drive generations.

Why do larger models make offload easier?

Take the prefill cost per token for a dense model as about 2 * P FLOPs, where P is the parameter count. KV bytes per token are layers * 2 * kv_heads * head_dim * dtype_bytes. Dividing the two gives the bandwidth needed to keep pace with the GPU at long context, ignoring attention and overhead:

break_even_bandwidth  ~=  kv_bytes_per_token * effective_FLOPs / (2 * P)

Grouped-query attention keeps kv_heads at 8 for many current open models, so KV bytes per token barely move between a 3B and a 70B model, while P grows more than twentyfold. I read the configs for three open models from their Hugging Face repositories and took the Llama 3.2 3B value from the paper:

Model	Layers	KV heads	Head dim	KV bytes per token
Llama 3.2 3B	28	8	128	114,688 (112 KiB)
Qwen3 4B Instruct 2507	36	8	128	147,456 (144 KiB)
Qwen3 32B	64	8	128	262,144 (256 KiB)
Llama 3.3 70B Instruct	80	8	128	327,680 (320 KiB)

Here is the calculation I used, which you can adapt to your own GPU and model:

FL = 989e12 * 0.50   # H100 SXM BF16 dense peak x 0.50 achievable (illustrative)

def t_prefill(P, layers, q_heads, head_dim, n_tokens):
    matmul = 2 * P * n_tokens
    attn = layers * 2 * n_tokens**2 * q_heads * head_dim   # causal QK^T and AV
    return (matmul + attn) / FL

def required_bw(kv_bytes_per_token, P, layers, q_heads, head_dim, n_tokens):
    return kv_bytes_per_token * n_tokens / t_prefill(P, layers, q_heads, head_dim, n_tokens)

# Llama 3.3 70B at 32,768 tokens
print(required_bw(327_680, 70.55e9, 80, 64, 128, 32_768) / 1e9)   # about 0.88 GB/s

The results of that model, for prefixes of 8k and 32k tokens:

Model	8k tokens	32k tokens
Llama 3.2 3B	7.24 GB/s	4.70 GB/s
Qwen3 4B	6.97 GB/s	4.12 GB/s
Qwen3 32B	1.75 GB/s	1.30 GB/s
Llama 3.3 70B	1.07 GB/s	0.88 GB/s

How to read this honestly: these are illustrative numbers from the stated assumptions (H100 SXM class compute at 50% utilization, no fixed overhead), not measurements. They are a lower bound. For Llama 3.2 3B the model gives 7.24 GB/s at 8k and 2.76 GB/s at 80k, against the measured 10.4 and 3.5 GB/s. The gap is the fixed per-hit overhead that the formula ignores, and it matters most for short prompts. The paper also ran on a different H100 variant than the one my constants describe, so do not compare absolute values too closely. What carries over is the shape: the requirement falls with prompt length and falls sharply with model size.

Two practical readings. A 70B-class model has plenty of headroom even on a PCIe 4.0 drive (about 1 GB/s needed against 6 GB/s available), so offload economics look much better for large dense models. And anything that shrinks KV bytes per token lowers the requirement in proportion: an FP8 KV cache halves it relative to BF16, which ties into the earlier post on KV cache quantization linked below. With tensor parallelism, KV bytes and FLOPs both shard across GPUs, so to first order the per-GPU requirement is unchanged, provided each GPU has its own path to storage.

Does the spec sheet bandwidth match what you actually get?

Not by default. The py-kvcache study found several places where the delivered number fell away from the device number:

Data amplification. In the llm-d connector, a request holding 4.4 GB of useful KV data caused roughly 10 GB to be copied and read or written, because of a minimum staging allocation. The authors reported it upstream and it was fixed in llm-d pull request 589. After removing the minimum allocation, TTFT fell about 50% below LMCache in that test.
Asymmetric reads and writes. The connector's reads reached more than 12 GB/s while writes stayed around 2 GB/s. A workload that writes many new prefixes will see a different picture than one that mostly reads.
Request size. Block tracing showed the filesystem issuing mostly 512 KiB requests. In a standalone benchmark, I/O engines differed widely at 4 KiB, but at 1 MiB with one thread every engine clustered near the 6 GB/s that fio measured on that drive. Swapping the I/O engine did not improve TTFT. KV blocks are far larger than 1 MiB (one 256-token block of Llama 3.2 3B is about 28 MiB), so the workload is limited by bandwidth and transfer placement, not by small-I/O rate.
Scheduling. Where a transfer enters the request schedule matters. The authors' py-kvcache connector starts disk reads while a request is still queued, and preloading contributed a 1.34x improvement at 80k tokens.

For capacity planning, measure end-to-end TTFT with your connector on your drives and treat the fio number as an upper bound.

When does external KV caching not pay off?

The most useful result in the study is a negative one. The authors replayed three production-style traces from Alibaba Cloud Bailian (a coder trace, an interactive trace, and an API-driven trace) on two machines.

On the smaller GPU (an RTX 4000 Ada with 20 GiB), py-kvcache was 1.12x to 1.79x faster than GPU prefix caching alone and 1.10x to 1.25x faster than LMCache.
On the H100 node, the average request length was below the measured 6,203-token SSD break-even point for Qwen3 4B, and the larger GPU memory kept a large share of the working set resident. TTFT with GPU prefix caching alone was identical to py-kvcache, and vLLM's native offload implementation was 0.2 s slower.

The same traces flipped from "offload helps" to "offload costs you" depending on the GPU. The authors conclude that external KV caching is a setup-specific admission decision, and that recomputation is a valid fallback rather than a cache miss.

Where it does pay, the numbers are large. At 80k tokens, loading from disk with py-kvcache was 2.0x faster than LMCache. With GPU, CPU, and disk tiers all enabled it was 1.23x faster than LMCache and within about 4% of vLLM's native KV offload implementation.

How do you configure KV offload in vLLM, and how do you gate it?

In the vLLM v0.32.0 source (released October 18, 2026), CacheConfig exposes two offload options. kv_offloading_size is the buffer size in GiB (summed across tensor-parallel ranks), and offloading is only active when it is set. kv_offloading_backend accepts native or lmcache, and the docstring describes the offload target as CPU. If you want an NVMe tier, the study's comparisons use LMCache's disk support and an external connector, so check what your chosen connector supports.

# CPU DRAM tier with the native backend (GiB, summed across TP ranks)
vllm serve <model> --kv-offloading-size 64 --kv-offloading-backend native

The admission rule is separate from the tier. The py-kvcache design runs an offline calibration once per node and model, writes a small file with the break-even prefix length per source tier, and at lookup time declines a load that is predicted to cost more than recomputation. You can approximate the same policy:

def should_load(prefix_tokens, kv_bytes_per_token, measured_bw_Bps,
                t_recompute_s, t_fixed_s):
    t_load = t_fixed_s + prefix_tokens * kv_bytes_per_token / measured_bw_Bps
    return t_load < t_recompute_s

Fill the inputs from your own sweep, not from a data sheet:

Send prompts at several lengths (for example 1k, 4k, 8k, 16k, 32k, 64k) with all caches cold and record TTFT. That is t_recompute_s.
Send the same prompts again with the prefix resident only in the offload tier, and record TTFT. Subtract the non-transfer part to get the transfer time and the effective bandwidth.
Repeat with prefix fractions of 10%, 50%, and 90% of the prompt, since the break-even frontier depends on both length and fraction.
Store the break-even prefix length per tier and re-run the sweep whenever the GPU, model, KV dtype, or drive changes.

One caveat from the same paper: a static threshold on prefix length and bandwidth cannot tell a demand read from a preload that queueing may hide, so a stronger policy would also track staging capacity and per-tier transfer rates.

A separate benchmark, PrefixBench-H100 (arXiv 2609.19657), reports that cache effectiveness is largely insensitive to concurrency and output length on a single H100. That supports calibrating the break-even point offline.

FAQ

Is NVMe KV cache offloading faster than recomputing the prefill?
Only above a break-even bandwidth that depends on prompt length, model, and GPU. In the measured study, a 13.5 GB/s drive beat recomputation for Llama 3.2 3B from 8k tokens up, but a 1k-token prompt needed 23.2 GB/s.

How do I calculate the break-even bandwidth for my model?
Divide the bytes of KV for the cached prefix by the time budget left after subtracting non-transfer overhead from the recompute time. A first-order estimate is kv_bytes_per_token * effective_FLOPs / (2 * parameters). Measure the fixed overhead on your own stack, because it dominates short prompts.

Why do larger models need less offload bandwidth?
With grouped-query attention, KV bytes per token change slowly with model size, while prefill FLOPs per token scale with parameter count. More compute per byte of KV means recomputation is slower, so a slower drive is enough to beat it.

Should I enable KV offload on an H100?
It depends on your prefix length distribution. In the study's trace replay on an H100 node, the average request was below the SSD break-even point and GPU prefix caching alone matched the offload connector. Measure your own trace before committing storage hardware.

Does vLLM offload KV cache to disk by default?
No. In v0.32.0 offloading is off unless kv_offloading_size is set, and the native backend targets CPU memory per the config docstring. Disk tiers come from connectors such as LMCache or external ones.

Sources

Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs. arXiv 2609.11744 (September 2026). All measured figures attributed to "the study" come from this paper.
PrefixBench-H100: Characterizing Prefix Reuse and Time-to-First-Token in H100 LLM Serving. arXiv 2609.19657 (September 2026).
vLLM. v0.32.0 release and vllm/config/cache.py at v0.32.0 (KV offloading options).
Model configs read from Hugging Face: Qwen/Qwen3-4B-Instruct-2507, Qwen/Qwen3-32B, unsloth/Llama-3.3-70B-Instruct (mirror of the Llama 3.3 70B config).
NVIDIA. H100 Tensor Core GPU (peak compute figure used in the illustrative model).
Model outputs in the tables for the four models above are illustrative calculations from the stated assumptions, not measurements.

Speculative Decoding at High Batch Size: Where the 3x Speedup Turns Into a Tax

AI Tech News — Tue, 20 Oct 2026 14:05:46 +0000

TL;DR

Published speculative decoding speedups (2x to 3x) are almost always measured at low concurrency. The two vLLM posts I read this week report no concurrency sweep.
Speculation spends spare compute to save memory traffic. When the verify pass stops being memory-bound, the spare compute is gone and the extra draft tokens become pure overhead.
In an illustrative roofline model (27B dense verifier, H100-class GPU, 2k context), verifying 8 tokens per sequence turns compute-bound at about 30 concurrent sequences. Verifying 4 tokens per sequence holds out to about 85.
The throughput-optimal speculation length falls from 8 at batch 1 to 3 at batch 128. A fixed num_speculative_tokens is the wrong setting for a server whose load changes during the day.
The model overstates absolute speedups. Use it for the shape of the curve, then measure your own server with the sweep at the end.

Why do the headline speedups all come from batch 1?

Decode is memory-bound at small batch. Every step streams the full set of weights from HBM to produce one token per sequence, and the arithmetic units sit mostly idle. Speculative decoding exploits that idle compute: a cheap drafter proposes several tokens, and the large model verifies all of them in a single forward pass. Rejection sampling keeps the output distribution identical to ordinary decoding, so quality is unchanged (Leviathan et al., 2022).

The newest drafters make the drafting side cheaper. The vLLM Speculators team describes three parallel drafting methods: P-EAGLE, DFlash and DSpark. Instead of generating draft tokens one forward pass at a time like EAGLE-3, they predict a whole block of candidates in one pass, which decouples drafting latency from speculation length (vLLM blog, July 2026). The example launch command in that post looks like this:

vllm serve Qwen/Qwen3-30B-A3B \
  --tensor-parallel-size 2 \
  --reasoning-parser qwen3 \
  --speculative-config '{
    "model": "RedHatAI/Qwen3-30B-A3B-speculator.dflash",
    "num_speculative_tokens": 7,
    "method": "dflash"
  }'

A follow-up post on AMD GPUs reports measured speedups on MI300X and MI355X: 2.87x for DFlash on Gemma-4-26B-A4B (MATH500) and 2.34x on Gemma-4-31B. The spread across tasks is wide. DSpark on Qwen3-8B ranged from 1.15x on MATH500 to 1.63x on GSM8K (vLLM blog, August 2026). That post also notes that a broader evaluation across concurrency levels and batch sizes would help, which is the gap this article tries to fill with arithmetic rather than a new benchmark.

None of this means the technique is overrated. It means the batch size at which a number was measured belongs next to the number.

What does one speculative iteration actually cost?

The original analysis gives the expected number of tokens produced per verify pass when each draft token is accepted independently with probability alpha and the drafter proposes gamma tokens:

E(alpha, gamma) = (1 - alpha^(gamma + 1)) / (1 - alpha)

With alpha = 0.8 and gamma = 7 that is 4.16 tokens per iteration. The iteration is not free. It costs one verify pass over (gamma + 1) tokens per sequence plus one drafter pass. Throughput speedup is therefore:

speedup(B) = E * t_base(B) / ( t_verify(B, gamma + 1) + t_draft(B, gamma) )

Everything interesting hides in how t_verify grows with batch size B. At small B it barely grows, because the step is bound by weight and KV-cache reads that do not depend on how many tokens you verify. At large B it grows almost linearly with (gamma + 1), because the step is bound by compute.

How does the roofline model work?

I built the smallest model that captures that transition. Treat every number below as illustrative, not as a measurement of any specific deployment.

Assumption	Value
Verifier	27B dense, BF16 (54 GB of weights)
GPU	H100 SXM class: 3.35 TB/s HBM3, 989 TFLOPS BF16 dense (NVIDIA H100)
Achievable fraction	80% of peak bandwidth, 50% of peak compute
KV cache	64 layers, 8 KV heads, head dim 128, BF16: 262,144 bytes per token
Context	2,048 tokens per sequence (0.54 GB of KV read per step per sequence)
Drafter	Parallel drafter, 4% of verifier parameters, one pass per iteration
Step time	max(memory time, compute time)

The core of the script is short:

def t_step(B, tokens_per_seq, frac=1.0):
    mem = (W * frac + B * KV_SEQ) / BW          # weights + KV reads
    cmp_ = 2 * P * frac * B * tokens_per_seq / FL
    return max(mem, cmp_)

def speedup(B, gamma, expected_tokens):
    base = t_step(B, 1)
    spec = t_step(B, gamma + 1) + t_step(B, gamma, 0.04)
    return expected_tokens * base / spec

One closed form falls out of the model. With weight-read time t_w (20.1 ms here), KV-read time per sequence k (0.20 ms), and compute time per token c (0.109 ms), the verify pass becomes compute-bound at:

B* = t_w / ((gamma + 1) * c - k)

gamma	verify turns compute-bound at
1	about 1,100 sequences
3	about 85 sequences
5	about 44 sequences
7	about 30 sequences

Past B*, every extra draft token costs real GPU time. Below it, extra draft tokens are almost free. That single threshold explains the rest of the article.

Why should you not trust an independent-acceptance model for long drafts?

The formula above assumes every position is accepted with the same probability. Real drafters degrade with distance. The AMD post reports, for Gemma-4-26B with Gemma 4 MTP at N=5, a 94% first-position acceptance, 66% at the fifth position, and 80.0% overall. Under independent acceptance at 80%, gamma = 7 gives 4.16 tokens per iteration. If acceptance declines linearly from 94% by 7 points per position (my interpolation between the two reported endpoints, treating each as a conditional acceptance), gamma = 7 gives 4.49 tokens, but the marginal gain from each extra position collapses:

gamma	expected tokens per iteration (decaying acceptance)
1	1.94
2	2.76
3	3.41
5	4.20
7	4.49
8	4.53

Going from gamma = 5 to gamma = 8 buys 0.33 tokens per iteration while adding 3 more verified tokens per sequence. At low batch that trade is free. At high batch it is not.

What does the speedup curve look like as concurrency grows?

I ran the model with the decaying-acceptance assumption. Values are throughput relative to plain decoding at the same batch size.

Concurrent sequences	gamma = 2	gamma = 4	gamma = 7	Best gamma (speedup)
1	2.63x	3.71x	4.28x	8 (4.32x)
8	2.48x	3.50x	4.04x	8 (4.08x)
32	2.17x	3.06x	3.39x	6 (3.45x)
64	1.95x	2.64x	2.13x	4 (2.64x)
128	1.75x	1.85x	1.49x	3 (1.90x)
256	1.45x	1.45x	1.16x	3 (1.49x)

Three things stand out.

The curves cross. The long draft (gamma = 7) wins until about 32 sequences, then falls behind the short one by 64. That matches the closed form: gamma = 7 turns compute-bound near 30 sequences.
The optimum moves. The best speculation length is 8 up to 16 sequences, 6 at 32, 4 at 64, and 3 from 128 up.
Speculation rarely turns negative in this model, but it can. With the decaying acceptance curve, gamma = 7 first drops below 1.0x at about 513 sequences and gamma = 8 at about 329. With independent acceptance at alpha = 0.6, gamma = 7 breaks even at only 84 sequences. A weaker drafter or a harder task moves the break-even left fast.

What does this model leave out?

Be suspicious of the absolute numbers. A 4.3x speedup at batch 1 is higher than the 2x to 3x measured in the posts above, because the model ignores scheduler overhead, drafter launch latency, sampling, and the real shape of acceptance across tasks. It also has these specific blind spots:

Mixture-of-experts verifiers. A dense model touches all weights at every step. An MoE touches only the experts routed to. Verifying more tokens per sequence activates more distinct experts, so the verify pass reads more weight bytes as gamma grows. That shifts the break-even left compared with a dense model of the same total size. I did not model it.
Longer context. The KV term k grows with context length. A larger k makes the base step more memory-bound for longer, which pushes B* to the right (the denominator in the closed form shrinks). Long-context workloads can tolerate longer drafts at a given batch size.
Prefix caching and chunked prefill. Prefill chunks share each step with decode tokens and consume compute that the verify pass would otherwise use. The model treats decode in isolation.
Drafter quality varies by traffic. Acceptance on code and math is usually higher than on open-ended chat. A single alpha across your traffic mix is a fiction.

How do you measure this on your own server?

Sweep concurrency with speculation on and off, and log acceptance per position. Use identical prompts and output lengths for both arms, and warm any prefix cache the same way before each run so one arm does not inherit a hotter cache. Flag names below match recent vLLM; check vllm bench serve --help on your build.

# Arm A: speculation off.  Arm B: same command plus --speculative-config.
for C in 1 8 32 64 128 256; do
  vllm bench serve --model "$MODEL" --dataset-name random \
    --random-input-len 1024 --random-output-len 256 \
    --num-prompts $((C * 8)) --max-concurrency $C \
    --save-result --result-filename "specoff_c${C}.json"
done

Then read the acceptance counters from the server's Prometheus endpoint. Metric names starting with vllm:spec_decode carry draft and accepted token counts, and recent builds expose accepted tokens per position:

curl -s localhost:8000/metrics | grep '^vllm:spec_decode'

Compute accepted tokens per position as a conditional rate (position k accepted given k-1 accepted) before fitting any model. If conditional acceptance at position 6 is below 50%, a gamma of 7 is probably costing you at any load above light.

Two more rules from earlier posts in this series apply here. Record the effective configuration the engine reports rather than the one you intended, and count failed or timed-out requests as outcomes instead of dropping them from the denominator.

What is a practical tuning rule?

Compute B* for your hardware and context length with the closed form, using your measured weight-read time per step and measured per-token compute time.
Pick gamma so that B* for that gamma sits above your p95 concurrency per replica, then verify with the sweep.
If load swings widely, either route low-concurrency traffic to a speculative replica and high-concurrency traffic to a plain one, or check whether your engine can shorten or skip speculation dynamically under load. Do not assume it does; look for it in the release notes of your version.
Re-measure whenever the drafter, the model or the traffic mix changes. Acceptance is a property of the traffic, not of the model card.

FAQ

Does speculative decoding change the output of the model?
No. With rejection sampling, the output distribution matches ordinary decoding exactly, so quality is mathematically identical (Leviathan et al.; the vLLM Speculators post makes the same statement). What changes is latency and throughput.

Why does speculative decoding help less at high batch size?
At high batch size the GPU is already busy producing tokens for many sequences, so there is little idle compute to spend on verification. Each extra draft token then costs real compute time while the number of accepted tokens per iteration saturates.

What speculation length should I use in vLLM?
Start from the value in the drafter's model card (the vLLM example above uses 7 for DFlash), then sweep lower values under your real concurrency. In the illustrative model the best value fell from 8 at batch 1 to 3 at batch 128.

Are parallel drafters like DFlash and P-EAGLE better than EAGLE-3 under load?
They make drafting cheaper because one drafter pass covers the whole block, which helps when drafting latency matters. They do not change the verify pass, so the compute-bound threshold for the verifier is unchanged. Compare them with your own concurrency sweep instead of relying on a single-batch number.

Can speculative decoding make throughput worse?
Yes, when the verify pass is compute-bound and acceptance is low. In this model, a weak drafter (alpha of 0.6) with gamma = 7 falls below plain decoding at about 84 concurrent sequences under independent acceptance.

Sources

Leviathan, Kalman, Matias. Fast Inference from Transformers via Speculative Decoding
vLLM blog. Parallel All the Way Down: Beyond Single-Token Generation with Speculative Decoding (July 28, 2026)
vLLM blog. Exploring Speculative Decoding in vLLM on AMD GPUs (August 23, 2026)
NVIDIA. H100 Tensor Core GPU (peak bandwidth and compute figures)
All model outputs in this article are illustrative calculations from the stated assumptions, not measurements.

DEV Community: AI Tech News

Day-0 Model Support Is a Fork: What 26 NVIDIA Dynamo Snapshot Builds Say About Pinning Your LLM Serving Stack

TL;DR

What exactly did I measure, and where does the data come from?

How stale is the engine inside a Dynamo stable release?

Why does a stable pin turn into a stream of model snapshots?

What do those snapshot builds actually sit on?

Does the same model keep changing its base?

Where does this bite a multi-node deployment?

How do you keep a mixed fleet from silently breaking?

What are the limits of this analysis?

FAQ

Further reading

Sources

vLLM 0.33 Ships an ABI-Stable Wheel, but PyPI Still Pins torch==2.14.0: What You Can Actually Swap

TL;DR

What did vLLM actually change in 0.33?

What does the audit say about the wheel you can download?

If the binary is stable, why can't I just pip install a newer torch?

How long did the old model make you wait?

What should an operator do with this today?

FAQ

Further reading

Sources

Running an MoE Model Larger Than Your VRAM: SGLang Paged Experts, the PCIe Math, and the 4-Request Cliff

TL;DR

What does Paged Experts actually do?

How fast is it, and against what?

Why is it fast when experts keep crossing PCIe?

What happens with more than one request?

What does a resident slot cost in context length?

Why do short prompts get slower first tokens?

Can paging change the model's outputs?

A preflight script before you turn it on

A rollout checklist

FAQ

Further reading

Sources

vLLM 0.33 Adds a Lossy Speculative Verifier: Why Single-Run Accuracy Tables Cannot Tell You It Is Safe

TL;DR

What does FLy change in speculative verification?

What do the published speed numbers show?

Can the accuracy tables show that FLy is safe?

How many questions does a real quality gate need?

What is the bigger win in the same release?

What else in 0.33 touches a speculative serving stack?

A rollout checklist for FLy

FAQ

Further reading

Sources

vLLM Fast Start Explained: What Preload and Snapshots Save, What They Cannot, and the Fingerprint Gap

TL;DR

What did vLLM actually ship for fast restarts?

Which startup events does a faster restart actually help?

How much utilization does a faster start buy you?

Can the weight cache serve the wrong checkpoint?

A pre-flight checklist for adopting fast start

FAQ

Further reading

When Should an LLM Router Stop Being Sticky? llm-d 1.0 Cache-Affinity Routing, Break-Even Math, and a Simulation

TL;DR

What did llm-d 1.0 actually ship?

Why does cache-affinity routing hot-spot?

How does llm-d decide when to stop being sticky?

What throughput did the benchmark report?

What is the break-even point for stickiness?

What happens if you set the limit at break-even?

How do you calibrate this on your own fleet?

What should you not conclude from this?

FAQ

Further reading

When Is a KV Cache Slot Actually Free? Lessons From the SGLang and Cloudflare Cross-Request Exposure

TL;DR

What actually happened in the Cloudflare and SGLang disclosure?

Why can a KV slot be "free" and still get written to?

How likely is corruption if you free slots at abort time?

What does the fix hold, and for how long?

How long should the timeout be?

What does holding cost in capacity?

Which pre-flight check catches the configuration half?