TL;DR
- Cadence: In the 365 days to 2026-10-31, vLLM published 33 stable releases on PyPI, including 21 new minor series (0.12 through 0.32). The median gap between minor series was 13.5 days. SGLang published 19 releases, all inside 0.5.x (median gap 14.6 days). LMDeploy published 12 (median 26.5 days).
- Breaking-change volume: The vLLM release notes for 0.28 through 0.32 list 30 items under "Breaking changes". The per-release count rose every time: 4, 5, 6, 7, 8.
- The risky class: By my own classification, 8 of the 30 items (27%) change a default or a behavior while the exact same launch command still starts and passes a health check.
- Lag has a price too: Staying on 0.28 and jumping to 0.32 means reviewing 26 items at once, and 0.32.0 is the first of the five releases whose notes cite security advisories (three of them).
- What to do: Put an upgrade gate in front of every version bump: extract the breaking items automatically, diff the resolved server config, replay a golden request set. Working code for the first step is below.
- Scope: I did not run an inference engine for this post. Every number comes from the PyPI JSON API and the GitHub release notes, and I say where my judgment enters.
How often do LLM serving engines actually ship?
Most teams I read about treat a serving engine like a database: pick a version, pin it, upgrade "when needed". The release data says the engine moves much faster than that mental model.
Method: I pulled https://pypi.org/pypi/<package>/json, took the earliest upload time of any file for each version, and dropped release candidates, dev builds and post releases. The window is 2025-10-31 to 2026-10-31.
| Engine | Stable releases | Median gap (days) | Longest gap (days) | Latest in window |
|---|---|---|---|---|
| vLLM | 33 | 9.9 | 32.3 | 0.32.0 (2026-10-18) |
| SGLang | 19 | 14.6 | 41.7 | 0.5.23 (2026-10-30) |
| LMDeploy | 12 | 26.5 | 43.1 | 0.18.0 (2026-09-28) |
I left TensorRT-LLM out on purpose. Its PyPI listing shows 24 releases and a last upload of 1.2.1 on 2026-04-20, and I cannot tell whether PyPI is its primary channel. Guessing from a possibly incomplete source would put a number in this table that I cannot defend.
Of vLLM's 33 releases, 12 are patch releases. If you count only the first release of each minor series, you get 21 series with gaps of 15, 32, 9, 27, 9, 13, 13, 24, 17, 14, 14, 16, 11, 13, 16, 15, 13, 12, 13 and 13 days. The median is 13.5, the shortest is 9 and the longest is 32. The last four gaps (0.28 to 0.29 to 0.30 to 0.31 to 0.32) are 13, 12, 13 and 13 days, which is 52 days from 0.28 to 0.32.
Does the version number tell you how risky an upgrade is?
No, and the two big engines show why. vLLM is on 0.y.z, and the middle number moves with every scheduled release, so "0.31 to 0.32" is the normal unit of change. SGLang stayed in 0.5 for all 19 releases, so its normal unit is the patch number. A "patch bump" in one project is the same size of event as a "minor bump" in the other. Neither number carries a compatibility promise you can read off. The release notes are the only signal.
What did vLLM list as breaking changes from 0.28 to 0.32?
The GitHub notes for vLLM 0.28 through 0.32 each contain a bullet named "Breaking changes" with semicolon-separated entries. I counted entries. Two caveats: older releases label this differently (the 0.27.0 notes, for example, call the PyTorch upgrade "a breaking environment change" inside a highlight and have no dedicated bullet), so I used only the five releases with the dedicated bullet. And one entry can bundle several changes, such as "two environment variables removed", so the counts are a floor on what changed, not an exact tally.
| Release | Date | Items | Examples |
|---|---|---|---|
| 0.28.0 | 2026-08-26 | 4 | bitsandbytes moved to an out-of-tree plugin; calculate_kv_scales removed |
| 0.29.0 | 2026-09-09 | 5 | ten deprecated model architectures removed; PyAV video decoder backend removed |
| 0.30.0 | 2026-09-22 | 6 | scale-out endpoints now opt-in via --enable-scale-out; GPTQ g_idx removed |
| 0.31.0 | 2026-10-05 | 7 |
tokenizer_mode="slow" removed; --enforce-eager now also disables JIT kernel warmup |
| 0.32.0 | 2026-10-18 | 8 | Python 3.10 dropped; default CORS origins restricted; max_tokens no longer reserved from the context window |
I then sorted all 30 entries into four classes. This is my judgment, and a few entries could go either way.
| Class | Count | Examples from the notes |
|---|---|---|
| Dependency or Python pin | 3 | Python 3.10 dropped (0.32); transformers>=5.16.1,<5.20.0 (0.32); Transformers bumped to 5.15.0 (0.28) |
| Default or behavior changed | 8 | default CORS origins restricted (0.32); max_tokens no longer reserved from the context window (0.32); --enforce-eager also disables JIT warmup (0.31); YaRN aligned with Transformers so vendor aliases no longer re-scale max_model_len (0.30) |
| Removed or renamed flag, env var, feature, backend | 15 |
tokenizer_mode="slow" removed (0.31); --enable-mamba-fine-grained-prefix-cache renamed (0.31); calculate_kv_scales removed (0.28) |
| Deprecation or API rename | 4 |
python -m vllm.entrypoints.openai.api_server deprecated for vllm serve (0.29); Responses API reasoning stream events renamed (0.32) |
The four classes add up to 3 + 8 + 15 + 4 = 30.
Why are the behavior-change items the dangerous ones?
A removed flag usually announces itself: the deploy fails at launch, or your staging smoke test does. A removed environment variable is murkier, because plenty of programs ignore variables they do not recognize. Whether yours errors out is something to test on your exact version, not something to assume.
The 8 behavior items are different in kind. The command is identical. The process starts. The health endpoint returns 200. What changed is what the server does with the traffic. Four of them, read as an operator:
-
max_tokensno longer reserved from the context window (0.32). Capacity plans that assumed a longmax_tokensreserves context for every request may now behave differently. The notes do not say whether admitted concurrency goes up or down for your workload. I would replay traffic with realisticmax_tokensvalues and compare, rather than guess. -
--enforce-eagernow also disables JIT kernel warmup (0.31). If you run eager mode to save memory or to debug, I would expect first-request behavior to change. I have not measured it. Cold-start p99 is the number to compare. -
YaRN aligned with Transformers (0.30). If you serve a long-context checkpoint that relies on YaRN scaling, the effective
max_model_lencan differ from before. Compare the context length the server reports at startup, old version against new. - Default CORS origins restricted (0.32). This is a security improvement and the direction is right. It is still a behavior change that can break a browser client that talks to the server directly.
This is the same family of failure as the baked-in config trap I covered in an earlier post on KV cache dtype: identical flags, different conditions. The fix there was to compare the resolved configuration, not the command line.
What does falling behind cost?
Arithmetic from the same five release notes. Upgrading one release at a time, you review 5, 6, 7 and 8 items on the way from 0.28 to 0.32. Staying on 0.28 and jumping straight to 0.32, you review 5 + 6 + 7 + 8 = 26 items in a single change window, and you cannot bisect which one broke you because they all landed together.
The pinned option also has a cost on the security side. The 0.32.0 notes cite three GitHub security advisories as fixed: GHSA-4hhp-h66f-j5j7 (chat template resource exhaustion), GHSA-q97c-p368-75c4 (tokenizer max_token_id) and GHSA-frvg-qw88-hww9 (P2P ZMQ socket exhaustion). A text search for "GHSA" in the 0.28 to 0.31 bodies finds none. Maintainers can describe a fix without citing an ID, and the notes do not say whether anything is backported to older lines, so read this as "the three advisories I can see are in 0.32.0", not as proof that earlier versions were unsafe.
At a 13.5-day median gap, a "we upgrade quarterly" policy skips roughly 6 to 7 minor series per jump. Small, frequent, gated upgrades give you a smaller diff each time and let you pick up fixes sooner. That is an opinion, but it follows directly from the numbers above.
How do you build an upgrade gate?
Four layers, each aimed at a class from the table:
| Class | What catches it |
|---|---|
| Dependency or Python pin | Build the new image in CI, run pip check, boot it |
| Removed or renamed | Startup smoke test with the exact production arguments |
| Default or behavior changed | Resolved-config diff plus golden request replay |
| Deprecation or API rename | Grep startup logs for deprecation warnings, schedule the removal |
Step 1: extract the breaking items automatically
This script pulls the "Breaking changes" bullet for any release tags you give it. I ran it against v0.29.0 through v0.32.0 and it returned 5, 6, 7 and 8 items, 26 in total, which matches the counts above.
import json, re, sys, urllib.request
def breaking_items(tag, repo="vllm-project/vllm"):
url = f"https://api.github.com/repos/{repo}/releases/tags/{tag}"
body = json.load(urllib.request.urlopen(url, timeout=30))["body"] or ""
m = re.search(r"\* \*\*Breaking changes?\*\*:(.*?)\n", body)
return [s.strip() for s in m.group(1).split(";")] if m else []
if __name__ == "__main__":
total = 0
for tag in sys.argv[1:]:
items = breaking_items(tag)
total += len(items)
print(f"## {tag}: {len(items)} items")
for it in items:
print(" -", it)
print("total:", total)
Run it as python breaking.py v0.30.0 v0.31.0 v0.32.0. It depends on the notes keeping this bullet format, which held for 0.28 through 0.32 and did not hold for earlier releases, so treat an empty result as "go read the notes", not as "nothing broke".
Step 2: diff the resolved config, not the command line
Start the candidate version in staging with the exact production arguments. Save the startup log and the model-listing response from the old and the new server, then diff them. You are looking for any value you did not set that moved: context length, default dtypes, enabled endpoints, warmup behavior. This is what catches the YaRN and --enforce-eager style changes.
Step 3: replay a golden request set
The sketch below is illustrative. I did not run it here because it needs two live endpoints. It sends a fixed set of prompts at temperature 0 to the old and the new server and compares text, finish reason, token count and latency.
import json, statistics, time, urllib.request
def call(base, model, prompt, max_tokens=256):
body = json.dumps({"model": model, "temperature": 0, "max_tokens": max_tokens,
"messages": [{"role": "user", "content": prompt}]}).encode()
req = urllib.request.Request(base + "/v1/chat/completions", body,
{"Content-Type": "application/json"})
t0 = time.perf_counter()
r = json.load(urllib.request.urlopen(req, timeout=120))
c = r["choices"][0]
return (c["message"]["content"], c["finish_reason"],
r["usage"]["completion_tokens"], time.perf_counter() - t0)
def compare(old, new, model, prompts):
same, lat_old, lat_new, fin = 0, [], [], 0
for p in prompts:
a, b = call(old, model, p), call(new, model, p)
same += a[0] == b[0]
fin += a[1] != b[1]
lat_old.append(a[3]); lat_new.append(b[3])
n = len(prompts)
return {"identical_text": same / n, "finish_reason_changed": fin / n,
"p50_old": statistics.median(lat_old), "p50_new": statistics.median(lat_new)}
Temperature 0 does not guarantee identical text across engine versions, or even across batch compositions on the same version, because kernels and reduction orders change. So do not demand 100% identical text. Track the identical fraction as a trend, treat a jump in finish_reason_changed as a red flag, and score a task-level metric (exact match, pass rate) on the same prompts for the decision that matters.
Step 4: canary with a rollback condition
Send a small share of live traffic to the new version and write the rollback condition down before you start: error rate, p99 latency, and one quality proxy. If you cannot name the number that triggers rollback, you are not ready to canary.
What are the limits of this analysis?
- Release times are PyPI upload times, not GitHub tag times. The two can differ by hours.
- The breaking-change counts depend on how maintainers wrote one bullet. A longer bullet is not necessarily a bigger break.
- The four-way classification is mine. Reasonable people would move a few entries.
- I covered one engine's notes in depth. SGLang and LMDeploy appear only in the cadence table.
- Nothing here measures performance or accuracy. It measures how much changes, and how loudly.
FAQ
How often does vLLM release a new version?
Over the 365 days to 2026-10-31, PyPI shows 33 stable releases, including 21 new minor series. The median gap between minor series was 13.5 days, and the last four gaps were 13, 12, 13 and 13 days.
Does the version number tell me whether an upgrade is breaking?
Not in vLLM or SGLang. vLLM bumps the middle number every scheduled release, and SGLang stayed within 0.5.x for 19 releases. Read the release notes instead.
Which breaking changes are most dangerous for production?
The ones where your launch command still works but behavior differs: changed defaults, changed accounting such as how max_tokens is treated, changed context-length handling. In the five releases I classified, that was 8 of 30 items.
Should I pin an old version for stability?
Pinning is fine as a short-term choice, but it grows the eventual jump (26 items from 0.28 to 0.32) and delays fixes. The 0.32.0 notes list three security advisory fixes. A small upgrade every few weeks, behind a gate, is easier to debug than one big jump.
How do I find breaking changes without reading every note?
Parse the "Breaking changes" bullet with the script above, then read only the entries it returns. Empty output means the format changed, so read the full notes.
Does temperature 0 give identical output across engine versions?
No guarantee. Kernel and scheduling changes can alter floating-point reduction order. Compare identical-text rate as a trend and use a task metric for pass/fail.
Sources
- vllm on PyPI, sglang on PyPI, lmdeploy on PyPI
- vLLM release notes: v0.28.0, v0.29.0, v0.30.0, v0.31.0, v0.32.0
Top comments (1)
The class table is the useful part, and the 8 behavior-change items are the ones I'd build the gate around. A removed flag fails at the deploy event: something, somewhere, turns red. A changed default passes every signal tied to the event - the build is green, the process starts, health returns 200, the smoke test exercises the same path it always did. The failure only exists in the traffic shape, and production is the first place that has it.
That's what makes golden replay load-bearing and also its own trap. The replay verifies the behaviors the golden set happens to exercise. max_tokens reservation only shows up under realistic max_tokens values, cold-start regression only if the replay measures first-request latency. So the golden set isn't really a gate; it's a coverage claim, and it deserves the same discipline as the release notes themselves: versioned, enumerated, with the gaps named. Otherwise the gate quietly upgrades "we checked" into a stronger claim than what was actually checked.
The quarterly arithmetic is the part I'd tape to the wall. 26 unbisectable changes in one window is how teams end up saying "we can't upgrade, last time it broke everything" - which reads as caution and functions as a commitment to review all of them at once, later, under incident pressure.