TL;DR
NVIDIA Dynamo ships two kinds of releases: stable versions (v1.6.2 on 2026-11-05) and short-lived model snapshot builds such as v1.7.0-kimi-k3.1-pro-dev.1 (2026-11-06). I pulled every Dynamo release from the GitHub API, the engine release dates from PyPI, and classified what each snapshot build sits on. Everything below is reproducible with the script in this post.
- Stable pins are never current. Across 12 stable releases, the pinned vLLM was a median 30 days old on release day (range 17 to 56), and every single one had at least one newer vLLM minor already on PyPI. SGLang looked the same: 11 of 12 had a newer release available.
- Snapshots outnumber stables 2.6 to 1. Between 2026-07-17 and 2026-11-06 there were 10 stable releases and 26 model snapshot builds. All 26 carry a notice that they are not for production.
- Most snapshots are not on a plain engine release. Only 10 of 26 build on an unmodified released vLLM, SGLang or TensorRT-LLM tag. The other 16 use a patched release (3), a nightly or raw commit (5), or a model-specific image or fork (8).
- The fleet-level risk is documented, not hypothetical. One build states that its patched NIXL transfer protocol 14 is not compatible with unpatched protocol 13 workers. Another requires matching KV block size and speculative decoding settings on prefill and decode.
- What I did not do. I ran no GPU workload and deployed none of these images. This is a study of release metadata, with my own classification, and the snapshot builds are explicitly labeled as early access.
What exactly did I measure, and where does the data come from?
Three sources, all public. The Dynamo release list and notes come from the GitHub releases API for ai-dynamo/dynamo (73 entries as of 2026-11-09). Engine release dates come from PyPI upload timestamps for vllm and sglang. The snapshot classification comes from the header lines of each build's release notes, the ones that start with **vLLM**:, **SGLang**: or **TensorRT-LLM**:.
Two definitions matter. A stable release is any non-prerelease entry. A snapshot is any prerelease entry whose tag names a model, for example v1.6.0-deepseek-v4.1-flash-dev.2-post.1. The snapshot window is the 16 weeks from 2026-07-17 to 2026-11-06, the newest release on the day I ran this. Earlier builds often omit the engine header lines, so I left them out. The stable pin analysis uses every stable release whose notes include a pin table, 12 of them since 2026-04-23.
For stable releases, the notes include a "Key Dependencies" table listing the pinned SGLang, TensorRT-LLM and vLLM versions. I parse that table and compare the pinned engine's PyPI upload date with the Dynamo release timestamp.
import json, re, urllib.request
from collections import Counter
from datetime import datetime, timezone
def get(url):
req = urllib.request.Request(url, headers={"User-Agent": "audit/1.0"})
return json.load(urllib.request.urlopen(req, timeout=60))
rels = get("https://api.github.com/repos/ai-dynamo/dynamo/releases?per_page=100")
UTC = timezone.utc
when = lambda s: datetime.fromisoformat(s.replace("Z", "+00:00"))
def pypi_dates(pkg, pat):
out = {}
for v, files in get(f"https://pypi.org/pypi/{pkg}/json")["releases"].items():
if files and re.fullmatch(pat, v):
out[v] = min(datetime.fromisoformat(f["upload_time"]).replace(tzinfo=UTC) for f in files)
return out
vllm = pypi_dates("vllm", r"0\.\d+\.\d+")
sglang = pypi_dates("sglang", r"0\.5\.\d+(\.post\d+)?")
third = lambda v: int(v.split(".")[2])
# 1. How stale is the engine a Dynamo stable release pins, on release day?
ages = []
print("stable vLLM pin age newer | SGLang pin age newer")
for r in sorted(rels, key=lambda r: r["published_at"]):
if r["prerelease"] or "| :---" not in (r["body"] or ""):
continue
row = re.search(r"\| \*\*v([\d.]+)\*\* \| `v?([^`]+)` \| `v?([^`]+)` \| `v?([^`]+)`", r["body"])
if not row:
continue
tag, sg, trt, vl = row.groups()
t = when(r["published_at"])
vl_newer = len({v.split(".")[1] for v, d in vllm.items() if int(v.split(".")[1]) > int(vl.split(".")[1]) and d <= t})
sg_newer = sum(1 for v, d in sglang.items() if "post" not in v and third(v) > third(sg.split(".post")[0]) and d <= t)
ages.append((t - vllm[vl]).days)
print(f"v{tag:6} {vl:8} {(t - vllm[vl]).days:4}d {vl_newer:3} | {sg:13} {(t - sglang[sg]).days:4}d {sg_newer:3}")
ages.sort()
print(f"{len(ages)} stable releases, vLLM pin age: min {ages[0]}d, median {(ages[len(ages)//2 - 1] + ages[len(ages)//2]) / 2:g}d, max {ages[-1]}d")
# 2. What do the model-specific snapshot builds sit on?
def kind(line):
line = re.sub(r",? ?NIXL.*", "", line).lower()
if "ghcr.io" in line:
return "model-specific"
if "nightly" in line or re.search(r"build `[0-9a-f]{7,}`", line):
return "nightly or commit"
if "patch" in line or "backport" in line:
return "released + patches"
if re.search(r"(vllm-openai|sglang):v?\d|`v?0\.\d+\.\d+|`1\.\d+\.\d+rc", line):
return "released"
return "model-specific"
RANK = ["model-specific", "nightly or commit", "released + patches", "released"]
snaps = Counter()
for r in rels:
if not r["prerelease"] or r["published_at"] < "2026-07-17":
continue
lines = re.findall(r"^\*\*(?:vLLM|SGLang|TensorRT-LLM)\*\*:(.*)$", r["body"] or "", re.M)
kinds = [kind(l) for l in lines]
if kinds:
snaps[min(kinds, key=RANK.index)] += 1
print("\nsnapshots since 2026-07-17:", sum(snaps.values()))
for k in RANK:
print(f" {k:20} {snaps[k]}")
Output when I ran it on 2026-11-09:
stable vLLM pin age newer | SGLang pin age newer
v1.0.2 0.16.0 56d 3 | 0.5.9 58d 1
v1.1.0 0.19.0 31d 1 | 0.5.10.post1 25d 0
v1.2.0 0.20.1 29d 2 | 0.5.11 27d 1
v1.3.0 0.23.0 38d 2 | 0.5.14 26d 1
v1.4.0 0.26.0 21d 1 | 0.5.16 21d 1
v1.4.1 0.26.0 28d 1 | 0.5.16 28d 2
v1.4.2 0.26.0 34d 2 | 0.5.16 35d 2
v1.5.0 0.28.0 26d 1 | 0.5.18 30d 2
v1.5.1 0.28.0 41d 3 | 0.5.18 46d 3
v1.6.0 0.31.0 17d 1 | 0.5.21 21d 1
v1.6.1 0.31.0 22d 1 | 0.5.21 26d 1
v1.6.2 0.31.0 31d 2 | 0.5.21 35d 2
12 stable releases, vLLM pin age: min 17d, median 30d, max 56d
snapshots since 2026-07-17: 26
model-specific 8
nightly or commit 5
released + patches 3
released 10
The "newer" column counts distinct vLLM minor versions (or SGLang 0.5.N releases) newer than the pin that were already on PyPI at the moment Dynamo shipped.
How stale is the engine inside a Dynamo stable release?
Always somewhat, and it does not shrink. The vLLM pin was 17 days old at the freshest (v1.6.0) and 56 days old at the stalest (v1.0.2). The median across 12 releases is 30 days. The three most recent releases sit at 17, 22 and 31 days, but all three pin the same vLLM 0.31.0, so that growth is just the clock running between a minor release and its patches.
Take the newest stable as a concrete case. Dynamo v1.6.2 (2026-11-05) pins vLLM v0.31.0, SGLang v0.5.21 and TensorRT-LLM v1.3.0rc29. On that day PyPI already had vLLM 0.32.0 (2026-10-18) and 0.33.0 (2026-11-02), plus SGLang 0.5.22 and 0.5.23. TensorRT-LLM had already started its 1.4.0 release candidate line on 2026-10-22. None of that is a criticism. A stable release that bundles three engines, NIXL, UCX and Kubernetes recipes has to freeze something, and v1.6.2 says plainly that it is a patch release refreshing OS and Python packages.
The point is what the gap does to anyone who needs a model that shipped after the freeze.
Why does a stable pin turn into a stream of model snapshots?
My reading, and I flag it as interpretation rather than something the notes state: a brand new model usually needs engine changes (a kernel, a tool-call parser, a tokenizer path, a cache layout) that the pinned engine does not have yet. Waiting for the next stable means waiting out the pin lag above. So the project cuts a branch, bases it on whatever engine build already runs that model, and publishes it under a tag that carries the model name.
The volume shows how often this happens. In the 16 weeks I looked at, snapshots appeared in every month, and in September and October they outnumbered stable releases by a wide margin.
Every one of the 26 snapshot builds in the window carries a notice that it is not for production, and the ones I read in full also say it is not QA-gated. So this is not a story about a vendor shipping broken things. It is a story about where the bleeding edge of model support actually lives: on branches that are labeled as early access, built on engine bases that your normal upgrade process has never seen.
What do those snapshot builds actually sit on?
I classified each build by the least conventional engine line in its header, using this precedence: model-specific image or fork, then nightly or raw commit, then released tag with patches, then plain released tag. The classifier in the script above is a few regular expressions, and its totals matched my hand labels exactly (10, 3, 5 and 8).
| Base under the snapshot | Builds | Examples from the notes |
|---|---|---|
| Plain released engine tag | 10 |
v1.6.0-beam-1.0-501b-dev.1 on vLLM v0.31.0 and SGLang v0.5.21 |
| Released tag plus patches | 3 |
v1.4.1-a.x-k2-post.1 on vLLM 0.26.0 with four patches |
| Nightly or raw commit | 5 |
v1.7.0-mistral-large-4-dev.1 on a cu134-nightly vLLM image |
| Model-specific image or fork | 8 |
v1.7.0-kimi-k3.1-pro-dev.1 on vllm/vllm-openai:kimi-k31
|
Sixteen of 26 is 62 percent. Two caveats keep this honest. First, the labels come from header lines that different release authors fill in with different levels of detail, for example "MiniMax M3.1 Flash build" with no version at all, which I counted as model-specific. Second, a "plain released tag" build can still carry Dynamo-side patches in its runtime layer, which the header does not always say. Treat 62 percent as a rough measure of how often the base is non-standard, not an exact one.
Does the same model keep changing its base?
Yes, and the sequence is instructive. DeepSeek-V4.1-Flash has three Dynamo builds in the window:
| Build | Date | Engine base named in the notes |
|---|---|---|
v1.6.0-deepseek-v4.1-flash-dev.1 |
2026-09-12 | SGLang dev-dsv41 image, NIXL v1.4.0 |
v1.6.0-deepseek-v4.1-flash-dev.2 |
2026-10-08 | vLLM v0.31.0 image, NIXL v1.3.2 (connector protocol 14) |
v1.6.0-deepseek-v4.1-flash-dev.2-post.1 |
2026-11-05 | vLLM nightly ac9126e... dated 2026-10-01, NIXL v1.3.2 (protocol 14) |
The third build's notes say it returns to a pinned nightly instead of the v0.31.0 base used by dev.2. In other words, the project moved a model from a released engine to a nightly. The notes I read do not say why it moved back to a nightly. They list a fix so prefill health probes stop leaking KV leases and refreshed recipes, so I will not guess beyond that. The practical lesson is that "which engine version is this model on" can change between two builds with nearly identical names.
Kimi-K3 shows the opposite direction. Its first build (2026-07-27) used the model-specific image vllm/vllm-openai:kimi-k3. By 2026-09-09, 44 days later, the build used v0.28.0-ubuntu2404, a normal released tag, with SGLang on v0.5.17. Model-specific images look like bridges that later converge onto released engines, at least for this model.
Where does this bite a multi-node deployment?
At the seams between workers. Disaggregated serving moves KV cache from a prefill worker to a decode worker, so both ends must agree on the details. The release notes name several such constraints:
- The DeepSeek post.1 notes: the patched NIXL transfer protocol (14) is not compatible with unpatched protocol 13 workers, and all workers in a KV transfer must use that runtime.
- The Mistral Large 4 notes: disaggregated recipes require matching block size (64) and speculative decoding settings on prefill and decode, and the Vera Rubin recipes store KV in NVFP4, which changes numerics, so re-run an accuracy check after any model or image change.
- The Kimi-K3.1-Pro notes: this build relies on NIXL v1.5.0, which the notes say is required for Kimi strided descriptors.
NIXL itself is a moving part. Of the 26 snapshot builds, 20 name a NIXL version, and they name six different ones.
Even the stable v1.6.2 mixes NIXL versions by backend: v1.5.0 for the frontend and SGLang image, v1.3.1 for TensorRT-LLM and v1.4.1 for vLLM. That is a normal packaging decision when each backend image is tested on its own. It becomes a trap only if you assume one "Dynamo version" implies one transfer stack across every worker you run.
How do you keep a mixed fleet from silently breaking?
Five habits, none of which depend on Dynamo specifically.
-
Pin by digest, record the digest. Most snapshot notes give a branch tip and a base image, and several pin a sha256 digest. Copy that into your deployment record. A tag like
kimi-k31can move; a digest cannot. - Treat prefill and decode as one unit. If you upgrade one role, upgrade the group. Never leave a rollout half done across a KV transfer boundary.
- Gate on the fields the notes call out. Image digest, KV block size, speculative decoding config and transfer protocol version are the ones named above. Compare them across every worker in a group before traffic is allowed.
- Re-run accuracy after any image change that touches numerics. The Mistral notes say it directly for NVFP4 KV. Throughput numbers from a recipe do not tell you whether outputs moved.
- Keep experimental builds off the default path. Route them through a separate pool or namespace so a snapshot cannot become your production baseline by accident.
Here is a small gate for item 3. The manifest is illustrative, written for this post, not taken from any real deployment. One group has a deliberate mismatch.
import sys
from collections import defaultdict
MUST_MATCH = ("image_digest", "kv_block_size", "spec_decode", "nixl_protocol")
fleet = [ # illustrative manifest, not a real deployment
{"name": "p0", "group": "g1", "role": "prefill", "image_digest": "sha256:aa11", "kv_block_size": 64, "spec_decode": "off", "nixl_protocol": 14},
{"name": "d0", "group": "g1", "role": "decode", "image_digest": "sha256:aa11", "kv_block_size": 64, "spec_decode": "off", "nixl_protocol": 14},
{"name": "p1", "group": "g2", "role": "prefill", "image_digest": "sha256:aa11", "kv_block_size": 64, "spec_decode": "off", "nixl_protocol": 14},
{"name": "d1", "group": "g2", "role": "decode", "image_digest": "sha256:bb22", "kv_block_size": 32, "spec_decode": "off", "nixl_protocol": 13},
]
groups = defaultdict(list)
for w in fleet:
groups[w["group"]].append(w)
bad = 0
for g, workers in sorted(groups.items()):
for field in MUST_MATCH:
values = {w["name"]: w[field] for w in workers}
if len(set(values.values())) > 1:
bad += 1
print(f"FAIL group={g} field={field}: {values}")
print("OK" if not bad else f"{bad} mismatches")
sys.exit(1 if bad else 0)
FAIL group=g2 field=image_digest: {'p1': 'sha256:aa11', 'd1': 'sha256:bb22'}
FAIL group=g2 field=kv_block_size: {'p1': 64, 'd1': 32}
FAIL group=g2 field=nixl_protocol: {'p1': 14, 'd1': 13}
3 mismatches
It exits with status 1 on any mismatch, so it drops straight into a CI step or an admission check. Extend MUST_MATCH with whatever your engine's notes call out for your model.
What are the limits of this analysis?
It reads release notes, not behavior. I did not boot any of these images, so I cannot say whether a snapshot is faster, slower or more fragile than the stable build, and nothing here compares their performance. The classification depends on how consistently release authors fill in the header lines. The age calculation uses PyPI upload times, which can differ by hours from GitHub tag times. Finally, the window is 16 weeks long, so a different window could shift the percentages.
FAQ
Is it safe to run a Dynamo model snapshot build in production?
Every one of the 26 builds I checked carries a notice that it is not for production, and the notes I read in full call them experimental and not QA-gated. Use them for evaluation and early feedback, and keep them off your default serving path.
Why does Dynamo ship model-specific builds at all?
The notes do not give a general reason. The pattern in the data fits a simple explanation: new models need engine changes sooner than a stable release can absorb them, so a branch carries the needed base until the change lands in a released engine.
How old is the vLLM version pinned in a Dynamo stable release?
Across the 12 stable releases with a pin table, 17 to 56 days old on release day, median 30. Dynamo v1.6.2 pins vLLM 0.31.0, while 0.32.0 and 0.33.0 were already on PyPI.
Can I mix workers from different Dynamo builds in one disaggregated deployment?
Not safely without checking. The DeepSeek-V4.1-Flash post.1 notes say workers on the patched NIXL protocol 14 are not compatible with unpatched protocol 13 workers and that all workers in a KV transfer must use the same runtime.
How do I reproduce the numbers?
Run the script in this post. It needs only the Python standard library, one GitHub API call for the release list and two PyPI JSON calls. Unauthenticated GitHub access is enough for one run.
What should I record when I deploy a snapshot build?
The release tag, branch tip commit, base image digest per backend, NIXL version and transfer protocol, KV block size, and speculative decoding settings. Those are the fields the notes treat as compatibility-relevant.
Further reading
- vLLM 0.33 Ships an ABI-Stable Wheel, but PyPI Still Pins torch==2.14.0: What You Can Actually Swap
- Your LLM Serving Engine Ships Every 13 Days: Reading vLLM's Breaking-Change Lists as Upgrade Risk Data
Top comments (0)