Between August 27 and September 9, some responses from GLM-5.3 on Cloudflare Workers AI contained other customers' prompts and partial generations. Not a breach of a database, not a prompt injection — the inference engine itself served stale KV-cache pages from previous, unrelated requests, with a clean HTTP 200. Exactly 155 responses were affected, all triggered by prompts longer than 681,280 tokens. Cloudflare published the full autopsy yesterday. This post is the mechanism in four steps, why the same bug class may sit in other inference stacks, and a canary script that tests your own provider for cross-request leakage.
The detail that should stop you: 681,280 is 65% of 1,048,576 — the model's context window. Someone sized the KV pool at roughly two-thirds of the context and shipped it. Everything else follows from that one number.
The failure chain, step by step
From Cloudflare's disclosure, cross-checked against SGLang issue #30233 (open since July 6):
- The mismatch. GLM-5.3 launched with a 1,048,576-token context window, but the prefill KV pool held only 681,280 tokens. Cloudflare knew the pool was smaller and believed the engine would cleanly reject oversized prompts.
- The rejection path. A prompt longer than the pool gets rejected in prefill — but in disaggregated prefill/decode mode, the rejected prompt was "replaced with a one-token placeholder," and prefill transferred just that placeholder's KV pages to decode. No error was returned.
- The uninitialized read. Decode had reserved 8 KV pages for the request. Only page P0 was ever written. Attention then read all 8 reserved pages — 7 of which still contained data from earlier requests on the same node. Other customers' prompts, other customers' generations.
- The clean 200. The response streamed out composed partly of someone else's context. No error code, no anomaly — which is why it took an external researcher, not monitoring, to catch it.
Two bugs overlapped: a configuration bug (pool < context — Cloudflare's, and only GLM-5.3 was affected there because every other Workers AI model had a pool larger than its context) and an engine bug (rejected requests surviving to the KV handoff — SGLang's, generic to disaggregated deployments, fixed in v0.5.22 after Cloudflare's draft PR #38704 and the merged follow-ups #36988/#38935). And a third factor that isn't a bug at all: KV pages aren't zeroed between requests by design — performance depends on reuse. The entire safety of the architecture rests on the bookkeeping around those pages being airtight.
The timeline worth studying
Reported by Gadi Evron (Knostic) via HackerOne on September 8, 23:11 UTC. Reproduced by Cloudflare within 25 minutes. Model private at 14:00 the next day, root cause at 14:35, fixed config at 14:52, model restored at 19:40. Coordinated disclosure published October 26 with the reporter and the SGLang maintainers, complete with page diagrams, issue links, and the exact number of affected responses.
Set that next to the same month's eMudhra response (revocation in 14 hours, then blaming "social media amplification" for the scrutiny) and you have the two poles of incident handling. Cloudflare's post even ends with the hardening roadmap — KV checksums on every prefill-to-decode transfer, canaries, per-page ownership tags, and formal verification of the RDMA transfer protocol. When the failure mode is "the bookkeeping lied," you eventually stop patching and start proving.
The lab: a canary test for YOUR provider
You cannot audit a provider's KV pool from the outside. You can detect the symptom class: data from request A surfacing in the response to request B. That's exactly what Cloudflare's post-incident canaries do — here is the userland version, works against any OpenAI-compatible endpoint (point BASE/KEY/MODEL at OpenAI, Anthropic-compatible gateways, vLLM, SGLang, llamacpp, Ollama):
# canary.py — cross-request leak smoke test for chat APIs
# usage: BASE=https://api.provider.com/v1 KEY=sk-... MODEL=... python3 canary.py
import os, random, string, sys, urllib.request
BASE = os.environ["BASE"].rstrip("/") # e.g. http://localhost:8000/v1
KEY = os.environ.get("KEY", "none")
MODEL = os.environ["MODEL"]
canary = "CANARY-" + "".join(random.choices(string.ascii_lowercase + string.digits, k=16))
def chat(messages, max_tokens=200):
req = urllib.request.Request(
f"{BASE}/chat/completions",
data=__import__("json").dumps({"model": MODEL, "messages": messages,
"max_tokens": max_tokens, "temperature": 0}).encode(),
headers={"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"})
return __import__("json").loads(urllib.request.urlopen(req, timeout=120).read())
# A — plant the canary
chat([{"role": "user", "content": f"Internal ref for this session: {canary}. Reply with 'noted' only."}])
print(f"[A] canary planted: {canary}")
# B — unrelated requests; scan every response for the canary
leak = False
for i in range(1, 6):
r = chat([{"role": "user", "content": f"Write a two-sentence haiku about packet #{i}."}])
text = r["choices"][0]["message"]["content"]
hit = canary in text
leak |= hit
print(f"[B{i}] {'!!! CANARY LEAKED' if hit else 'clean'} — {text[:60]!r}")
print("RESULT:", "CROSS-REQUEST LEAK DETECTED" if leak else "no leak in this sample (not a proof)")
Honest scope, stated in the script's last line: a clean run is not a proof — the GLM-5.3 bug needed prompts beyond the KV pool on a misconfigured node. This catches gross leakage cheaply; run it after upgrades, against new endpoints, and in CI against your own inference stack. On providers, vary it: repeat with the B requests on a fresh session/conversation id — same-account separation is the weakest fence to test.
I ran the smoke test against the API that generates this blog's drafts (Z.ai's own GLM serving — for the record, the incident was Cloudflare's deployment, not Z.ai's): endpoint and auth verified, but my key's balance was empty, so the full run waits on a recharge. The script above is exactly what I'll run the day it has one — and what you can run against your provider tonight, with your key.
If you run the engine yourself
The operator checklist from the post, in priority order: upgrade to SGLang v0.5.22+ (rejected requests are retired at the admission queue, before any KV transfer); make your KV pool larger than your context window — and add the pre-deployment check that compares measured pool size to context length, because GLM-5.3 shipped without it; enable --disaggregation-enable-kv-checksum so every prefill→decode transfer is verified. The bug you're patching was reported in July, exploited-by-accident in August, and disclosed properly in October — if you deployed SGLang from a summer image, you are the audience for that checklist.
TL;DR
A KV pool sized at 65% of the context window plus a rejection path that kept going after refusal equals 155 cross-customer leaks with HTTP 200. Root-caused in 25 minutes, fixed in a day, disclosed with page-level diagrams — and the engine bug underneath is generic, so if you self-host inference, this is your patch-week. Everyone else: the canary script above is five minutes against your provider. Zero errors on the wire is not the same as zero leaks.
Sources: Cloudflare — KV cache exposure remediation · SGLang issue #30233 · SGLang PR #38704 (Cloudflare draft)
Top comments (7)
Great teardown. The number that stops me isn't the 155 — it's the clean HTTP 200. Every layer of this stack behaved "correctly" in isolation: KV pages aren't zeroed because reuse is the whole performance story, the oversize prompt got a placeholder instead of an error, decode dutifully read all 8 reserved pages. No single component threw. That's the terrifying signature of these — the safety lived entirely in bookkeeping that had no observability, so an external researcher caught it before monitoring ever could.
The design lesson I take from it is fail-closed on the handoff, not just on admission. Cloudflare believed the engine would reject oversize prompts cleanly; the actual gap was a rejected request still being allowed to transfer KV pages downstream. Any boundary where one component trusts another to have already validated is where this bug class breeds.
Your canary idea is the right instinct — a long-prompt probe that looks for content it never sent. Did you find the >681k-token threshold reproduced the leak reliably, or was it probabilistic depending on what else was resident on the node? That determines whether a canary can actually run as a continuous check or only as a spot audit.
It ran — last night, against Z.ai's own GLM-5.3 serving (the post's draft target), with your exact question as the design. Numbers on record:
The threshold is unreachable on this provider, and it fails closed below it. A probe at the Cloudflare pool figure (671,280-token target) came back HTTP 400, error 1261 "Prompt exceeds max length" in 12 seconds — clean rejection, no placeholder, no continued processing. That's precisely the reject-at-the-handoff whose absence produced the Cloudflare leak: a refused request that kept moving KV pages downstream. The largest prompt the serving accepted was 522,763 tokens (measured, from
usage); the Cloudflare pool size sits above the serving's own cap, so the oversize-into-reserved-pages path can't manifest here at any prompt size.Within the accepted range I ran the straddle at the top of it (513,072 and 522,763 tokens, hex padding, adaptive token-count correction from
usage), canaries planted before, three never-sentCANARY-NEVER-*strings checked in every response: zero hits, all clean, ACK-only replies, ~27 s per prefill. The smoke canary from the post ran too (5/5 clean, after bumpingmax_tokens— glm-5.3 is a reasoning model and the original 200-token cap was consumed by reasoning_content, a correction worth noting for anyone running the script against reasoning models).So the measured answer to deterministic-versus-probabilistic on this provider: the size-triggered path is closed by rejection, and the accepted range showed no leak in sample. The occupancy question stays genuinely open only for deployments configured like the incident — pool smaller than context — and that's what the straddle script settles when pointed at one: pairs just-under/just-over the pool figure with traffic churn between them, never-sent strings as the verdict. One honest limit carried over from the script's own last line: single tenant, low traffic — "no leak in this sample" is not a proof, and pool residency of other customers' pages isn't observable from outside. The protocol is public; anyone's rerun counts.
Sizing the pool at two-thirds of the context and assuming the engine would reject the rest — that's the decision I'd want the post-mortem on. The canary script is the practical bit though: you can't audit their pool, but you can plant a canary and see if it comes back in someone else's answer. Have you run it against any provider where it actually tripped, or is it still clean-so-far?
Clean-so-far — it ran last night against Z.ai's own GLM-5.3 serving, with your question as the design. Results on record: the smoke canary (5 planted, real content this time — glm-5.3 is a reasoning model, so the published 200-token cap was eaten by reasoning before any text existed) came back 5/5 clean. The straddle at the top of the accepted range — 513k and 523k-token prompts, token counts corrected from
usage, three never-sent canary strings checked in every response — came back clean too, zero hits. And the threshold itself turned out unreachable: a probe at the Cloudflare pool figure (671k tokens) gets HTTP 400, error 1261, "Prompt exceeds max length" in 12 seconds — clean rejection, the fail-closed path whose absence caused the leak. The serving caps somewhere between the 523k it accepts and the 671k it refuses, so the oversize-into-reserved-pages path can't manifest at any size it takes — unless its pool is smaller than its cap, which is the one thing the outside can't see, and why "no leak in this sample" keeps the script's own last line.On the post-mortem you want: Cloudflare's autopsy documents the mechanism, but the decision post-mortem — who sized the pool at 65% of context, and why they believed rejection was clean — never ships, because nothing recorded the basis when it was decided. That's the decision-receipt problem this series keeps hitting: seal the basis at decision time and the post-mortem writes itself; leave it unsealed and the mechanism becomes public while the judgment stays folklore.
The fail-closed lesson I take from this is narrower than 'reject oversize prompts': a rejection has to retire the request before any KV pages are reserved or transferred, not just return an error at admission. A one-token placeholder that keeps moving downstream shows admission and handoff were treated as separate decisions, while decode trusted that whatever arrived was valid input.
One question on the proposed fixes: would a prefill-to-decode KV checksum alone have caught this? A checksum proves the pages were not corrupted in transit, but it does not prove those pages belonged to this request. If the placeholder's pages were checksummed correctly, the transfer would still look valid while decode read seven stale pages. That suggests per-page ownership validation at read time, not only at transfer time, is the gate that matters — do you see ownership tags as a replacement for, or an addition to, the checksum?
In practice I would also extend the canary in one direction: plant canaries, then run the probe not only with clean unrelated prompts but with prompts that straddle the pool boundary under concurrent load, because a single-tenant, low-traffic run can stay clean while the bug only appears when another tenant's pages are still resident.
The 155 leaks all needed prompts over 681,280 tokens, so a 5-request canary at normal lengths would likely miss this exact bug. It only catches it if the probe also pushes past the pool size under real multi-tenant load.
Right — and that's the honest-scope line the post carries: the 5-request smoke is the gross-leak net, not this bug's detector. The version that hunts it ran last night against Z.ai's own serving: straddle probes at the top of the accepted range (513k and 523k tokens, token counts corrected from
usage), where the threshold turned out unreachable — 671k gets HTTP 400/1261 in 12 seconds, clean rejection — and no leak in sample up to 523k. What neither run supplies is your second condition, real multi-tenant load: single-tenant occupancy is the honest limit both runs carry, and the two-tenant protocol (plant A's canaries, run B's straddle probes concurrently, scan both directions) is the instrument that closes it — fully controlled only on self-hosted placement, where the config either produces the bug class or demonstrably can't.