On September 8, 2026, a security researcher told Cloudflare that very long prompts sent to one of its hosted models sometimes came back with text from other customers' sessions. Cloudflare reproduced the leak within a day, took the model offline forty minutes after declaring an incident, and restored it the same evening with a fix. Yesterday, October 26, they published the full postmortem alongside SGLang, the open-source inference engine involved. It is one of the most instructive infrastructure write-ups of the year, and the bug at its center is almost insultingly small: a one-token placeholder.
Here is what actually happened, why it happened, and what it should change about how you think about hosted inference.
The facts
Between August 27 and September 9, 2026, the GLM-5.3 model on Cloudflare Workers AI (@cf/zai-org/glm-5.3, a text model from Z.ai aimed at agentic coding) had a tenant isolation flaw. Prompts longer than 681,280 tokens could receive responses containing data from other customers' recent requests to the same model. In total, 155 responses were affected. Cloudflare says the timing and volume of the affected requests are consistent with ordinary long-context use, not with someone repeatedly exploiting the bug, and that they found no evidence of deliberate exploitation. No other Workers AI model was affected.
The report came in through HackerOne from Gadi Evron of Knostic. The timeline is worth reading in full, but the shape of it: report at 23:11 UTC on September 8, verification starting 08:00 the next morning, incident declared 13:22 after reproducing the leak with two independent accounts in 25 minutes, model set to private at 14:00, root cause identified at 14:35, corrected configuration deployed by 19:04, model public again at 19:40. Same-day root cause and remediation. Coordinated public disclosure six weeks later, after upstream fixes landed and shipped in SGLang v0.5.21 and v0.5.22.
Two bugs, one leak
The exposure needed both of these to exist at once.
Bug one: a configuration mismatch that everyone knew about. Workers AI runs GLM-5.3 in a disaggregated setup: prefill GPUs process the prompt and build the KV cache, then hand it to decode GPUs over RDMA (via Mooncake) that generate the response. The KV cache is a shared pool of fixed-size pages in GPU memory. Critically, the pool size is not a configuration value: SGLang sizes it at startup from whatever GPU memory remains after loading the model weights. The model's advertised context length is a separate value. Nothing forces the two to agree.
On the GLM-5.3 prefill nodes, the pool came out to 681,280 tokens while the configured context length was 1,048,576. Cloudflare was aware of the mismatch before launch. Their long-context tests returned successful HTTP status codes with streaming bodies and no error frames, so they assumed the engine handled the gap gracefully. It did not. The postmortem names this precisely: they validated response codes and performance at long contexts, but never validated the model's output and coherence. That is the testing gap that let a launch-day misconfiguration sit in production for two weeks.
Bug two: the engine's rejection path was not actually a rejection. In SGLang before v0.5.21, when a prompt arrived that was longer than the KV pool could hold, the scheduler rejected it. But the rejection path (Req.set_finish_with_abort) did not stop the request. It recorded the error for the client and swapped the prompt for a one-token placeholder, so the request could drain through the normal batch machinery without allocating real memory. In a single-node design that is mostly harmless. In disaggregated mode it was fatal: prefill generated the KV cache for the placeholder and transferred it to the decode node. The decode node received a "transfer complete" message and assumed prefill had overwritten the cache from previous requests with the full prompt. It had not. Pages are not zeroed between requests — zeroing would cost performance, so the next request simply overwrites what it is given — which meant most of the previous state was still sitting there. The decode node then generated a response while attending over stale cache pages holding other users' data.
Read that again: the request was already rejected. The error existed. The machinery just had no way to stop a rejected request from flowing downstream, and the downstream had no way to verify what it received.
The fix, in two layers
Cloudflare fixed the configuration and the engine separately, and the reasoning is the part worth stealing.
Layer one: make the pool hold the full context. They adjusted the GLM-5.3 configuration to match their GLM-5.2 setup, freeing enough GPU memory that the prefill KV pool now holds about 2.04 million tokens, roughly twice the context length. A prompt that fits the context window now always fits the pool, so the rejection path is never reached for a valid request.
Layer two: make the rejection path safe anyway. Their own words, paraphrased: relying on pool size alone is fragile, because quantization, tensor parallelism, CUDA graph sizes, or a future model version can silently change the pool size again. So they patched the engine so a request carrying a pending rejection is retired at the disaggregation admission queue, with its original error, before any bootstrap, allocation, or KV transfer happens — and had the decode node verify that the number of tokens transferred matches what it reserved. They proposed this upstream (PR #38704), found the fix already on SGLang main for a related path (#36988), and the maintainers merged a further early-rejection change (#38935). Both changes were validated against the researcher's reproduction script before the model went back online.
They also now check KV pool size against context length directly, from the running engine, before any disaggregated deployment change ships — instead of inferring safety from HTTP 200s.
This is a class of bug, not a bug
The postmortem is honest about something most vendors would bury: this belongs to a whole class of KV cache memory-safety issues Cloudflare has been chasing for months. A page gets freed, reused, or read while something else still holds a claim on it. They describe two common causes: page lifetime within a single node (token generation, offload, the KV transfer receiver, and abort paths can all hold the same page — their earlier fixes for decode offload and double release on grammar errors were this kind), and races across nodes (a request aborted while prefill is still writing its KV cache to decode over RDMA). Upstream work since mid-September includes the admission fix, per-transfer checksums, abort-acknowledgement preservation, and deferred KV release turned on by default.
If you run SGLang yourself — and plenty of self-hosted deployments do — the actionable line is simple: be on v0.5.21 or later, and treat any mismatch between your served context length and the engine's actual KV pool as a live vulnerability, not a performance footnote.
What this means if you are a customer, not an operator
Hosted inference is multi-tenant by design. Your prompt shares GPU memory pages with strangers' prompts, and isolation is a software property, not a physical one. Cloudflare did everything right after the report came in, and the write-up is a model of transparency — but the exposure existed from launch day, in a mismatch that was visible in startup logs and metrics the whole time. Two weeks, 155 responses, no zeroing between tenants.
Three takeaways I am keeping:
- A passing status code is not evidence of correctness. Cloudflare's own pre-launch tests green-lit a broken configuration because nobody checked whether the output made sense at long contexts. Whatever you are testing, validate the content, not the envelope.
- Rejected work must never reach downstream stages. An error path that lets the request keep flowing — even as a placeholder, even for good batching reasons — is an incident waiting for a second bug. The fix was not "make rejection impossible"; it was "make rejection actually stop the request."
- Configuration invariants need runtime checks. "Nothing forces the two to agree, and it is our job to check that they do" should be printed on a poster in every platform team. If two values must match, check them from the running system, on every deploy.
And if your threat model genuinely cannot tolerate cross-tenant exposure of any kind, the honest answer remains the one this series keeps coming back to: run the model on your own hardware, where there is no other tenant in the page pool. For everything else, this postmortem is a good reminder that "the provider handles isolation" is a claim to verify, not a default to assume.
Sources: Cloudflare blog, "Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang" (October 26, 2026) — https://blog.cloudflare.com/kv-cache-exposure-remediation/ — and the upstream SGLang pull requests referenced there (#38704, #36988, #38935).
Top comments (0)