When Is a KV Cache Slot Actually Free? Lessons From the SGLang and Cloudflare Cross-Request Exposure

TL;DR

• What was reported: On 2026-10-26, Cloudflare and the SGLang team disclosed that Cloudflare's GLM-5.3 deployment on Workers AI returned 155 responses containing data from other customers' requests between 2026-08-27 and 2026-09-09. Cloudflare says all 155 were answers to prompts longer than 681,280 tokens, and that no other Workers AI model was affected.
• Two bugs, one question: A configuration gap (prefill KV pool of 681,280 tokens against a 1,048,576-token context window) made an error path reachable. Separately, SGLang's disaggregated serving could free a decode-side KV slot while a launched RDMA write was still on its way to it. Both come down to: when is a slot really free?
• The invariant: A slot is safe to reuse only when no remote writer can still touch it. The SGLang write-up calls the abort race "a distributed use-after-free."
• The fix has a formula: hold time must exceed abort propagation + remaining chunks + the call already in flight. If the acknowledgement path dies, every abort waits out the full timeout, and held capacity is roughly abort rate x hold time x pages per request.
• What to do today: Check pool size against context length after every config change, run SGLang v0.5.21 or later for PD disaggregation, pair aborts at the router, and watch the release outcome metric. Scripts below.

What actually happened in the Cloudflare and SGLang disclosure?

Cloudflare's report describes a model that launched on 2026-08-27 with a configured context window of 1,048,576 tokens. The prefill KV cache pool could hold only 681,280 tokens. That is 65.0% of the window, a gap of 367,296 tokens or 35.0%. A request could therefore fit the context window and still exceed the memory available to process it. In Cloudflare's words, nothing forced the two numbers to agree.

On the engine side, the SGLang write-up says that in disaggregated prefill/decode serving, "a request marked for rejection could still proceed through the handoff. Decode could then read cache pages that had not been populated for that request." Cloudflare describes the same thing from the outside: the rejected request was replaced by a one-token placeholder, decode got a "transfer complete" message, and it assumed prefill had overwritten the old cache contents. Prefill had not, so stale pages from earlier requests survived.

The timeline, from the two primary documents:

Date (2026)	Event
Aug 27	Exposure window opens (GLM-5.3 launch on Workers AI)
Sep 9	Cloudflare disables public access to the model, notifies SGLang maintainers, restores service that evening after a larger pool and an engine patch
Sep 16	SGLang merges the intake admission fix (#38935) and optional KV transfer checksums (#39500)
Oct 2	v0.5.21: deferred KV release on by default for Mooncake and NIXL
Oct 3	Generation-tagged acks and MORI support (#37077), shipped in v0.5.22
Oct 26	Coordinated disclosure

One scope note matters for your risk assessment: the SGLang post says default PD configurations were affected from the v0.4.x line until v0.5.21 (v0.5.22 for MORI), and that the fix line was not backported to earlier release branches. A single-instance deployment without the cross-engine transfer path is not affected by the abort race.

Why can a KV slot be "free" and still get written to?

The decode engine pre-reserves destination slots for each incoming request, and its allocator marks them allocated or free. The disclosure's key observation is that "free" has a second, hidden condition. The allocator's bit says nothing about writes that were already posted to a NIC queue.

The failure takes four steps:

1. A client disconnect, admission rejection, or transfer fault aborts request A while at least one launched write has not landed.
2. Decode frees A's slots. The allocator marks them free.
3. The allocator hands those slots to request B.
4. A's late write lands inside B's KV cache. B decodes on foreign keys and values, with no error anywhere.

The write-up gives four facts that make step 2 to step 4 possible, and each is worth knowing if you run any disaggregated stack:

• The allocator can mark a slot free while writes are in flight. It and the NIC do not share state.
• Launched RDMA work cannot be cancelled. There is no call to cancel one work request already posted to the queue pair, and Python cannot interrupt a blocking C++ transfer call mid-flight. A pull design does not escape this, because a posted READ is as hard to take back as a posted WRITE.
• Router cancellation helps but cannot finish the job. An abort is only safe when both halves of a prefill/decode pair learn about it, yet whatever the transfer engine already launched continues regardless.
• The window scales with context length. A short prompt is in flight for milliseconds. A 128K to 1M-token context keeps writes in flight for seconds to minutes, so the same abort rate produces far more exposure on long-context traffic.

The corruption is silent by design. The transfer reports success and no component logs an error. A single contaminated slot derails a whole generation because attention reads every position. In the team's 2xH200 reproduction, the victim counted correctly up to 80 and then its output turned into the donor's prompt text.

The race is probabilistic in production, but a client can raise the odds by sending long-context requests and disconnecting mid-transfer. The client cannot choose the victim and learns nothing back, so the impact is one-way leakage plus an integrity problem.

How likely is corruption if you free slots at abort time?

I wrote a small Monte Carlo to see how the three ingredients interact. This is an illustrative model with stated assumptions, not a measurement of SGLang. The core of it is below; the assumptions are:

• KV size of 37.5 KB per token, derived from the one data point in the post: a single call for an 800K-token request carried 30 GB (taking GB as 10^9 bytes).
• 8,192-token chunks at an effective 25 GB/s, so one chunk write takes about 12.3 ms.
• Prefill at 20,000 tokens/s, so 128K tokens take 6.6 s and 1M tokens take 52.4 s.
• The allocator reuses freed slots after an exponentially distributed delay. The x-axis is the mean of that delay, standing in for memory pressure.
• Every abort lands mid-transfer, as the disclosure's conditions require.

import random

def p_contam_free_at_abort(tail_fn, mean_reuse, n=200_000):
    """Slots are freed at abort and reused after Exp(mean_reuse).
    Corruption if a launched write lands after the reuse."""
    bad = 0
    for _ in range(n):
        reuse = random.expovariate(1.0 / mean_reuse)
        bad += tail_fn() > reuse
    return bad / n

chunk_t = 8192 * 37_500 / 25e9                      # one chunk write, about 12.3 ms
tail_notify = lambda: chunk_t * random.random()     # abort reached prefill
tail_silent = lambda ctx: random.random() * ctx / 20_000   # prefill keeps pushing

Two readings follow. When the abort does not reach prefill, the tail is every chunk prefill still has to push, so corruption is near certain unless reuse takes seconds. Notifying prefill (SGLang's #27372) shrinks the tail to one in-flight chunk and collapses the curve, but at a 10 ms mean reuse delay my model still corrupts 43% of mid-transfer aborts. Notification narrows the window. Only holding the slots closes it. The disclosure frames it the same way: the router decides who dies, and the engine decides when memory is safe.

What does the fix hold, and for how long?

Decode cannot cancel prefill's launched writes, so it keeps the aborted request's slots reserved for longer than those writes can keep landing:

T_hold > t_abort_propagation + t_remaining_chunks + t_in_flight_call

The design makes every abort path hold the slots, then bounds each term on the right:

1. Notify prefill on abort (#27372). Decode sends ABORT to every prefill rank, and each rank drops chunks still waiting in its software queue. That drives t_remaining_chunks toward zero.
2. Hold, do not free (#35049, NIXL port #35360). On abort, decode holds the request's KV slots, its row in the request pool, and its metadata buffer entry. Each prefill rank counts outstanding work per room and replies ABORT_ACK only when that count drains to zero. Decode releases when every notified rank has acknowledged, or when a timeout fires. The default timeout in that PR is 30 seconds.
3. Gate by capability, then default on (#37454, #41023). The hold only helps on backends that acknowledge, so it shipped opt-in and became the default in v0.5.21.

The timeout is described as "a practical backstop, not a proof", because a call that times out can leave writes the engine is still retrying. That is why #43719 (merged 2026-10-12) makes prefill withhold the drain ack for any room whose send failed. Before it, a failed Mooncake send still acked as drained.

The design rule in the conclusion is worth copying: make every degradation lengthen the hold.

How long should the timeout be?

The post recommends 60 to 120 seconds for SGLANG_DISAGGREGATION_DEFERRED_DECODE_KV_RELEASE_TIMEOUT, and says a larger value "adds little safety and multiplies held KVCache during abort storms." I checked the shape of that claim with a second illustrative model: 5% of aborts take a degraded path where writes keep retrying past the 30 s transfer timeout (30 s plus an exponential with a 20 s mean), and the release is timeout-only.

Below the transfer timeout the hold is useless on the degraded path (about 500 per 10,000 corrupted, which is just the 5% degraded share). At 60 s it falls to 113, at 120 s to 6, and at 240 s to 0 in 200,000 trials. The tail decays exponentially under my assumption, so the second doubling buys almost nothing while it doubles the cost below. Measure your own retry distribution before copying a number.

What does holding cost in capacity?

The post gives the sizing rule: if releases fall back to the timeout, held capacity is about abort rate x T_hold x pages per request. With the healthy ack path, a hold is released milliseconds after the drain and costs nothing noticeable. With a dead ack path, every abort pays the full timeout.

Abort rate	T_hold	Held KV at 128K-token reservations (4.8 GB each)
1 per minute	30 s	2.4 GB
1 per minute	120 s	9.6 GB
6 per minute	30 s	14.4 GB
6 per minute	120 s	57.6 GB

At the 37.5 KB/token figure, one 800K-token reservation is 30 GB, so a handful of long-context aborts waiting out a 120 s hold can pin a large share of a decode pool. This is why the metric below matters as much as the fix itself: a dead ack path turns a safety feature into an admission-latency problem.

The SGLang team measured the steady-state price as small: across 21 runs on 2xH100 (1P1D Mooncake, 1024 requests at concurrency 256), throughput was statistically indistinguishable with the feature on or off, all deltas within plus or minus 2%. The hold only applies on the abort path.

Which pre-flight check catches the configuration half?

Cloudflare's first recommendation is to measure the KV pool against context length after every configuration change. That check needs no GPU. It needs two numbers from your engine: the token capacity of the prefill KV pool and the configured context length. Field names differ across versions, so read them from your engine's startup log or info endpoint and verify on your build.

import re

def pool_gate(pool_tokens: int, context_len: int, max_ctx_per_req: int | None = None) -> str:
    """Fail the deploy if a request that fits the context window cannot fit the prefill KV pool."""
    need = max_ctx_per_req or context_len
    if pool_tokens < need:
        short = need - pool_tokens
        return f"FAIL pool={pool_tokens} < context={need} (short by {short}, {short / need:.1%}); lower context_len or grow the pool"
    return f"OK pool={pool_tokens} covers context={need}"

print(pool_gate(681_280, 1_048_576))
# FAIL pool=681280 < context=1048576 (short by 367296, 35.0%); lower context_len or grow the pool

Make FAIL block the rollout. It would have flagged the reported configuration on day one.

How do you know the protection is actually on?

SGLang v0.5.22 ships a counter, sglang:decode_deferred_kv_release_total{outcome}. The post says a healthy deployment releases almost all holds through drained, while a timeout-heavy ratio means the ack path needs attention. Here is a parser and a verdict function; I ran it against the synthetic sample shown, whose numbers are made up.

PAT = re.compile(r'sglang:decode_deferred_kv_release_total\{([^}]*)\}\s+([0-9.eE+-]+)')

def release_outcomes(metrics_text: str) -> dict:
    out = {}
    for labels, val in PAT.findall(metrics_text):
        m = re.search(r'outcome="([^"]+)"', labels)
        if m:
            out[m.group(1)] = out.get(m.group(1), 0.0) + float(val)
    return out

def verdict(outcomes: dict, min_drained: float = 0.95) -> str:
    total = sum(outcomes.values())
    if total == 0:
        return "NO DATA: no holds yet, or the metric is missing on this build"
    drained = outcomes.get("drained", 0.0) / total
    return ("HEALTHY" if drained >= min_drained else "ACK PATH SUSPECT") + f" drained={drained:.1%} of {int(total)} holds"

sample = '''sglang:decode_deferred_kv_release_total{outcome="drained",tp_rank="0"} 9712
sglang:decode_deferred_kv_release_total{outcome="timeout",tp_rank="0"} 41
sglang:decode_deferred_kv_release_total{outcome="error",tp_rank="0"} 3'''
print(verdict(release_outcomes(sample)))
# HEALTHY drained=99.5% of 9756 holds

The 0.95 threshold is my choice, not SGLang's. Set yours from your own baseline after rollout.

Beyond the counter, the engine offers an opt-in integrity check: --disaggregation-enable-kv-checksum on both prefill and decode (their KV layouts must match). It computes an Adler-32 checksum on the GPU and verifies the received cache before the request joins a running batch. It is off by default because each transfer pays an extra GPU read and a synchronization point, and it catches only late writes that land before the commit point. The hold covers the rest.

Operators still on v0.5.19 or v0.5.20 are told to set SGLANG_DISAGGREGATION_DEFERRED_DECODE_KV_RELEASE=1 on both servers, plus SGLANG_MOONCAKE_MAX_TRANSFER_BATCH_INDICES on v0.5.20. The post is explicit that this does not replace upgrading, because failure kinds other than decode-initiated aborts stay unprotected before v0.5.21.

Does this apply to engines other than SGLang?

The write-up says the four steps apply to "any PD implementation that moves KVCache between engines with asynchronous DMA, push or pull." It does not say other engines are affected, and I have not tested any. Ask three questions of your own stack, whichever engine it runs:

• When a request dies mid-transfer, what delays the allocator's reuse of its destination slots?
• Is that delay tied to a positive signal that the writer finished, or only to a timer?
• If the signal path breaks, does the system get safer (longer hold) or less safe (earlier release)?

If you cannot answer these from the code or docs, you have found your next audit item.

FAQ

What exactly leaked in the Cloudflare incident?
Cloudflare reports 155 responses that contained data from other customers' requests, all to prompts longer than 681,280 tokens, between 2026-08-27 and 2026-09-09. Only GLM-5.3 on Workers AI was affected, per its investigation.

Is my single-node SGLang deployment affected?
The SGLang post says a single-instance deployment without the cross-engine KV transfer path is not affected by the abort race. The exposed path is PD disaggregation with Mooncake, NIXL, MORI, or a custom transfer backend on affected versions.

Which version should I run?
Per the post, v0.5.21 protects Mooncake and NIXL by default, v0.5.22 covers MORI, and the fix was not backported. Some press coverage says the fix landed earlier; I follow the engine team's own release mapping.

Can I tell whether I was exposed before upgrading?
Not reliably. The corruption leaves no error or log line, and the opt-in checksum only helps going forward.

Does a bigger timeout make it safer?
Only until it exceeds the longest realistic write tail. Past that it mostly increases held capacity during abort storms.

Are the charts measurements?
The cover uses the reported pool and window sizes. The other three come from my own simulation, so read them as shapes, not SGLang numbers.

Sources

• SGLang security update: improving request isolation (LMSYS, 2026-10-26)
• Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang (Cloudflare)
• SGLang pull requests cited: #38935, #39500, #32758, #37077
• SGLang releases: v0.5.21, v0.5.22
• Simulation assumptions: KV 37.5 KB/token (30 GB per 800K-token call, from #32758's tests as quoted in the post), 8,192-token chunks at 25 GB/s, prefill 20,000 tokens/s, exponential reuse delay, 5% degraded aborts with a 30 s timeout plus a 20 s-mean retry tail, 200,000 trials per point, seed 7.

Further reading

• Prefill/Decode Disaggregation in vLLM: The KV Transfer Math That Decides If It Pays Off
• Your LLM Serving Engine Ships Every 13 Days: Reading vLLM's Breaking-Change Lists as Upgrade Risk Data