An LLM request is two workloads in a trench coat — a heavy, bursty prefill and a stream of tiny latency-sensitive decodes. Running them on the same engines lets one big prompt stall everyone. Splitting them fixes it.
TL;DR: Every LLM request is two very different jobs. Prefill reads the whole prompt — heavy, bursty, and slow for long contexts. Decode then emits tokens one at a time — tiny, but latency-sensitive. Run them on the same engines and a big prefill jumps ahead of everyone's decodes: head-of-line blocking, and the token stream stutters. Prefill/decode disaggregation puts prefill and decode on separate pools so decodes never queue behind a prefill. In a runnable Go simulation, splitting the pools cut p99 inter-token latency by 66% (88ms → 30ms) — trading a little time-to-first-token for a far smoother stream. This is now standard practice in frontier serving stacks (DistServe, Splitwise, vLLM × Mooncake).
Mental model: a coffee shop with one worker who both grinds beans and pours espresso. A customer orders a giant batch grind and everyone waiting for a simple pour is stuck behind it. Split the shop into a grinder station and a pour station and the pours keep flowing no matter how big the grind.
The problem: two workloads, one queue
Serving an LLM token is not one kind of work — it's two:
- Prefill processes the entire prompt to build the KV cache. It's a big, compute-bound burst, and it scales with prompt length: a 100k-token context is a genuinely slow operation.
- Decode generates the response one token at a time, each step cheap but on the critical path of what the user feels — the inter-token latency (ITL) is the smoothness of the stream.
Colocate them — the default — and they fight for the same compute. Continuous batching helps throughput but doesn't remove the conflict: when the engine runs a long prefill, the decodes it's also hosting have to wait. And they can't flee to a less-busy engine, because this engine holds their KV cache. So one long-context prompt lands and every active token stream on that engine stutters. Your p99 ITL is hostage to your longest prompt.
The pattern: split the pools
Disaggregation (DistServe, Splitwise) separates the two phases onto different hardware:
- A prefill pool does nothing but build KV caches — bursty, compute-heavy work, isolated.
- A decode pool does nothing but stream tokens — steady, latency-sensitive work, isolated.
- The KV cache is handed off from prefill to decode (the expensive-to-move part — this is exactly what fast KV transfer layers like Mooncake exist to make cheap).
Now a giant prefill can't block anyone's decodes, because it physically runs on a different pool. Each pool can also be tuned and scaled independently for its own SLO (TTFT for prefill, ITL for decode) instead of compromising on one knob for both.
The simulation pins each request's decodes to the engine that ran its prefill (KV-cache locality) — the constraint that makes colocated blocking unavoidable:
if disaggregated {
if j.prefill {
server, dur = 0, w.prefill[j.req] // prefill pool
} else {
server, dur = 1, decodeDur // decode pool — never behind a prefill
}
} else {
server = j.req % 2 // pinned engine holds this request's KV cache
// ...its decodes are stuck behind whatever prefill lands here
}
The result
Prefill/Decode Disaggregation — keep long prefills from stalling the token stream
before → after: p99 inter-token latency 88ms (colocated) → 30ms (disaggregated) (66% lower)
240 requests, 2 servers, 20% long-context bursts (700–1800ms prefill), 20 decode steps × 5ms.
layout p99 token lat mean token lat mean TTFT
colocated (shared) 88 ms 15 ms 426 ms
disaggregated (split) 30 ms 11 ms 1032 ms
Same workload, same total hardware (2 engines each). Colocated lets bursty prefills jump ahead of tiny decodes, so the p99 token stutters to 88ms. Disaggregated isolates decodes and holds p99 to 30ms — a 66% smoother stream. The honest cost is TTFT: with only one engine dedicated to prefill, first tokens arrive later (426ms → 1032ms). That's the real knob disaggregation gives you — pool sizing lets you buy back TTFT by provisioning prefill and decode independently.
Reality check: these are simulated queue latencies, not GPU numbers — directional only. The direction is well-established: DistServe and Splitwise report multiples-higher goodput under latency SLOs from PD disaggregation, and the real-world win hinges on how cheap your KV-cache transfer is and how much prefill actually contends with decode.
Why this is where 2026 is heading
Disaggregation went from research idea to default in two years. DistServe showed that separating prefill and decode and sizing each for its own SLO can serve multiples more requests under latency constraints; Splitwise made the same case for splitting the phases across different hardware. By 2026 it's productized: vLLM ships PD disaggregation, and the vLLM × Mooncake work pairs it with a distributed KV cache pool so prefill and decode engines — even on different machines — share caches over fast transport. The load-bearing enabler is exactly the KV-cache handoff this demo hand-waves.
The transferable idea generalizes past LLMs: when one queue mixes bursty heavy work with steady latency-sensitive work, isolate them. It's the same instinct as separating batch from interactive traffic, or OLAP from OLTP — applied at the token level.
How faithful is this demo?
It models queueing and head-of-line blocking, not GPUs: "prefill" and "decode" are service times, and it ignores continuous batching, the real (non-zero) cost of KV-cache transfer, and memory pressure — all of which real systems must handle, and which is why cheap KV transport matters so much. Two honest caveats: disaggregation isn't free (you pay a transfer and a TTFT hop, visible above), and it only pays off when prefill bursts actually contend with latency-sensitive decodes. Short, uniform prompts under light load won't show the gap — measure your ITL tail before splitting.
When not to use this
- Small scale / a single GPU. With nothing to disaggregate across, the KV-transfer hop and the TTFT cost dominate any benefit.
- Short, uniform prompts. No bursty long prefills means no head-of-line blocking to fix.
- TTFT is your primary SLO. Disaggregation trades first-token latency for a smoother stream; if users care most about time-to-first-token, it can be a net loss.
Try it
go run . # standard library only
Sources & further reading
Papers
- Zhong et al. — DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (OSDI 2024, arXiv 2401.09670) — the case for splitting the phases and sizing each pool for its own latency target.
- Patel et al. — Splitwise: Efficient Generative LLM Inference Using Phase Splitting (ISCA 2024, arXiv 2311.18677) — splits prefill and decode across distinct machines to raise throughput per dollar and per watt.
- Qin et al. — Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving (FAST 2025, arXiv 2407.00079) — trades more storage for less compute; the KV-cache pool that makes cross-engine handoff practical.
Engineering
- Serving Agentic Workloads at Scale with vLLM × Mooncake (2026) — production PD disaggregation plus a distributed KV cache pool for multi-turn, agentic serving.
- KV Cache Offloading: LMCache vs Mooncake vs Dynamo — how the KV-transfer tier underneath disaggregation actually works.
Top comments (0)