TL;DR
- Prefill/decode (P/D) disaggregation puts the two phases of LLM inference on separate engine instances. vLLM's own docs are blunt about the trade: it lets you tune time to first token (TTFT) and inter-token latency (ITL) independently and controls tail ITL, but "disaggregated prefill does not improve throughput."
- The price you pay is moving the KV cache across a network for every request. That cost is a ratio you can compute before you buy a single NIC.
- For a 70B-class model with 8 KV heads, the KV cache is 320 KiB per token in BF16. Against an illustrative prefill rate, shipping it over a 400 Gb link costs about 7% of prefill time, over 100 GbE about 30%. FP8 KV halves both.
- Because both transfer and prefill scale linearly with prompt length, the ratio does not change with prompt length. What changes is how badly colocated prefill stalls your decode batch, and that grows with every long prompt.
- Decision rule at the end, plus a script to measure the thing that actually matters: p99 ITL under mixed load.
All throughput and latency figures below come from arithmetic on public model configs and stated assumptions. They are marked illustrative. Measure on your own hardware before you act.
What problem does disaggregated prefill actually solve?
LLM inference has two phases with opposite hardware personalities.
Prefill processes the whole prompt in one forward pass. It is compute bound: large matrix multiplies, high arithmetic intensity, GPU tensor cores busy.
Decode generates one token per step per sequence. It is memory bandwidth bound: every step reads the full weights and the growing KV cache to produce a single token per request.
When both run on the same engine, they interfere. A long prompt arriving mid-flight injects a compute-heavy step into a batch of sequences that are all waiting for their next token. Every one of those sequences sees an ITL spike equal to roughly the duration of the prefill step. Users experience this as a stream that stutters whenever someone else submits a big document.
Chunked prefill is the usual first mitigation: split the prompt into chunks and interleave them with decode steps, so each stall is bounded by the chunk size. P/D disaggregation is the stronger one: prefill runs on its own instances, decode instances never execute a prefill step, and the decode step cadence becomes steady. The vLLM documentation states that this reduces tail ITL more reliably than chunked prefilling.
Two other benefits follow from separation. You can give each phase a different parallelism strategy (for example, more tensor parallelism on the prefill side to cut TTFT), and you can scale the two pools independently when your traffic mix shifts.
What does it cost? The KV transfer arithmetic
Once prefill happens elsewhere, the resulting KV cache has to reach the decode instance before the first token can be produced. The bytes to move per token are:
kv_bytes_per_token = 2 * layers * kv_heads * head_dim * dtype_bytes
The factor 2 is for keys and values. I pulled the shape values from the public config.json files of two open models:
| Model | Layers | KV heads | Head dim | BF16 per token | FP8 per token |
|---|---|---|---|---|---|
| Llama-3.1-8B | 32 | 8 | 128 | 128 KiB | 64 KiB |
| Llama-3.1-70B | 80 | 8 | 128 | 320 KiB | 160 KiB |
Grouped-query attention is why these numbers are tolerable. A 70B model with full multi-head attention (64 KV heads) would carry 8 times more.
An 8,192-token prompt on the 70B model therefore produces 2.68 GB of KV cache in BF16 (8,192 x 327,680 bytes). That is the payload for one request.
How long does the transfer take compared with prefill?
Here is the calculator I used. It is deliberately small so you can swap in your own model shape, link speed, and measured prefill throughput.
def kv_bytes_per_token(layers, kv_heads, head_dim, dtype_bytes):
return 2 * layers * kv_heads * head_dim * dtype_bytes
def prefill_seconds(params_b, tokens, gpus, eff_tflops):
# dense-model approximation: 2 * params FLOPs per token, attention term ignored
return 2 * params_b * 1e9 * tokens / (gpus * eff_tflops * 1e12)
LINKS_GBPS = {"100GbE": 12.5, "200Gb": 25.0, "400Gb": 50.0} # GB/s at line rate
tokens = 8192
pf = prefill_seconds(70, tokens, gpus=4, eff_tflops=400) # assumption, replace with a measurement
size = tokens * kv_bytes_per_token(80, 8, 128, 2) # BF16 KV
for name, gbs in LINKS_GBPS.items():
t = size / (gbs * 1e9)
print(f"{name}: {t*1000:.0f} ms = {t/pf*100:.0f}% of prefill ({pf*1000:.0f} ms)")
The assumptions are the weak point, so I will state them plainly. Illustrative inputs: a 70B dense model, four GPUs sharing the prefill, 400 TFLOPS of effective throughput per GPU, link speeds at line rate with no protocol overhead. Real prefill throughput depends on your kernels, tensor parallel efficiency, and sequence length. Replace eff_tflops with your own measurement.
With those inputs:
| Prompt tokens | KV (BF16) | Prefill | 100GbE | 200Gb | 400Gb |
|---|---|---|---|---|---|
| 2,048 | 0.67 GB | 179 ms | 54 ms (30%) | 27 ms (15%) | 13 ms (7%) |
| 8,192 | 2.68 GB | 717 ms | 215 ms (30%) | 107 ms (15%) | 54 ms (7%) |
| 32,768 | 10.74 GB | 2,867 ms | 859 ms (30%) | 429 ms (15%) | 215 ms (7%) |
The percentage column is identical in every row. That is not a coincidence. Both quantities are linear in prompt length, so the ratio collapses to:
transfer / prefill = (kv_bytes_per_token / link_bandwidth) / (2 * params / (gpus * flops))
Prompt length cancels. The practical consequences:
- You can decide on hardware alone. Model shape, link bandwidth, and prefill throughput set the ratio. Traffic mix does not.
- Long prompts are, if anything, better for disaggregation than this table suggests. Real prefill includes an attention term that grows faster than linearly, so long prompts take relatively longer to compute than to ship.
- Tensor parallelism helps the transfer side too. KV heads are sharded across the prefill GPUs, so if each GPU has its own NIC path, the aggregate bandwidth scales with the group. The 50 GB/s figure above is a single 400 Gb link. Per-GPU links change the picture a lot, and it is worth checking your topology rather than assuming.
- FP8 KV cache pays twice. It halves the memory for concurrent sequences and halves the bytes on the wire. If you already quantize the KV cache, the transfer ratio gets cheaper for free. I covered how to measure that quantization fairly in an earlier post (linked below).
A caveat on overlap: transfer is not necessarily serialized after prefill. Implementations can start moving layers or blocks while later ones are still being computed, which hides part of the cost. The numbers here are the worst case where everything waits. The upside from overlap is real but depends on the connector and your network.
Why does colocated prefill hurt decode so much?
The other side of the ledger is what you are paying to avoid. A decode step on a loaded engine takes tens of milliseconds. A prefill step of 8,192 tokens, under the same illustrative assumptions, takes 717 ms. During that time no sequence in the decode batch advances.
Chunking shrinks the stall (179 ms at 2,048 tokens, 45 ms at 512) but never removes it, and small chunks have their own cost: more steps per prompt, lower prefill efficiency, and a longer TTFT for the prompt being chunked. Disaggregation removes the interference entirely on the decode side. What it adds is the transfer step, which in the table above is 4% to 30% of a prefill that no longer blocks anyone.
So the real comparison is not "transfer cost versus nothing." It is transfer cost versus the p99 ITL damage of colocated prefill at your actual request mix. If your workload is short chat turns, the damage is small and the transfer is pure overhead. If it is long documents, long RAG contexts, or agent traces with big tool outputs, the damage can dominate your tail latency.
How do you set it up in vLLM?
The feature is configured with --kv-transfer-config, a JSON string that selects a connector. The vLLM documentation lists several connectors, including NixlConnector, MooncakeConnector, LMCacheConnectorV1, and OffloadingConnector, and marks the feature as experimental and subject to change. A documented NixlConnector example looks like this:
vllm serve <model> \
--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both","kv_buffer_device":"cuda","kv_connector_extra_config":{"backends":["UCX","GDS"]}}'
Check the docs for your installed version for the exact role values per instance and for the proxy or router that sends the prompt to a prefill instance and then the decode instance. The interface has been moving.
It is moving in an interesting direction. In a recent post on the vLLM blog (October 14, 2026), the project describes a push-based NIXL connector. In the traditional pull model the decode instance requests KV from the prefill instance. In the push model the prefill instance writes KV directly into decode memory over RDMA. The stated goal is to remove three steps from the TTFT critical path: the router relay, decode-side allocation, and transfer setup. That matters for this article's math because those fixed costs are not in the bandwidth ratio above. At small prompt sizes, per-request setup can dominate the pure byte-transfer time, so the ratio table is a floor on overhead, not a ceiling.
How do you measure whether it helped?
Do not decide from average latency or from throughput. Disaggregation is a tail-latency tool, and the vLLM docs say it does not improve throughput, so a throughput benchmark will mislead you in both directions.
Measure p50 and p99 ITL while a background stream of long prompts is arriving. This client uses the OpenAI-compatible streaming endpoint and records the gap between consecutive chunks of each response:
import asyncio, json, time, httpx
async def stream_itl(client, url, model, prompt, max_tokens=256):
gaps, last = [], None
body = {"model": model, "prompt": prompt, "max_tokens": max_tokens,
"stream": True, "temperature": 0}
async with client.stream("POST", url, json=body, timeout=300) as r:
async for line in r.aiter_lines():
if not line.startswith("data: ") or line.endswith("[DONE]"):
continue
now = time.perf_counter()
if last is not None:
gaps.append(now - last)
last = now
return gaps
def pct(xs, q):
xs = sorted(xs)
return xs[min(len(xs) - 1, int(q * len(xs)))]
Run it with a fixed pool of "victim" short requests while a separate task injects long prompts at a constant rate, once against the colocated deployment and once against the disaggregated one. Report p50 and p99 ITL for the victims, plus TTFT for the long prompts.
Rules that keep the comparison fair, in the spirit of the benchmark traps I wrote about before:
- Same total GPUs on both sides. Splitting 8 GPUs into 4 prefill and 4 decode must be compared with 8 colocated, not with 4.
- Same arrival process and same prompt set, replayed from a file, not regenerated.
- Warm the prefix cache identically, or disable it, before every run. A reader pointed this out under my KV cache quantization post, and it applies here with more force. A warm shared cache flatters whichever configuration runs second, and in a P/D setup it also changes how much KV actually needs to be transferred.
- Dump the resolved engine config for both arms and diff them. Identical launch flags are not identical conditions.
- Run long enough to hit steady state, and report the offered load, not just the latency.
When should you use it, and when should you skip it?
| Situation | Verdict |
|---|---|
| Strict p99 ITL target, long or bursty prompts | Strong candidate |
| Different best parallelism for prefill and decode | Strong candidate |
| Fast interconnect between pools (RDMA, 200 Gb or more per path) | Favorable math |
| Short prompts, low concurrency | Skip, chunked prefill is enough |
| Offline batch jobs where only total throughput matters | Skip, no throughput gain |
| 100GbE or slower between pools and no FP8 KV | Check the ratio first, 30% overhead is hard to hide |
| Small cluster where splitting pools leaves each too small to batch well | Skip, utilization loss can exceed the gain |
The last row deserves emphasis. A decode pool needs enough concurrent sequences to fill its batches, and a prefill pool needs enough arrivals to stay busy. On a small cluster, splitting can leave both under-utilized. That cost does not show up in the transfer ratio at all.
FAQ
Does prefill/decode disaggregation improve throughput?
Not by itself. The vLLM documentation states that disaggregated prefill does not improve throughput. Its value is independent tuning of TTFT and ITL and tighter tail ITL control.
How big is the KV cache I have to transfer?
Use 2 * layers * kv_heads * head_dim * dtype_bytes per token. For Llama-3.1-70B in BF16 that is 320 KiB per token, about 2.68 GB for an 8,192-token prompt. FP8 KV halves it.
Is chunked prefill enough instead?
Often, for short prompts and moderate load. Chunking bounds the decode stall by chunk size but does not eliminate it. The vLLM docs say disaggregation controls tail ITL more reliably than chunked prefilling, at the cost of the KV transfer and a more complex deployment.
Which vLLM connector should I use?
The docs list several, including NixlConnector, MooncakeConnector, LMCacheConnectorV1, and OffloadingConnector. Choice depends on your interconnect and stack, and the feature is marked experimental, so pin your version and re-test on upgrade.
What is the difference between pull-based and push-based KV transfer?
In pull-based transfer the decode instance requests KV from the prefill instance. In push-based transfer the prefill instance writes KV directly into decode memory over RDMA, which the vLLM blog says removes the router relay, decode-side allocation, and transfer setup from the TTFT critical path.
Further reading
- Five Ways Your LLM Serving Benchmark Is Lying to You (and How to Catch Each One)
- KV Cache Quantization in LLM Serving: FP8 and INT8 Tradeoffs, the Silent config.json Trap, and How to Measure It Fairly
Sources: vLLM disaggregated prefilling docs, vLLM blog, push-based KV transfer (2026-10-14), Llama-3.1-70B config.json.
Top comments (0)