Mayur Agnani

Posted on Oct 23 Originally published at theaicommit.com

Why Split Prefill and Decode GPUs Lose Time in the Hand-Off, Not the Network

#ai #llm #performance #devops

Engineer's view. This is a Node.js server doing a synchronous parse on every upload. Each parse is quick, but it blocks the one loop every connection shares, so every response slows. With a fixed connection pool, slower responses then mean waiting for a free connection. The decode server's hand-off work behaves the same way.

TL;DR. When prompt processing and token generation run on separate GPUs, each request's cache must cross between them. The bytes are cheap. Setup work on the generation server's loop slows every token, and that can run the server out of slots.

Explain Like I'm 5

A bakery has one person mixing dough and another baking it. The mixer hands each bowl to the baker. Passing a bowl takes a second. But this baker stops all the ovens to write a label for every new bowl. Each label is quick. When bowls arrive fast, the ovens are stopped so often that every loaf bakes slower. Soon the oven shelves are full, and new bowls wait in line.

The Problem

You have shipped this bug before, in a system with no AI in it.

An API reads from a database through a pool of 20 connections. A release adds a small synchronous step to each query. Every query now holds its connection a little longer. At normal traffic nobody notices. At peak, all 20 connections are busy, and requests wait in line for one. The step costs milliseconds. The wait costs seconds. Nobody finds it by looking at the database, because the database is fine.

Disaggregated serving has the same bug. It splits an LLM server in two. Prefill GPUs read the whole prompt. Decode GPUs then write the answer one token at a time. Splitting them stops long prompts from stalling other users' answers. In between, each request's KV cache, the model's notes on the prompt, moves from one pool to the other.

Teams blame the network when the first token is slow. The numbers say otherwise. vLLM measured Qwen3-32B on two eight-GPU H200 servers. At 8 requests per second with 16,000-token prompts, the first token took about 1,744 ms with the default hand-off. A different hand-off cut it to about 540 ms. Moving that prompt's cache takes about 10 ms. How to Work Out How Much GPU Memory One Request Really Needs sized that cache.

The lost second is setup work. Part of it runs on the decode server's main loop, where it pauses every request in progress.

The fix: keep hand-off work off the decode loop, and size decode slots with headroom. Then the split pays for itself.

The Fix: Keep Hand-Off Work Off the Decode Loop

Why split prefill and decode at all?

Prefill is limited by arithmetic. Decode is limited by memory reads. On one server, a long prefill delays every answer being written. Splitting them lets you tune first-token time and per-token time separately. vLLM's own guide is blunt about the cost: disaggregation does not improve throughput. It buys predictable latency.

How much do the bytes actually cost?

Very little. Qwen3-32B stores 256 KiB per token. A 16,000-token prompt is 4.2 GB. With tensor parallelism across 8 GPUs, each GPU sends its own slice over its own links. At 50 GB/s per GPU, that takes about 10 ms.

Then where does the second go?

Into two places. First, pull runs its steps in series after prefill. The router relays where the blocks are. Decode allocates space. Then decode reads the blocks. Second, decode does that setup on its main loop. That loop also produces every token for every request. While it posts transfer work, nobody gets a token.

So the loop loses arrivals per second times setup time per request. A vLLM pull request measured 14.7 ms of posting per request at 2,000-token prompts. At 8 requests per second, the loop loses 12%. Moving that work to a thread took time per token from 6.14 ms to 5.52 ms.

Why does a slower token slow the first token?

Because a request holds a decode slot until its answer ends. Slots held equals arrivals times output tokens times time per token. That is Little's law. Past --max-num-seqs, new requests wait for a slot before their cache can land.

What does push change?

Decode registers its free blocks as soon as the request arrives. Prefill writes straight into them from a background thread. Nothing waits after prefill, and nothing runs on the decode loop.

What This Means for You

When this matters. You run prefill and decode on separate GPUs with vLLM, SGLang, Dynamo or llm-d. Your prompts are long, and your traffic is high.

How it affects you. Look at vLLM's 16,000-token run at 8 requests per second. Time per token rose from 7.0 ms with push to 10.4 ms with pull. With 128-token answers, that means 7.2 slots held with push and 10.6 with pull. The benchmark started its servers with --max-num-seqs 8. So pull asked for more slots than existed. That turns a few milliseconds per token into a second of waiting.

This is the same arithmetic as the connection pool. A small per-request cost decides nothing until the pool is nearly full. Then it decides everything. It also explains why the two modes match at low load. At 1 request per second, even pull holds barely one slot. The setup still happens, but nobody is waiting behind it.

What to do about it. Today, compute slots held for your own traffic at peak. Use your measured time per token, not the benchmark's. Keep it under about 80% of --max-num-seqs. Then move hand-off work off the decode loop, and re-measure. If you cannot change the hand-off this quarter, raise --max-num-seqs only if decode memory allows it. More slots need more KV cache space on every decode GPU.

Implementing It

The change.

Three roles touch this. Whoever plans capacity sizes the decode pool. Whoever owns the serving config picks the hand-off. Whoever runs the load test proves it.

Role 1, capacity. Compute the three numbers for your model and traffic. The first is usually small. The third is the one that bites:

def kv_bytes(tokens, layers, kv_heads, head_dim, dtype_bytes=2):
    return tokens * layers * 2 * kv_heads * head_dim * dtype_bytes

def transfer_ms(nbytes, link_gb_s, tp=1):
    return nbytes / tp / (link_gb_s * 1e9) * 1000   # each GPU sends its slice

def slots_held(qps, out_tokens, tpot_ms):
    return qps * out_tokens * tpot_ms / 1000         # Little's law

transfer_ms(kv_bytes(16_000, 64, 8, 128), 50, tp=8)  # 10.5 ms
slots_held(8, 128, 10.4)                             # 10.6, with 8 slots

Take tpot_ms from production at peak, not from a quiet benchmark. Use your real output length, too. Agents that write long answers hold slots far longer than chat does. A 2,000-token agent answer holds a slot sixteen times longer than a 128-token chat reply. So the same request rate needs sixteen times the slots.

Role 2, the serving config. The connector is a setting in --kv-transfer-config on both servers. The vLLM post ran push with this, on the prefill side:

vllm serve Qwen/Qwen3-32B --tensor-parallel-size 8 --max-num-seqs 8 --port 8100 \
  --kv-transfer-config '{"kv_connector":"NixlPushConnector","kv_buffer_device":"cuda",
    "kv_role":"kv_producer","engine_id":"prefill-engine-001","kv_rank":0,
    "kv_parallel_size":2,"kv_connector_extra_config":{"backends":["LIBFABRIC"]}}'

The decode server uses the same connector with "kv_role":"kv_consumer", as in vLLM's NIXL guide. Mixing modes crashes the server after the first request. Push also needs a proxy that knows the prefill server's address up front. The post uses disagg_proxy_pushconnector_demo.py. Pull stays the right choice if you route each request to the least-loaded decode server after prefill. Push picks decode early, so it gives that up.

Role 3, measuring. Sweep request rate and prompt length together, because the gap only opens at the top of both:

for qps in 1 2 4 8; do for len in 1024 16384; do
  vllm bench serve --model Qwen/Qwen3-32B --request-rate $qps \
    --random-input-len $len --random-output-len 128 --num-prompts 400
done; done

How you know it worked. Push and pull should match at low rate and short prompts. At the top of the sweep, time per token should stay near its light-load value. Watch vllm:num_requests_waiting on the decode server. It should stay near zero at your peak rate. If it climbs, you are out of slots, whatever the hand-off. Then split first-token time into its parts. Compare the prefill server's own first-token time with the proxy's. The difference is the hand-off plus any wait for a slot. After the change, that difference should shrink most at long prompts and high rates.

When Splitting Prefill and Decode Is the Wrong Tool

When you want more throughput. Splitting does not add any. It trades GPUs for steadier latency.

When your prompts are short. Prefill then barely disturbs decode, so there is little interference to remove. Chunked prefill on one server may be enough.

When you have one GPU node. The hand-off then crosses no network, but you still pay for two pools and a proxy.

When you need load-aware decode routing. Push chooses the decode server before prefill finishes. If your decode servers run unevenly, pull's late choice may be worth its setup cost.

When the decode pool is already full. No hand-off fixes too few slots. Add decode capacity first, then tune the hand-off.

Before you adopt it, ask:

How many decode slots does my peak traffic hold, by Little's law?
Is my first-token time spent in prefill, in the hand-off, or in a queue?
Do I need to route decode after prefill, or can I choose early?

Glossary

Disaggregated serving — running prompt reading and answer writing on separate GPU pools.
Prefill — the step that reads the whole prompt and builds its cache.
Decode — the step that writes the answer, one token at a time.
KV cache — the model's stored notes on every token so far, kept on the GPU.
Little's law — items in a system equal arrival rate times time each one stays.
slot — one of the requests a decode server can generate for at once.

DEV Community