DEV Community

Cover image for Why Split Prefill and Decode GPUs Lose Time in the Hand-Off, Not the Network
Mayur Agnani
Mayur Agnani

Posted on Edited on Originally published at theaicommit.com

Why Split Prefill and Decode GPUs Lose Time in the Hand-Off, Not the Network

Moving a long prompt's cache between GPUs takes about ten milliseconds. Yet the first token arrived more than a second later with the default hand-off. The time went to setup work, not the network.

Engineer's view. This is a Node.js server doing a synchronous parse on every upload. Each parse is quick, but it blocks the one loop every connection shares, so every response slows. With a fixed connection pool, slower responses then mean waiting for a free connection. The decode server's hand-off work behaves the same way.

TL;DR. When prompt processing and token generation run on separate GPUs, each request's cache must cross between them. The bytes are cheap. Setup work on the generation server's loop slows every token, and that can run the server out of slots.

Explain Like I'm 5

A bakery has one person mixing dough and another baking it. The mixer hands each bowl to the baker. Passing a bowl takes a second. But this baker stops all the ovens to write a label for every new bowl. Each label is quick. When bowls arrive fast, the ovens are stopped so often that every loaf bakes slower. Soon the oven shelves are full, and new bowls wait in line.

The Problem

You have shipped this bug before, in a system with no AI in it.

An API reads from a database through a pool of 20 connections. A release adds a small synchronous step to each query. Every query now holds its connection a little longer. At normal traffic nobody notices. At peak, all 20 connections are busy, and requests wait in line for one. The step costs milliseconds. The wait costs seconds. Nobody finds it by looking at the database, because the database is fine.

Disaggregated serving has the same bug. It splits an LLM server in two. Prefill GPUs read the whole prompt. Decode GPUs then write the answer one token at a time. Splitting them stops long prompts from stalling other users' answers. In between, each request's KV cache, the model's notes on the prompt, moves from one pool to the other.

Teams blame the network when the first token is slow. The numbers say otherwise. vLLM measured Qwen3-32B on two eight-GPU H200 servers. At 8 requests per second with 16,000-token prompts, the first token took about 1,744 ms with the default hand-off. A different hand-off cut it to about 540 ms. Moving that prompt's cache takes about 10 ms. How to Work Out How Much GPU Memory One Request Really Needs sized that cache.

The lost second is setup work. Part of it runs on the decode server's main loop, where it pauses every request in progress.

The fix: keep hand-off work off the decode loop, and size decode slots with headroom. Then the split pays for itself.

Read the full post on The AI Commit

Why Split Prefill and Decode GPUs Lose Time in the Hand-Off, Not the Network carries the rest: the fix and why it works, the code to change for each role, how to tell it worked, when it is the wrong tool, and an interactive model you can play with.


Originally published at theaicommit.com, a daily AI engineering lab for software engineers. The original has an interactive version you can play with, and the full runnable code is on GitHub.

Top comments (0)