TL;DR — LLM inference serving treats prefill and decode as one workload, but they have opposite resource profiles — prefill is compute-bound and parallel, decode is memory-bound and sequential. Continuous batching and quantization optimize around this mismatch instead of fixing it. Disaggregating prefill and decode onto separate pools is the actual structural fix, and it changes how you should think about latency, cost, and hardware allocation.
Every LLM inference request looks like one job: prompt in, tokens out. Under the hood it's two jobs wearing the same trench coat. Prefill reads your entire prompt and computes attention over it in parallel — it's compute-bound, it loves big batches, and a GPU chews through it at near-peak FLOPs. Decode generates one token at a time, autoregressively, each step depending on the last. It's memory-bandwidth-bound, it barely uses the GPU's compute, and it's the part of the request that actually determines how long your user stares at a spinner.
Most serving stacks run both phases on the same GPUs, in the same batches, scheduled by the same logic. That's the root cause of half the latency pathology teams fight with continuous batching, quantization tuning, and KV-cache sizing. You're not optimizing one workload. You're trying to find a single configuration that's good for two workloads with opposite resource signatures, and the compromise is worse than either optimum.
The request that isn't one workload
Think about what prefill actually does: it takes a few thousand tokens and runs one big forward pass, computing attention across the full sequence at once. This is embarrassingly parallel and matrix-multiply-heavy. A GPU doing prefill is close to its theoretical compute ceiling. More tokens in the prompt, more work, but it scales efficiently — this is why time-to-first-token for a short prompt is cheap and for a long one grows, but not catastrophically.
Decode is the opposite. Each step produces exactly one new token, which means one new row of activations, which means the GPU is moving enormous amounts of KV-cache memory in and out to compute a comparatively tiny amount of math. Decode is bandwidth-bound, not compute-bound. You can have an idle H100 sitting at single-digit compute utilization during decode because the bottleneck is how fast you can stream cached keys and values off HBM, not how fast you can multiply matrices.
These aren't two flavors of the same problem. They're different problems that happen to share a model and a weight set. Serving infrastructure that treats "a request" as the atomic unit of scheduling is optimizing the wrong grain.
Why continuous batching papers over the mismatch
Continuous batching — the technique behind most modern serving engines — solved a real problem: static batching left GPUs idle waiting for the slowest sequence in a batch to finish. By letting new requests join a batch mid-flight and evicting finished ones, you keep GPU occupancy high across a pool of concurrent decode steps. This is a genuine, important win.
But continuous batching is a decode-side optimization wearing a general-purpose label. When a new request arrives, its prefill has to happen somewhere, and in most implementations it happens inline, competing for the same GPU cycles as the decode steps of requests already in flight. A long prompt's prefill can stall the decode of a dozen other users who are mid-generation — this is the "prefill interference" effect that shows up as latency spikes with no obvious cause in your dashboards. You didn't get slower hardware. You got a big prompt that monopolized compute while everyone else's token stream stuttered.
Chunking prefill into smaller pieces and interleaving it with decode steps (a technique sometimes called chunked prefill) helps, but it's still a scheduling patch on top of a structural mismatch. You're time-slicing one resource between two workloads with different optimal batch sizes, different optimal precisions, and different sensitivity to latency.
Quantization doesn't fix an architecture problem
Quantization gets sold as the universal lever for inference cost: shrink the weights, shrink the KV-cache, fit more concurrent sequences per GPU, lower the bill. All true. But quantization mostly attacks the decode-side bottleneck — less memory to move means faster decode steps and a smaller cache footprint. It does comparatively little for prefill, which was never memory-bound in the first place; it was compute-bound, and compute doesn't shrink just because your weights got smaller precision.
So teams quantize aggressively, see decode throughput improve, and then wonder why time-to-first-token on long prompts barely moved, or why batching heterogeneous request lengths still produces unpredictable tail latency. The answer is that quantization optimized the half of the workload that was already the "easier" half to fix, and left the structural scheduling conflict between prefill and decode untouched. It's a real win stacked on top of an unsolved problem, not a replacement for solving it.
Disaggregation as the real lever
The structurally sound fix is to stop pretending prefill and decode belong on the same hardware at the same time. Disaggregated serving splits a cluster into a prefill pool and a decode pool. A request's prefill runs on compute-optimized nodes, tuned for large-batch parallel throughput, with no decode traffic competing for cycles. The resulting KV-cache is then transferred — over fast interconnect, not recomputed — to a decode pool tuned for memory bandwidth and high concurrency across many in-flight sequences.
This sounds like extra complexity for extra plumbing, and it is. You now need a cache-transfer path, you need to size two pools instead of one, and you need to route requests intelligently between them. But the payoff is that each pool can be tuned, scaled, and even hardware-selected independently. Prefill pools can run fewer, larger GPUs with high compute density. Decode pools can run more, cheaper memory-bandwidth-friendly hardware, sized for concurrency rather than raw FLOPs. You stop paying for compute you don't use during decode and memory bandwidth you don't need during prefill.
The economics shift in a specific way: your cost model goes from "GPU-hours per request" to "prefill-GPU-seconds plus decode-GPU-seconds plus cache-transfer-cost," and each of those three terms is independently optimizable. That's a better lever than tuning one global batch size and hoping it's a reasonable compromise for every request shape you'll ever see.
What this means for teams building serving stacks
If you're running your own inference infrastructure rather than calling a hosted API, the actionable takeaway is to stop treating "latency" as one number to chase. Time-to-first-token and inter-token latency are governed by different resources, and bundling them onto identical hardware guarantees you're always trading one off against the other. Measure them separately. Capacity-plan them separately. If your traffic mix has highly variable prompt lengths — long document contexts next to short chat turns — disaggregation isn't a nice-to-have, it's the only way to stop long prompts from degrading everyone else's decode latency.
Quantization, speculative decoding, and smarter batching heuristics are all still worth doing — they're real, additive wins inside each pool. But they're tactics. The strategic question is whether your serving architecture acknowledges that a single LLM request is secretly two workloads, or whether it's still pretending they're one and hoping the scheduler can smooth over the difference. It can't, not indefinitely. The mismatch is architectural, and architectural problems need architectural fixes.
Top comments (0)