Hello, I'm Shrijith Venkatramana. I'm building git-lrc, an AI code reviewer that runs on every commit. Star Us to help devs discover the project. Do give it a try and share your feedback for improving the product.
Prefill-Decode Disaggregation: Why LLM Inference Is Starting to Look Like a Distributed System
There is a strange thing about serving an LLM.
A user sends you 4,000 tokens and asks for 200 more.
The GPU spends one kind of computation processing those 4,000 tokens—and then spends the next 200 steps doing something fundamentally different.
Yet, in a conventional serving system, we often put both workloads on exactly the same GPUs, under the same scheduler, fighting for the same resources.
That is beginning to look increasingly unreasonable.
The idea of prefill-decode disaggregation is simple:
Run the part of inference that reads the prompt separately from the part that generates the answer.
This sounds like an obvious optimization once you see it. But getting it right involves KV caches, GPU memory, network bandwidth, batching, latency SLOs, scheduling, and some surprisingly interesting economics.
The idea was explored systematically in Microsoft's Splitwise work, published at ISCA 2024, and then pushed further for latency-sensitive serving by DistServe, published at OSDI 2024.
Let's build the idea from first principles.
1. An LLM request actually contains two very different jobs
Suppose you send this to a model:
Explain how TCP congestion control works.
Assume I already understand IP routing.
Give me a detailed explanation with examples.
Imagine that becomes 25 input tokens and the model generates 300 output tokens.
Inference looks roughly like this:
INPUT
|
v
+-----------------+
| PREFILL |
| |
| Process all |
| input tokens |
+--------+--------+
|
| KV cache
v
+-----------------+
| DECODE |
| |
| token -> token |
| -> token -> ... |
+--------+--------+
|
v
OUTPUT
Prefill
The model processes the entire prompt.
If the prompt contains 4,000 tokens, the transformer can process those tokens largely in parallel.
This is a large, dense computation.
GPUs are extremely good at this.
Decode
Now the model generates the answer one token at a time:
token 1
|
v
token 2
|
v
token 3
|
v
token 4
|
v
...
There is an unavoidable autoregressive dependency:
P(token_n | token_1 ... token_(n-1))
You cannot generally generate token 400 before knowing token 399.
So decode consists of many relatively small GPU operations.
This creates an important asymmetry:
Prefill wants compute throughput. Decode wants memory bandwidth and predictable latency.
Microsoft researchers Pratyush Patel, Esha Choukse, Chaojie Zhang and colleagues characterized exactly this difference in their Splitwise work. They found prompt processing to be compute-intensive while token generation was much more memory-intensive and underutilized expensive accelerator resources.
This is the fundamental observation behind disaggregation.
2. The surprising part: the same GPU is a bad compromise
Imagine a restaurant with two kinds of customers.
One customer arrives with 100 people and wants to place a large order.
Another customer arrives alone and wants one sandwich every 30 seconds for the next ten minutes.
If you force both groups through the same kitchen workflow, optimizing for one will hurt the other.
LLM serving has a similar problem.
Consider a decode batch:
request A -> generate next token
request B -> generate next token
request C -> generate next token
request D -> generate next token
...
Batching is extremely useful here.
The GPU can reuse the model weights across many requests, so the cost of reading the weights gets amortized.
Now a new request arrives with a 20,000-token prompt.
Its prefill is a large compute-heavy operation.
If the scheduler inserts that prefill into the same GPU workload, it can temporarily consume enormous amounts of compute and memory bandwidth.
The users already generating tokens don't care that the new request has a giant prompt.
They just experience:
token
token
token
<- giant prefill
...
token
Their inter-token latency spikes.
This is why TTFT and TPOT/ITL matter.
- TTFT — Time To First Token: how long the user waits before seeing anything.
- ITL — Inter-Token Latency: how long the user waits between generated tokens.
- TPOT — Time Per Output Token: closely related to ITL in serving measurements.
A system can have excellent average throughput while feeling terrible interactively.
DistServe's central argument was that prefill and decode interfere with one another strongly enough that treating them as one homogeneous workload makes it difficult to simultaneously satisfy TTFT and TPOT requirements.
3. What is actually being "disaggregated"?
Here is where the implementation gets interesting.
You don't split the neural network into:
GPU A:
layers 1-40
GPU B:
layers 41-80
That's ordinary model parallelism.
Instead, you run essentially the same model in two different pools:
Request
|
v
+----------------+
| Prefill Pool |
| |
| GPU GPU GPU |
+-------+--------+
|
| KV cache
|
v
+----------------+
| Decode Pool |
| |
| GPU GPU GPU |
+-------+--------+
|
v
Tokens
The prefill workers process the prompt.
They produce the intermediate state needed by the decoder: primarily the key/value (KV) cache.
That cache is transferred to a decode worker.
The decode worker then continues generation.
So the architecture becomes:
network / interconnect
----------------------------->
PREFILL GPU DECODE GPU
---------- ----------
prompt
|
v
transformer
|
v
KV cache ------------------------> KV cache
|
v
generate
token 1
|
token 2
|
token 3
This is why the technique is much more interesting than simply running two queues.
The KV cache becomes a distributed-system object.
And that changes everything.
4. The KV cache is the thing holding the whole system together
During attention, the model needs information about previous tokens.
Instead of recomputing the entire history for every generated token, serving systems store the relevant keys and values in the KV cache.
A simplified picture is:
Prompt:
A B C D E
KV cache:
K(A) V(A)
K(B) V(B)
K(C) V(C)
K(D) V(D)
K(E) V(E)
When generating F, the model can attend to this existing state.
Then it adds:
K(F) V(F)
and continues.
This cache can become enormous.
A rough formula for KV-cache memory is:
M_KV ~= 2 * L * H_KV * D * T * B
where:
L = number of transformer layers
H_KV = number of KV heads
D = head dimension
T = number of tokens
B = bytes per element
The factor 2 represents keys + values.
Consider a model with:
80 layers
8 KV heads
128-dimensional heads
BF16 KV cache = 2 bytes
4,000 tokens
Then:
M_KV
~= 2 * 80 * 8 * 128 * 4000 * 2
~= 1.31 billion bytes
or roughly 1.2 GiB per request.
At 32K tokens, the same request is roughly:
9.8 GiB of KV cache.
Now the distributed-systems problem becomes obvious.
Suppose you have to move 1.2 GiB from a prefill GPU to a decode GPU.
With a theoretical 400 Gb/s interconnect:
400 Gb/s = 50 GB/s
so the absolute bandwidth floor is approximately:
1.2 GB / 50 GB/s
~= 24 ms
That isn't necessarily the actual latency—you have protocol overhead, topology, serialization, synchronization, GPU copies, contention, and so on.
But it gives you the right intuition:
Disaggregation replaces GPU interference with a network-transfer problem.
That trade is worthwhile only when the interference you eliminate is more expensive than the communication you introduce.
This is why DistServe explicitly considers cluster bandwidth and KV-cache transfer when deciding where to place prefill and decode workers.
5. The math explains why decode is such a strange workload
Here's a useful back-of-the-envelope calculation.
Suppose you have a 70B-parameter model running in BF16.
The weights alone occupy roughly:
70B parameters * 2 bytes
~= 140 GB
A rough rule for transformer inference is around:
2 * parameters
FLOPs per token for the dense model.
So:
70B * 2
~= 140 GFLOPs/token
For a 4,000-token prompt, that gives approximately:
140 GFLOPs/token * 4000 tokens
= 560 TFLOPs
of model computation, ignoring architectural details and attention-specific terms.
That is a substantial matrix-multiplication workload.
Now consider decoding one token.
It's only around:
140 GFLOPs
Yet the model still needs to use its enormous parameter set.
This is why decode benefits enormously from batching.
Imagine 32 requests decoding simultaneously.
The same model weights can participate in the computation for 32 tokens:
Model weights
|
+----------------+----------------+
v v v
req A req B req C ...
Instead of paying the weight-access cost independently for each request.
This is one reason a decode GPU can look strangely underutilized in conventional compute-utilization metrics while still being the bottleneck for latency.
It isn't that the GPU has "nothing to do."
It is that its workload has a very different arithmetic intensity from prefill.
This distinction is central to Splitwise's economic argument: the latest, most expensive GPU is extraordinarily valuable for compute-heavy prefill, but its additional compute capability is much less valuable for decode.
6. Now the economics become interesting
Suppose you have two kinds of machines:
Prefill Decode
------- ------
Compute extremely useful less important
Memory bandwidth useful extremely important
Power high lower preferred
Cost high lower preferred
A conventional deployment might look like:
GPU GPU GPU GPU
| | | |
+---+---+---+
everything
Every GPU has to support both workloads.
But with disaggregation:
PREFILL DECODE
H100 H100 H100 A100 A100 A100
| |
+------- network -------+
you can independently scale the two pools.
Suppose your workload suddenly changes:
Before:
10,000 short prompts
500 output tokens each
After:
2,000 enormous prompts
500 output tokens each
The prefill workload has exploded.
You may want:
Prefill:
3 GPUs -> 8 GPUs
Decode:
4 GPUs -> 4 GPUs
With a monolithic serving architecture, you're effectively scaling both together.
With disaggregation, they become separate capacity-planning problems.
This is exactly the kind of optimization Splitwise explored.
The authors reported up to 1.4x higher throughput at 20% lower cost, or 2.35x more throughput under the same cost and power budgets, depending on the cluster configuration.
Those numbers are important because they demonstrate that this isn't merely a latency trick.
It can become a datacenter economics optimization.
7. But you don't always need disaggregation
This is perhaps the most important practical point.
If you're running:
7B model
8 concurrent users
2K context
single GPU
you probably shouldn't build a distributed prefill/decode architecture.
The complexity is ridiculous relative to the workload.
There are intermediate techniques.
One particularly important one is chunked prefill.
Instead of allowing a 20,000-token prompt to monopolize the GPU:
20K-token prefill
████████████████████████
you split it:
chunk 1
████
chunk 2
████
chunk 3
████
and schedule decode work between those chunks.
Some systems explored this approach, combining chunked prefills with decode-heavy batching. On their tested workloads, they reported substantial improvements, including up to 10x decode throughput for LLaMA-13B on an A6000 and 1.33x end-to-end throughput.
So there is a spectrum:
Simple
|
+-- Continuous batching
|
+-- Chunked prefill
|
+-- Better scheduling
|
+-- Prefill/decode disaggregation
|
v
Complex
The last step makes sense when the scale and latency requirements justify it.
Current vLLM documentation reflects exactly this distinction: its disaggregated-prefill implementation runs separate prefill and decode instances and transfers KV cache between them, while explicitly noting that disaggregation by itself does not necessarily improve raw throughput. Its major benefit is giving operators independent control over TTFT and inter-token latency and avoiding prefill-induced latency spikes.
That qualification matters.
Disaggregation isn't magic.
It gives you a better set of knobs.
Whether those knobs improve your system depends on your workload.
8. The deeper lesson: inference is becoming an operating-systems problem
The interesting thing about this research is that the model itself isn't changing.
The transformer is still:
attention
|
MLP
|
attention
|
MLP
|
...
What is changing is everything around it.
Once models become sufficiently expensive, inference starts looking less like:
"Run this neural network on a GPU."
and more like:
"Schedule heterogeneous computational phases across a distributed hardware system while managing a huge state object under latency SLOs."
That sounds remarkably like distributed systems.
You now have:
- queues
- admission control
- scheduling
- cache placement
- memory management
- network topology
- batching
- backpressure
- load balancing
- tail latency
- capacity planning
- heterogeneous hardware
- cost optimization
And the KV cache is effectively distributed state.
Splitwise asked: why not put the phases on different machines?
DistServe went further and treated TTFT and TPOT as separate SLOs, optimizing resource allocation and parallelism for each phase.
And the idea has since moved into real inference software: vLLM now exposes experimental disaggregated-prefill machinery based around separate prefill/decode instances and KV-cache transfer.
The trajectory is revealing.
We started by optimizing the neural network.
Then we optimized the GPU kernels.
Then batching and memory management.
Now we're increasingly optimizing the entire inference system.
That is probably the more important shift.
Conclusion: Don't think of an LLM request as one computation
The useful mental model is no longer:
request
|
v
LLM
|
v
response
Think:
REQUEST
|
v
+--------------+
| PREFILL |
| |
| compute-heavy|
+------+-------+
|
KV CACHE
|
-------+-------
network
-------+-------
|
+------+-------+
| DECODE |
| |
| memory-heavy |
| autoregressive|
+------+-------+
|
v
RESPONSE
Once you see those as two different workloads, a lot of otherwise strange behavior in LLM serving starts making sense.
Why does a long prompt suddenly destroy token latency?
Why does GPU utilization fail to tell you whether your inference system is healthy?
Why can adding GPUs fail to improve interactive latency?
Why might a cheaper GPU be perfectly adequate for part of an LLM workload?
Why does network bandwidth suddenly become an inference bottleneck?
And why are systems researchers talking about LLM serving in terms of queues, SLOs, cache placement and goodput rather than simply FLOPS?
Because the neural network is only one component of the system anymore.
The interesting question for developers is:
As LLM inference gets increasingly disaggregated, which other parts of the inference stack do you think will become independently schedulable next?
*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.
git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*
Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.
HexmosTech
/
git-lrc
Free, Micro AI Code Reviews That Run on Git Commit
| 🇩🇰 Dansk | 🇪🇸 Español | 🇮🇷 Farsi | 🇫🇮 Suomi | 🇯🇵 日本語 | 🇳🇴 Norsk | 🇵🇹 Português | 🇷🇺 Русский | 🇦🇱 Shqip | 🇨🇳 中文 | 🇮🇳 हिन्दी |
git-lrc
Free, Micro AI Code Reviews That Run on Commit
GenAI today is a race car without brakes. It accelerates fast -- you describe something, and large blocks of code appear instantly. But AI agents silently break things: they remove logic, relax constraints, introduce expensive cloud calls, leak credentials, and change behavior -- without telling you. You often find out in production.
git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.
In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen
At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…
Top comments (0)