If you have used any large language model product in the last year, you have run into a piece of infrastructure most people never think about: the serving system that turns a prompt into a stream of tokens. For a long time, that system worked the same way regardless of scale. One GPU, or one small cluster of GPUs, would take your prompt, process it, and then generate the response token by token, all on the same hardware, in the same process.
That approach is quietly breaking down. Context windows have stretched past 100,000 tokens, request volumes have grown by orders of magnitude, and the cost of running inference has become a board level conversation at most AI companies. The fix that has emerged, and is now running in production at some of the largest LLM providers in the world, is called disaggregated prefill/decode serving. It sounds like a narrow infrastructure detail, but it is one of the more interesting distributed systems problems in production today, and it has not been written about nearly as much as caching or load balancing.
This post explains what the problem actually is, how the architecture works, how real systems like Mooncake and NVIDIA Dynamo implement it, and when it is genuinely worth adopting versus when it is just extra operational burden.
Why prefill and decode do not want to live together
Every LLM inference request has two distinct phases.
Prefill is when the model reads your entire prompt and builds up its internal representation of it, layer by layer, for every token you sent in. This step is compute bound. It uses the GPU's raw math throughput heavily, and it happens once per request, regardless of how long the eventual response is.
Decode is what happens after that: the model generates the response one token at a time, each new token depending on everything that came before it. This step is memory bandwidth bound. The GPU spends most of its time moving the key value cache (the KV cache, which stores the model's running memory of the conversation) in and out of memory rather than doing heavy computation.
These two phases have almost opposite hardware profiles. Prefill wants raw FLOPs. Decode wants memory bandwidth and low latency access to the KV cache. When both run on the same GPU in the same batch, they compete for the same resources, and one interferes with the other. A long prompt arriving for prefill can stall the token generation of requests that are already mid decode, which shows up to the end user as a sudden latency spike.
Academic work on this problem, including the DistServe and Splitwise papers, and production systems like Mooncake, converge on the same conclusion: separate the two phases onto different hardware, and let each one be scheduled and scaled independently.
The core architecture
At a high level, disaggregated serving introduces a new component that does not exist in a monolithic setup: a layer that moves the KV cache from the machines doing prefill to the machines doing decode.
The router (sometimes called a scheduler, sometimes a conductor, depending on the system) decides which prefill instance and which decode instance should handle a given request. It is trying to balance several things at once: current load on each pool, whether part of the KV cache for this request already exists somewhere in the cluster from a previous turn, and the latency target for this particular request.
Once a prefill instance finishes processing the prompt, it does not throw away its work. It streams the resulting KV cache, often layer by layer as it is produced rather than waiting for the whole thing to finish, over to the assigned decode instance. The decode instance loads that cache and begins generating tokens, which get streamed back to the client as they are produced.
Here is what a single request actually experiences as it moves through the system.
The hard part: moving the KV cache fast enough
The entire architecture only works if the KV cache transfer is fast and does not become its own bottleneck. If moving the cache takes longer than the time saved by separating the phases, disaggregation is pointless.
This is why the transfer layer has become its own area of serious engineering effort. In 2026, most production systems rely on RDMA (remote direct memory access) over InfiniBand or RoCE networking to move KV cache data directly between GPU memory pools with very low latency, falling back to plain TCP when that hardware is not available. NVIDIA's NIXL and Moonshot AI's Transfer Engine (the foundation of Mooncake) are the two most widely used implementations of this idea, and both vLLM and SGLang have built native support for them.
The amount of data being moved is not trivial either. For a rough sense of scale, the size of the KV cache for a single request scales with the number of layers in the model, the number of attention heads, the size of each head, and the length of the sequence. For a long conversation or a large document being summarized, this can be gigabytes of data that needs to move between machines before the first output token is even generated, which is why the transfer engine, not just the model itself, has become a serious point of optimization.
Case study: Mooncake, the platform behind Kimi
The clearest public example of this architecture running at real scale is Mooncake, the serving platform Moonshot AI built for its Kimi chatbot. Mooncake was one of the first systems to fully commit to a KV cache centric design rather than treating the cache as a side effect of inference.
At the center of Mooncake sits a global scheduler called the Conductor. For every incoming request, the Conductor decides which prefill and decode instance pair should handle it, taking into account how much of the relevant KV cache already exists somewhere in the cluster and can be reused rather than recomputed from scratch. Mooncake also does something unusual: it uses otherwise idle CPU memory, DRAM, and even SSD storage across the GPU cluster to build a much larger, distributed KV cache pool than would fit in GPU memory alone.
The results reported by the team are substantial. In their published research, Mooncake increased effective request capacity by roughly 60 to nearly 500 percent over baseline approaches in various tested scenarios, while still meeting latency service level objectives, with the biggest gains showing up in long context workloads where the cost of recomputing versus reusing cache matters most. As of their most recent public updates, the system is running across thousands of nodes and processing over 100 billion tokens per day in production, not a lab benchmark.
One detail that is easy to miss: Mooncake does not assume every request should be processed no matter what. Under heavy load, it uses a prediction based early rejection policy, essentially deciding as early as possible that a request cannot realistically meet its latency target and rejecting it immediately, rather than letting it consume GPU resources partway through and then failing anyway. That is a meaningfully different design philosophy from most traditional web backends, where the instinct is usually to queue and retry rather than reject early.
The rest of the ecosystem is converging on the same idea
Mooncake is not an isolated experiment. By 2026, prefill/decode disaggregation has effectively become the industry default rather than a niche optimization.
vLLM, one of the most widely used open source inference engines, supports disaggregated serving natively, using NIXL as its standard transfer mechanism alongside NVIDIA Dynamo. SGLang offers a similar setup through its own router and disaggregation mode, and its RadixAttention design gives it a particular advantage for workloads with heavy prompt reuse. LMDeploy has added support through a comparable mechanism. NVIDIA's own Dynamo platform builds a routing and orchestration layer on top of this pattern for large scale deployments.
Perhaps the clearest sign that this has moved from research to infrastructure standard is llm-d, a project from Red Hat, IBM, and Google that brings Kubernetes native disaggregated inference on top of vLLM. It was donated to the CNCF in early 2026, and its default configuration reportedly delivers a meaningful performance improvement with no manual tuning required, with later releases showing significantly lower per token latency for large models on modern GPU hardware.
Production adoption backs this up. Reports throughout 2026 point to Meta, LinkedIn, Mistral, and Hugging Face running vLLM with disaggregated serving in production, and DeepSeek and Google's Gemini team have both discussed similar architectures at scale. Even at the hardware level, the idea is expanding. AWS partnered with Cerebras this year to bring disaggregated inference to Amazon Bedrock, using one type of chip for prompt processing and an entirely different type of chip for token generation, running side by side in the same cloud region.
Pushing disaggregation further: splitting more than just prefill and decode
Once you accept the idea that different parts of inference have different resource profiles, it is natural to ask whether the split should go deeper than just prefill versus decode. Some of the newest research does exactly that.
ByteDance's MegaScale Infer system takes disaggregation a step further by splitting the attention computation from the feedforward network (FFN) computation within the decode phase itself, since these two sub components also have different bandwidth and compute characteristics. Their reported deployment runs across close to 10,000 GPUs, using one GPU type optimized for the bandwidth heavy attention work and a different GPU type for the compute heavy expert layers.
There is also work extending this pattern to multimodal models, where an additional pool of instances handles encoding images, audio, or video before the text pipeline even begins, effectively creating a three stage pipeline of encode, prefill, and decode rather than just two.
The direction is clear: the industry is moving toward treating inference as a pipeline of specialized stages that can each be placed on the hardware best suited to them, rather than a single monolithic job.
Scheduling: the part nobody talks about enough
A disaggregated architecture is only as good as the scheduler deciding how to use it. This is where most of the ongoing research effort is actually concentrated, more so than the transfer mechanics themselves.
The key metric these systems optimize for is not raw throughput but goodput, meaning the number of requests that are fully completed within their latency service level objective. A request that gets 80 percent of the way through and then misses its deadline has still consumed GPU time and produced nothing useful, so it counts against the system rather than for it. Recent scheduling research also focuses on request imbalance, meaning what happens when a burst of long prompts arrives and threatens to starve the decode pool of attention, since prefill and decode instances scale independently and a mismatch between the two creates its own kind of bottleneck.
This is a genuinely different scheduling problem from most systems most engineers have built before. It is closer to job scheduling in a data center than request routing in a typical web service, because the "jobs" have two dependent phases running on different machines with a real time data transfer between them.
When disaggregation is not worth it
None of this is free. Splitting prefill and decode introduces real operational cost: two separate clusters to monitor, a KV cache transfer layer that can fail independently of either compute pool, cache miss handling, and a more complex deployment story overall. Teams that adopt this pattern before they actually need it often spend a meaningful amount of extra engineering time in the first several months for little practical benefit.
A reasonable way to think about whether disaggregation is worth adopting:
The signal worth paying attention to is a correlation between long prompt arrivals and latency spikes in ongoing token generation. If that pattern is not showing up in your metrics, the complexity of disaggregation is probably not paying for itself yet, and the better investment is tuning batch sizes and KV cache management on a single unified cluster.
Closing thoughts
Disaggregated prefill/decode serving is a good example of a broader trend in system design right now: the systems that matter most in 2026 are the ones built specifically for how large language models actually behave, rather than general purpose infrastructure patterns borrowed from earlier eras of web architecture. Prefill and decode are fundamentally different computational problems, and treating them that way, with dedicated hardware pools and a fast cache transfer layer between them, has produced real, measurable gains in production at some of the largest AI companies in the world.
It is also a useful case study in judgment. The right answer is not "always disaggregate." It is understanding your own workload well enough to know whether the added complexity buys you something real, which is the same question that sits underneath almost every good system design decision.
Top comments (0)