A coding agent takes 11 seconds to rename a variable.
The model is fine. The GPU is fine. The p99 is 11 seconds because one user pasted a 100K-token repository into the same pool, and everyone else's decode step is stuck behind its prefill.
That's the part of LLM serving that never shows up in the demo. Your p99 is not a property of your model. It's a property of your batch.
Nobody is attacking you. The longest prompt in the batch is.
The setup
You own LLM serving. The traffic is not toy chat. It's coding agents, RAG, long system prompts, and tool outputs. The 100K-token class is routine, and prefixes repeat. Prefill and decode share the same GPU pool. Tail latency blows up under load. GPUs are expensive.
Design the serving path:
- Where does the router live, and how does it guess cold versus warm?
- Which tier pays full compute, and which tier only reads cache?
- How many KV levels do you keep?
- How do you isolate decode from prefill noise?
Take a minute before you read the solution. The interesting part is not the boxes. It's what happens when traffic turns cold.
An LLM request is two workloads
A normal web request is one unit of work. An LLM request is two, and they scale in opposite directions.
Prefill runs the prompt through the model. It is a big matrix multiply, compute-bound, and bursty. A 100K-token prompt can occupy a batch slot for seconds.
Decode generates one token at a time. It is memory-bandwidth-bound, and every token has a deadline. The user feels the gap between tokens.
Put them in one continuous batch and the long prefill becomes a head-of-line block. Decode tokens for other requests sit in the queue. The average looks fine because most requests are short. The p99 belongs to the requests that arrived behind the 100K-token prompt.
This is why "add more GPUs" can make the p99 worse. You added capacity to a queue that is still mixing two workloads. The long prefill is still in front of the short decode.
And the context window makes this structural, not a bug. Coding agents, RAG, and long system prompts push more traffic into the 100K class every quarter. The tail is not an edge case anymore. It's the product.
Streaming makes it worse. Your SLO is not one latency number. It is TTFT, time to first token, and TPOT, time per output token. A user will forgive a two-second TTFT. They will not forgive a decode that stutters every 200 ms.
The fix, in four moves
1. Split the pools
Prefill and decode get separate GPU pools. Prefill can use tensor parallelism and large batches. Decode gets a smaller, latency-optimized pool. The KV cache is the handoff between them. No shared batch, no shared queue.
2. Route on cache warmth
The router hashes the prompt prefix, or walks a radix tree, and asks which replica already holds the KV blocks for that prefix. A warm prompt goes to the replica that can read the cache. A cold prompt goes to the prefill tier, which pays full compute and writes new blocks.
The router's job is cache locality plus load, not round-robin. In an agent product this matters more than in chat: the system prompt, tool definitions, and earlier turns are a prefix you can reuse. Miss that, and you pay prefill again on every turn.
Agents make it harder. Every tool call appends output, so the prefix grows and the cache key changes. A cache that ignores the tool transcript will miss. Track prefix hit rate per route, not a single global hit rate. A healthy global number can hide a coding-agent route that is always cold.
3. Tier the KV cache
L1 in GPU HBM, L2 in host DRAM, L3 in a distributed cache over RDMA. Hot blocks migrate up. Cold blocks go down. In the reported setup, a warm 100K-token context goes from seconds of compute to a few hundred milliseconds of TTFT, because the expensive part already happened.
4. Isolate decode and control admission
Continuous batching for decode. Chunked prefill or a separate prefill queue. Separate SLOs: TTFT for prefill, TPOT or ITL for decode. Admission control so a cold burst cannot borrow decode capacity. The reported setup caps max in-flight requests at 24.
Read the diagram from left to right. Start at the client. The prompt goes to the cache-aware router. The router forks: cold goes up to the pre-prefill tier, warm goes down to the prefill pool. The cold path writes new KV blocks into the cache. The warm path reads them back. The cache has three tiers: L1 GPU HBM, L2 host DRAM, L3 RDMA cluster. The prefill pool hands the KV state to the decode pool, and decode streams tokens to the client.
Notice what is missing. There is no arrow from the cold path into decode. The long compute cannot sit in the decode queue.
The numbers, with the caveat
Together AI reports 1.1 to 1.15 QPS per GPU for CPD, against a 0.75 to 0.8 baseline for the 2P1D setup. That is roughly 35 to 40 percent more sustained throughput under a tail-sensitive SLO, on B200 GPUs. Treat those as that source's reported numbers, not universal law. The useful part is the shape, not the decimal.
The math that makes it real
Take a 70B model with grouped-query attention: 80 layers, 8 KV heads, head_dim 128, fp16.
KV per token = 2 × 80 × 8 × 128 × 2 bytes = 320 KiB.
A 100K-token request = 320 KiB × 100,000 ≈ 32 GB.
The weights are about 140 GB in fp16. On a node with 4 × 192 GB, the weights take 140 GB and leave about 628 GB for KV. That is roughly 19 concurrent 100K-token requests before fragmentation, activations and cache tiers.
That is the real scheduling budget. Not "how many requests per second." How many long contexts can sit in memory at once. The KV cache is not an optimization. It is the thing you are actually selling.
The follow-up that separates answers
Almost every prompt is cold. Which piece breaks first?
The prefill pool saturates first. Then the cache-fill path: every cold request writes new KV blocks, and the distributed cache's write bandwidth becomes the bottleneck. The router's hit-rate assumption collapses. TTFT dies. Decode isolation still protects TPOT, which is the point: a good design fails in one lane, not both.
The fix is not more GPUs. It is admission control with deadlines, a reserved prefill lane for cold bursts, and a degrade path: shorter context, smaller model, or reject with a retry hint. If cold prefill borrows decode capacity, you lose both SLOs.
What a good answer demonstrates
- Why mixing prefill and decode hurts the tail.
- Cold versus warm routing and the cost of each branch.
- The KV hierarchy and how decode stays isolated.
- The failure mode when the cache misses.
Try it
This exercise is one of 47 in the DDIA Study Lab. The case diagram and the rest of the lab are in the repo. The exercise is deliberately unsolved, so bring your own design.
Repo: https://github.com/Hurdadd/ddia-study-lab
Site: https://ddia-study-lab.vercel.app/
If you try it, tell me where your design breaks. The cold-traffic answer is the one I care about.
Top comments (0)