DEV Community: Hurdad Ashouri

A 100K-token prompt is a denial-of-service attack you paid for

Hurdad Ashouri — Tue, 20 Oct 2026 17:43:51 +0000

A coding agent takes 11 seconds to rename a variable.

The model is fine. The GPU is fine. The p99 is 11 seconds because one user pasted a 100K-token repository into the same pool, and everyone else's decode step is stuck behind its prefill.

That's the part of LLM serving that never shows up in the demo. Your p99 is not a property of your model. It's a property of your batch.

Nobody is attacking you. The longest prompt in the batch is.

The setup

You own LLM serving. The traffic is not toy chat. It's coding agents, RAG, long system prompts, and tool outputs. The 100K-token class is routine, and prefixes repeat. Prefill and decode share the same GPU pool. Tail latency blows up under load. GPUs are expensive.

Design the serving path:

Where does the router live, and how does it guess cold versus warm?
Which tier pays full compute, and which tier only reads cache?
How many KV levels do you keep?
How do you isolate decode from prefill noise?

Take a minute before you read the solution. The interesting part is not the boxes. It's what happens when traffic turns cold.

An LLM request is two workloads

A normal web request is one unit of work. An LLM request is two, and they scale in opposite directions.

Prefill runs the prompt through the model. It is a big matrix multiply, compute-bound, and bursty. A 100K-token prompt can occupy a batch slot for seconds.

Decode generates one token at a time. It is memory-bandwidth-bound, and every token has a deadline. The user feels the gap between tokens.

Put them in one continuous batch and the long prefill becomes a head-of-line block. Decode tokens for other requests sit in the queue. The average looks fine because most requests are short. The p99 belongs to the requests that arrived behind the 100K-token prompt.

This is why "add more GPUs" can make the p99 worse. You added capacity to a queue that is still mixing two workloads. The long prefill is still in front of the short decode.

And the context window makes this structural, not a bug. Coding agents, RAG, and long system prompts push more traffic into the 100K class every quarter. The tail is not an edge case anymore. It's the product.

Streaming makes it worse. Your SLO is not one latency number. It is TTFT, time to first token, and TPOT, time per output token. A user will forgive a two-second TTFT. They will not forgive a decode that stutters every 200 ms.

The fix, in four moves

1. Split the pools

Prefill and decode get separate GPU pools. Prefill can use tensor parallelism and large batches. Decode gets a smaller, latency-optimized pool. The KV cache is the handoff between them. No shared batch, no shared queue.

2. Route on cache warmth

The router hashes the prompt prefix, or walks a radix tree, and asks which replica already holds the KV blocks for that prefix. A warm prompt goes to the replica that can read the cache. A cold prompt goes to the prefill tier, which pays full compute and writes new blocks.

The router's job is cache locality plus load, not round-robin. In an agent product this matters more than in chat: the system prompt, tool definitions, and earlier turns are a prefix you can reuse. Miss that, and you pay prefill again on every turn.

Agents make it harder. Every tool call appends output, so the prefix grows and the cache key changes. A cache that ignores the tool transcript will miss. Track prefix hit rate per route, not a single global hit rate. A healthy global number can hide a coding-agent route that is always cold.

3. Tier the KV cache

L1 in GPU HBM, L2 in host DRAM, L3 in a distributed cache over RDMA. Hot blocks migrate up. Cold blocks go down. In the reported setup, a warm 100K-token context goes from seconds of compute to a few hundred milliseconds of TTFT, because the expensive part already happened.

4. Isolate decode and control admission

Continuous batching for decode. Chunked prefill or a separate prefill queue. Separate SLOs: TTFT for prefill, TPOT or ITL for decode. Admission control so a cold burst cannot borrow decode capacity. The reported setup caps max in-flight requests at 24.

Read the diagram from left to right. Start at the client. The prompt goes to the cache-aware router. The router forks: cold goes up to the pre-prefill tier, warm goes down to the prefill pool. The cold path writes new KV blocks into the cache. The warm path reads them back. The cache has three tiers: L1 GPU HBM, L2 host DRAM, L3 RDMA cluster. The prefill pool hands the KV state to the decode pool, and decode streams tokens to the client.

Notice what is missing. There is no arrow from the cold path into decode. The long compute cannot sit in the decode queue.

The numbers, with the caveat

Together AI reports 1.1 to 1.15 QPS per GPU for CPD, against a 0.75 to 0.8 baseline for the 2P1D setup. That is roughly 35 to 40 percent more sustained throughput under a tail-sensitive SLO, on B200 GPUs. Treat those as that source's reported numbers, not universal law. The useful part is the shape, not the decimal.

The math that makes it real

Take a 70B model with grouped-query attention: 80 layers, 8 KV heads, head_dim 128, fp16.

KV per token = 2 × 80 × 8 × 128 × 2 bytes = 320 KiB.

A 100K-token request = 320 KiB × 100,000 ≈ 32 GB.

The weights are about 140 GB in fp16. On a node with 4 × 192 GB, the weights take 140 GB and leave about 628 GB for KV. That is roughly 19 concurrent 100K-token requests before fragmentation, activations and cache tiers.

That is the real scheduling budget. Not "how many requests per second." How many long contexts can sit in memory at once. The KV cache is not an optimization. It is the thing you are actually selling.

The follow-up that separates answers

Almost every prompt is cold. Which piece breaks first?

The prefill pool saturates first. Then the cache-fill path: every cold request writes new KV blocks, and the distributed cache's write bandwidth becomes the bottleneck. The router's hit-rate assumption collapses. TTFT dies. Decode isolation still protects TPOT, which is the point: a good design fails in one lane, not both.

The fix is not more GPUs. It is admission control with deadlines, a reserved prefill lane for cold bursts, and a degrade path: shorter context, smaller model, or reject with a retry hint. If cold prefill borrows decode capacity, you lose both SLOs.

What a good answer demonstrates

Why mixing prefill and decode hurts the tail.
Cold versus warm routing and the cost of each branch.
The KV hierarchy and how decode stays isolated.
The failure mode when the cache misses.

Try it

This exercise is one of 47 in the DDIA Study Lab. The case diagram and the rest of the lab are in the repo. The exercise is deliberately unsolved, so bring your own design.

Repo: https://github.com/Hurdadd/ddia-study-lab

Site: https://ddia-study-lab.vercel.app/

If you try it, tell me where your design breaks. The cold-traffic answer is the one I care about.

DDIA is a great book. It is a bad lab.

Hurdad Ashouri — Mon, 19 Oct 2026 17:44:22 +0000

DDIA is the best distributed systems book I have read. I finished the second edition with a head full of vocabulary and no way to check whether I actually understood any of it.

That is not a knock on the book. It has dozens of figures, real company stories, and code for a small key-value store. What it does not have is a feedback loop. No exercises. No answer key. No failure timeline you can step through. No numbers you can change and watch break.

The gap shows up in interviews. You can explain linearizability and still freeze when the old leader comes back with a stale lease. You can describe two-phase commit, or 2PC, and still not know what a participant does after it votes yes and the coordinator disappears.

So I built the lab I wanted while reading it.

The uncomfortable part

Reading DDIA cover to cover is not a flex. Most people finish it with a warm feeling and no changed decisions. I did that the first time.

The book is a map. Maps are useful. But you do not learn a city by reading a map, and you do not learn a distributed system by nodding at a diagram. You learn it when you have to choose between two designs, defend the choice, and find the failure that kills it.

Most study notes make this worse. They compress a chapter into definitions and arrows, then delete the part where you have to think. You finish the summary and feel productive. You still cannot answer the interview question.

The exercises here are not "summarize chapter 6." They are "the leader dies here, the client retries there, show me the duplicate and the fix."

If you have read DDIA and never changed a design because of it, you read a good book. You did not practice.

What is in it

14 chapters, in English and Persian, with the full content in both.

155 chapter diagrams. 89 are interactive step-throughs: write paths, failovers, quorums, 2PC, compaction. 66 are hand-built concept diagrams: UML, timelines, decision trees. Every diagram is followed by a teacher-style walkthrough that reads the figure step by step. Not a caption.

302 worked examples. Each one has the same shape: initial state, operation, result and reason. The numbers are teaching numbers, labeled as such.

47 design exercises with full worked solutions, plus 118 diagrams inside those solutions. The exercises are leveled for foundation, senior and staff roles, and tagged by priority.

A capstone that turns the chapters into one AI platform design: requirements, estimates, 15 ADRs, recovery targets and a developer journey.

The state that made 2PC click

Two-phase commit looks simple on a whiteboard. Prepare, commit, done. The hard part is the gap between a yes vote and a decision.

Initial state: participant A voted yes. It is prepared, holds locks, and waits for the coordinator.

Operation: the coordinator disappears. A times out.

Result: A cannot commit or abort on its own. It keeps the locks until it learns the decision, or until an operator decides by hand. A timeout is not information. Releasing the locks early breaks atomicity.

That is the kind of state I could not practice by reading. I could nod at it. I could not reason through it under pressure.

The diagram is not decoration. It is the exercise. You trace the arrows, find the two points of no return, and see why "just retry" is not a recovery strategy.

What it is not

It is not a replacement for DDIA. The book is the source. The lab is the practice.

It is not a production system. The capstone is a design workshop. Everything proposed is labeled proposed. There are no fake benchmarks or invented SLOs.

It is not a coding course. The 47 exercises are design and interview drills, not "implement a B-tree in 300 lines." Some have optional lab extensions, but the main path is reasoning on a whiteboard.

Why two languages

I read in English and explain in Persian. Most DDIA notes pick one language. Every chapter page has a toggle, and the same content exists in both. If you are learning technical English, the side-by-side version is useful on its own.

Try it

Start with chapter 4 if you want storage engines, or chapter 10 if you want consensus. The exercise audit page lists the level and evidence for every drill.

Repo: https://github.com/Hurdadd/ddia-study-lab

Site: https://ddia-study-lab.vercel.app/

If this helps you prepare, star the repo. If a diagram is wrong, open an issue. I would rather fix the diagram than keep the star.