One 4k prompt costs a serving node as much as 900 generated tokens

A node serving Llama-3.1-70B, 32 conversations streaming, GPU pegged at 95%. Then someone pastes a document. Every live conversation freezes for most of a second, then resumes. Nothing is broken and no config is wrong. The prefill is doing exactly what the arithmetic says it should, and the arithmetic says it costs far more than the time-to-first-token graphs imply.

Time to first token measures the request that just arrived. It does not measure the tax that request levies on the requests already in flight. That tax has a number, and it is bigger than most capacity plans assume.

What one decode step costs

Take the published config.json for Llama-3.1-70B: 80 layers, 8192 hidden, 64 heads, 8 KV heads of 128. FP8 weights are 70.5 GB. FP8 KV cache is 160 KiB per token.

A decode step reads every weight plus the cache of every sequence in the batch. On one H100 SXM at 3.35 TB/s, that sets a floor:

Batch @ 4k ctx	Step time	Reads	Per-sequence
1	21.3 ms	71 GB	47.0 tok/s
8	22.7	76 GB	44.2
32	27.5	92 GB	36.4
64	33.9	114 GB	29.5
256	72.3	242 GB	13.8
512	123.6	414 GB	8.1

Read it as aggregate bandwidth, since 70 B FP8 does not fit on one card: at batch 64 the table says a node sustains 1,889 tokens per second. Published 70B FP8 node benchmarks sit in that band, which is my check that the rest of this arithmetic is not invented. Every sequence in the batch gets its next token from the same weight read, which is why per-sequence speed falls only 6% when the batch grows eightfold: 47.0 to 44.2 tok/s, for eight times the users.

At 4k context this step never becomes compute-bound: each sequence adds 671 MB of cache read, which costs 0.20 ms of bandwidth against 0.18 ms of FLOPs, so memory stays the binding constraint at every batch in the table. At 512-token context the crossing lands near batch 137, and past it the batch stops being free.

What one prefill costs

Prefill is the opposite workload: compute-bound, quadratic in context.

FLOPs per prompt are 2 × N × T for the linear work plus 4 × L × H × d_head × T² for attention. At 70.5B parameters, 40% MFU, and 1,979 TFLOP/s of FP8 peak on an H100:

Context	Linear TFLOP	Attention TFLOP	Attention share	Time @ 40% MFU
1,024	144	3	2%	186 ms
4,096	578	44	7%	786 ms
16,384	2,312	704	23%	3,809 ms
32,768	4,624	2,815	38%	9,397 ms

A 4k prompt is 786 ms of one node's compute. At 30% MFU it is 1,048 ms; at 60% it is 524 ms. The band matters for the constant, not for the ordering.

Notice what is not in that budget: the KV write. A 4k prompt at FP8 writes 671 MB, which is 0.2 ms of bandwidth. Prefill is not a cache problem. It is FLOPs, and on a node that is already serving decode, it is FLOPs that arrive on top of a workload with no spare time.

The conversion nobody does

At batch 32, one generated token across the whole batch is 27.5 ms ÷ 32 = 0.86 ms of GPU time. A 4k prefill is 786 ms. Divide.

One 4k prompt costs the GPU time of 914 generated tokens.

That is the sentence I would put on the capacity spreadsheet. It is also not, by itself, the user-visible cost. The user-visible cost is that the prefill does not run on spare capacity. It runs instead of the decode step, and every live stream waits. Thirty-two sequences each lose 786 ms of wall clock, which at 36 tok/s is roughly 29 tokens each, or about 915 tokens of aggregate progress. The two numbers agree because they are the same number.

So a batch that looks efficient by tokens per second is exactly the batch that makes one arriving document expensive. Bigger batch, more user-seconds per prefill, same 786 ms of GPU.

Chunked prefill does not refund it

The standard fix is to split the prefill into chunks and interleave them with decode. It is a real fix for the spike and it is not a fix for the cost.

The break-even chunk is the chunk that takes exactly as long as one decode step. At 70 B FP8 that works out to 119 tokens, and it is independent of batch size: the decode step is memory-bound, so its time does not move with batch, while a prefill chunk is compute-bound with fixed coefficients.

Chunk	Chunk time	Decode steps lost
128	23 ms	1.08
512	92 ms	4.33
1,024	186 ms	8.75

A 512-token chunk is 4.3 decode steps, repeated eight times for a 4k prompt. Total stall: unchanged. Maximum stall: divided by eight. vLLM's 512 default is a compromise between a number this table can name and a cost this table cannot: tiny chunks raise kernel-launch overhead and shrink the GEMMs, so prefill throughput falls. I do not have a measurement for that penalty, so I will not put a number on it.

If you care about inter-token latency, the target is a chunk near 119 tokens. At 20 tok/s the normal gap between tokens is 50 ms; a 512-token chunk inserts 92 ms, about two tokens' worth of silence, every 512 prompt tokens. A 128-token chunk inserts 23 ms, which is invisible, and probably costs you prefill throughput. That trade is where the tuning lives.

Long context makes it worse in a way chunking cannot reach

Look at the attention column again. At 4k, attention is 7% of the prefill. At 32k it is 38%, and the term is quadratic in T, so it is untouched by chunk size: splitting a 32k prompt into 64 chunks still does 2,815 TFLOP of attention work, now with worse arithmetic intensity. Chunking converts a spike into a plateau. It does not move the plateau, and at long context the plateau is where the time went.

This is the same curve that makes long-context decode expensive, arriving from the other side.

The capacity number for a mixed workload

Put the two together. A node has one GPU-second per second. A 4k prefill consumes 0.79 of them. Decode at batch 32 consumes 0.86 ms per generated token.

Prefills/s (4k)	GPU spent on prefill	Decode sequences that still fit	Per-sequence
0.25	20%	25.7	36.4 tok/s
0.50	39%	19.4	36.4
1.00	79%	6.9	36.4
2.00	100%	0	—

One document per second eats four-fifths of the node. Two per second and the decode queue never drains; throughput does not fall, it stops. If you have ever wondered why a chat product's p99 collapses in the late afternoon while utilization reads healthy, this table is the shape of the answer.

Why prefill/decode disaggregation became practical

If the interference is structural, the clean answer is separate pools and ship the cache. The cost is the transfer, and it is now small.

Link	671 MB KV for a 4k prompt
NVLink 900 GB/s	0.7 ms
400G InfiniBand	13.4 ms
100G InfiniBand	53.7 ms

Against a 786 ms prefill, 400G adds 1.7%. On 100G it adds 7% and, worse, a variable 7%. That is roughly the story of why the architecture was proposed years ago and deployed later: the compute math was always fine, the interconnect was not.

The script

Everything above, runnable. Change MFU and the conclusions scale; change the config and the ordering does not.

L, D, H, KVH, HD = 80, 8192, 64, 8, 128   # Llama-3.1-70B config.json
N, BW, FPEAK = 70.55e9, 3.35e12, 1979e12  # params; H100 SXM bytes/s; FP8 TFLOP/s
KV = 2 * L * KVH * HD                     # bytes per token, FP8

def step(b, mfu=0.4, ctx=4096):
    mem = (N + b * KV * ctx) / BW
    flop = 2 * N * b / (mfu * FPEAK)
    return max(mem, flop)

def prefill(T, mfu=0.4):
    return (2 * N * T + 4 * L * H * HD * T * T) / (mfu * FPEAK)

s32 = step(32)
print(f"one 4k prefill = {prefill(4096)/s32*32:.0f} generated tokens")
print(f"break-even chunk = {step(1)*0.4*FPEAK/(2*N):.0f} tokens")
print(f"chunk 512 = {prefill(512)/step(1):.2f} decode steps")

If you run it and disagree with a constant, that is the useful kind of disagreement: the MFU and the peak are the two numbers I chose rather than derived, and both are checkable on your own card.

What I would measure next

The one thing this arithmetic cannot tell you is how much prefill throughput the 119-token chunk costs. That needs a card, a profiler, and two configs. If you have run that sweep, I would like the numbers; the comment section is where I keep the corrections.

The rest of this series, including the decode and KV-cache pieces these constants come from, is in my archive and on my dev.to profile.

---

Assumptions: FP8 weights and cache, no tensor parallelism overhead, no continuous-batching scheduler loss, 40% MFU for both prefill and decode. Every table is arithmetic from the published config and the H100 datasheet, not a benchmark.