TTFT and TPOT: The Latency Number You Report Hides the One Users Feel

Push a simulated serving queue from 0.2 to 3.0 requests a second and mean end-to-end latency gets 17x worse.

Over the same range, the 99th percentile time to first token gets 762x worse — 206 ms to 157 s.

Both numbers describe the same 300 requests. Only one of them describes the empty box a user is staring at.

👉 Live, with sliders for arrival rate, prompt length and batch size: https://dev48.in/ai/days/day83-ttft-vs-tpot.html

The workload is simulated and seeded

There is no GPU here and no model. There is a discrete-event simulation with a fixed seed (mulberry32, seed 20261109): 300 requests, exponential inter-arrival times, log-normal prompt lengths (median 512 tokens) and output lengths (median 180 tokens), a FIFO queue, and a batch.

The cost model is two lines:

prefill(group) = 8 ms  + 0.08 ms x (total prompt tokens)
decode step    = 18 ms + 1.2  ms x (sequences in the batch)

None of these figures are measurements of any real serving stack, and none of them are claims about one. They are the arithmetic of serving, which is the part that survives a change of hardware. Everything below comes out of that loop, and a Node verifier re-derives all of it from an independent token-by-token replay.

Two clocks, glued by an identity

Time to first token is the silence: queueing plus prefill, before anything appears. Time per output token is the cadence after it. They compose exactly:

e2e = TTFT + TPOT x (output tokens - 1)

Not approximately. Across all 300 requests the worst disagreement is 3.64e-12 ms. The minus one is there because the first token is already paid for inside TTFT.

That identity is why a single total is useless for diagnosis. A slow total is a long silence, a slow cadence, or a long answer — three different causes with three different fixes, and the total cannot tell you which. In the default run the request with the worst TTFT and the request with the worst end-to-end are not the same request.

The queue lands almost entirely on the silence

arrival rate	TTFT p50	TTFT p99	TPOT p50	tok/s
0.2 req/s	55.5 ms	206.3 ms	20.40 ms	48.2
0.8 req/s	71.1 ms	5,966 ms	26.46 ms	191.9
1.2 req/s	13,377 ms	24,010 ms	29.14 ms	260.2
2.0 req/s	53,606 ms	110,884 ms	29.30 ms	266.8
3.0 req/s	76,188 ms	157,311 ms	29.30 ms	267.4

TTFT p99 moves 762x. TPOT p50 moves 1.44x. Mean end-to-end — the column most dashboards actually plot — moves 17x, because it is dominated by how many tokens the answers had, which is a property of the questions and not of the server.

A deploy that moved the first column from 55 ms to 76 seconds would show up on a mean-latency chart as a moderate regression.

TPOT is batch size, and it is exactly batch size

A decode step reads the weights out of memory once and uses them for every sequence in the batch. Fill the batch and time it:

full batch of	TPOT	tokens/s to one user	tokens/s for the server	ms of step per sequence
1	19.20 ms	52.1	50.9	19.200
8	27.60 ms	36.2	246.8	3.450
32	56.40 ms	17.7	420.4	1.762
64	94.80 ms	10.5	476.2	1.481

The same number, divided two ways. Per token the fixed cost falls towards 1.2 ms; per sequence it only ever rises. The server gets 9.4x faster and every individual stream gets 4.9x slower, and only one of those is on a capacity dashboard.

Prompt length does not appear in the step expression at all. Measured in the clean room, TPOT across a 64x range of prompt lengths at four batch sizes has a total spread of 8.38e-13 ms — floating-point noise, nothing else.

Except that it leaks, and the leak is scheduling

Put the same server under load and prompt length suddenly does move TPOT. Batch 16, 1.4 req/s. The control column prices prefill at zero and changes nothing else — same arrivals, same batch, same decoding:

median prompt	TPOT, prefill free	TPOT, prefill priced	server time in prefill
128	31.035 ms	32.730 ms	2.96%
512	31.035 ms	36.462 ms	8.44%
2048	31.035 ms	49.119 ms	27.08%
4096	31.035 ms	63.025 ms	43.01%

The control column is identical to the last digit, because with prefill free the prompt does not exist. The priced column doubles. Nothing about decoding changed — somebody else's 4,000-token prefill is holding the accelerator while your sequence waits for its next token. This is the effect chunked prefill exists to spread out, and it is why a TPOT regression can be caused by a change nobody made to decoding.

The assertion that failed

I expected the classic trade: make the scheduler wait to accumulate a full batch before prefilling, fuse the prefills, buy throughput, pay in TTFT. Half of it showed up.

2,048-token prompts, 0.8 req/s, batch 16:

batch wait window	TTFT p50	tok/s	prefill share
0 ms	207.5 ms	191.7	17.726%
150 ms	369.2 ms	191.6	17.668%
400 ms	640.0 ms	191.5	17.578%
800 ms	1,052.6 ms	191.3	17.494%

The fusing worked exactly as advertised: prefill's share of server time falls monotonically. Throughput moves −0.22%. This server is decode-bound, so the time the optimisation freed was never the bottleneck.

TTFT p50 went up 5.1x anyway. The optimisation succeeded and only the bill arrived — charged to users, in a column the throughput dashboard does not have.

A percentile over requests is not a percentile over tokens

p99 TPOT usually means: average each request's cadence, take the percentile over requests. That gives a two-token reply exactly as many votes as a two-thousand-token one. Weight by tokens emitted instead and the number moves — at 1.4 req/s, 32.12 ms per request against 31.27 ms per token.

Neither is wrong. They answer different questions, and a dashboard that does not say which one it is answering is not reporting anything.

Budget the two separately, because one SLO cannot hold both

Take a budget of TTFT p90 ≤ 1 s and TPOT p90 ≤ 60 ms, and sweep batch size at 2.0 req/s:

batch	TTFT p90	TPOT p90	verdict
1	1,102,077 ms	19.2 ms	TTFT over
16	31,947 ms	42.6 ms	TTFT over
32	3,796 ms	64.8 ms	both over
64	154 ms	73.0 ms	TPOT over

Small batches blow the TTFT ceiling on queueing. Large ones blow the TPOT ceiling on step time. No batch size satisfies both. That is not a tuning problem, it is a capacity verdict — and the split metrics are the only thing that can state it, because the same budget at 0.8 req/s and batch 16 is met comfortably (TTFT p90 129 ms, TPOT p90 30.9 ms).

An end-to-end budget would have been satisfied or violated by a change in answer length that nobody deployed.

One bug worth keeping

The batch-wait scheduler advances the clock to head.arrival + batchWindow, then tests whether the window has expired. That sum rounds a hair low in a double — 299.9999999999999 for a 300 ms window — so the test says no, the clock is advanced to the same instant again, and the loop spins forever on one timestamp. Fourteen window values hung before the comparison got an epsilon. The page now asserts that all 28 window-and-batch combinations terminate and serve all 300 requests.

What is checked

The page runs 47 assertions in your browser, including the nearest-rank percentile against a hand-checked array, a fixed seed reproducing a run string-identically, the end-to-end identity to 1e-9 for every request, TTFT monotone in arrival rate, and TPOT invariant to prompt length in isolation and demonstrably not invariant under load.

A Node verifier lifts the same engine out of the HTML and adds 177 more, including an independent replay that records a timestamp for every one of ~70,000 tokens and re-derives TTFT, TPOT and end-to-end from that log rather than from the three scalars the page keeps. Both at zero failures.

Report TTFT and TPOT separately, at p90 or p99, and say which output length you assumed when you turned them into a total. Then when the total moves, the identity tells you which of the three terms moved.