A token you have shown is a token you have spent. Every guard worth running needs lookahead - it cannot rule on a claim until the claim closes - so streaming is a race between two cursors crawling left to right over the same tokens. The detector's sits L tokens behind the generator because that is how much context it needs. The consumer's is wherever the thing at the other end has got to, and it commits everything it passes.
No language is simulated. A response is a length, a set of sentence boundaries and at most one defect, and the policy is one line:
releaseIndex(policy, j) =
stream -> j // show it the moment it exists
hold / guarded -> min(j + B, n - 1) // keep B in hand, flush at the end
sentence / checked -> sentEnd[j] // release the sentence when it closes
commit(j) = max(commit(j-1), show(j)) + 1/R // machine: R = Infinity, so commit === show
Dependency-free JavaScript, five worlds of 500 responses: https://dev48.infy.uk/ai/days/day68-streaming-commitment.html
A naive stream puts defective text on screen in 100% of defective responses, 17.2 tokens of it. What is committed is decided by the consumer: a reader takes 5.5 tokens/s, speech 3.2, and a machine - a fired tool call, a webhook, a downstream parser - has no backlog at all. Same model, same policy, same defects: 5.6 leaked responses per thousand against 212, a 38x gap from nothing but reading speed. The horizon has a closed form, H = (R(L - B) - G + G*R*latency) / (G - R), and it agrees with the token-level simulation on 32,344 of 32,344 eligible responses. Its corollary is the opposite of the intuition: a faster model is a safer streamer, 98.0% down to 0.9% leak as speed goes 5 to 120 tok/s, because the reader simply falls further behind.
The recommendation I set out to make is Pareto-dominated
Release at sentence boundaries. It reads better, it matches how the guard thinks, and the check has exactly the sentence it needs the moment the sentence closes. Matched to an identical time to first token, against a machine consumer:
| policy | first token | leaked |
|---|---|---|
| sentence boundary | 777 ms | 87.1% |
| hold-back, 16 tokens | 775 ms | 53.3% |
| check-gated hold-back, 11 tokens | 770 ms | 45.2% |
| boundary, check-gated | +120 ms | 32.7% |
It loses on mean display lag too, so there is no axis on which it wins. The mechanism is embarrassing once you see it: a boundary release fires at the exact instant the detector receives its input, so it loses the race by precisely the detector's own latency - and because it releases a whole sentence at once, when it loses it loses 14.7 tokens rather than one. Gate the release on the check instead of on the boundary and it reads 32.7% at 20 ms, 120 ms and 600 ms of detector latency alike. Waiting for the check makes the check's speed irrelevant, which is what correct looks like in a race.
The floor no buffer removes
Hold-back leak against a machine consumer falls 100% to 53.3% to 26.5% as the buffer goes 0, 16, 40 tokens - then 18.3% at 80, 17.5% at 120, and 17.3% for ever after. At the plateau 96% of what is left was released by the end-of-stream flush, which fires on the generator's stop signal without waiting for the last check. Make the flush wait and the same 80-token buffer goes to 0.8%. Doubling the buffer instead costs a second of first-token latency and buys under a point.
64 verifier assertions, 18 in the page, and the null result lives in the test file where the inconvenient half cannot be quietly deleted. Part of a from-scratch series - one AI concept a day, measured in-browser: https://dev48.infy.uk/aifromzero.php
Top comments (0)