Delivered but unbilled is the streaming failure where your AI response renders the whole answer to the user, but the terminal usage frame is dropped, zeroed, or malformed, so your client logs 0 output tokens for a call that cost money. stream_billing_gate.py reconciles the delivered text against the logged usage, offline, and blocks when text shipped and nothing was billed.
AI disclosure: I wrote
stream_billing_gate.pywith an AI assistant and ran it myself, offline, on Python 3.13.5, standard library only, no network. Every number in the output blocks below is pasted from a real local run. I checked the exit codes (0 / 1 / 2), ran each scenario twice to confirm STDOUT is byte-for-byte identical, and had the tool print a sha256 of its own report so you can reproduce the exact bytes. The one external finding I cite (a Dev.to proof of concept) and the one external number (a Hacker News thread's score) are other people's, attributed inline, and kept out of my fixture numbers.
In short:
- A streamed response delivers text first and accounting last. If the terminal usage frame never lands, the text is already on the screen and your client logs 0 output tokens. The user got the whole answer. Your bill says it was free.
- The gate reads a recorded stream as data, rebuilds the delivered text, and reconciles it against the logged usage. When text was delivered and usage is 0 or absent, it blocks with
delivered-but-unbilled. - The demo that matters: two stream logs with byte-identical text deltas. Delete one line, the usage frame, and the verdict flips from OK exit 0 to BLIND exit 1.
- Token counting here is a conservative FLOOR (word count), not the vendor's bill. The hard claim is smaller and unbreakable: text was delivered, so real tokens are above zero, so a logged 0 is wrong.
- This is a post-hoc reconciliation of recorded streams. It does not intercept live traffic, call an API, or need a key.
How does a streamed answer get billed as zero?
A streaming response arrives in two parts that travel separately. First the content deltas: dozens of small events, each carrying a slice of text, which your client concatenates and paints on the screen. Then, at the very end, one terminal frame carrying the token accounting: input tokens, output tokens, the numbers your billing code reads to know what the call cost.
The order is the problem. By the time the accounting is supposed to arrive, the answer is already delivered. If that last frame is dropped by a flaky connection, truncated, or written with a zero, the user never notices. They got their answer. Your client, meanwhile, has nothing to log, so it keeps whatever value it initialized. For a lot of clients that value is zero. No exception. No crash. A response that cost real output tokens is recorded as free. That zero is a choice your client made, not a law of streaming, and how the accounting travels varies by provider, which the formats section below gets into.
On July 7, a developer publishing as kielltampubolon posted a proof of concept on Dev.to titled "Asynchronous Telemetry Blindness in AI Streaming Clients", showing this exact shape: the text deltas render the full answer while a dropped terminal usage frame leaves billing sitting at zero, with no error surfaced. That framing is his, and his write-up is a local-only proof of concept, not a production incident. I am not reusing his numbers. I built my own fixtures so the failure is reproducible on your logs, and so a test can fail closed on it.
Tracking is not control
Your dashboard says $0 for that call. The dashboard is not lying. It is faithfully reporting a number that is wrong at the source. This is the whole thesis of the pre-execution and reconciliation gates I keep building: tracking a number is not the same as controlling the thing the number describes. You tracked usage. The usage you tracked was zero. The tokens were spent anyway.
Here is the claim, stated so you can break it. Given a recorded stream where text deltas exist and the terminal usage frame is absent or zero, the gate must exit non-zero with reason delivered-but-unbilled. Given a stream with a usage frame whose output count meets the floor derived from the delivered text, it must exit 0. One removed line has to flip the verdict. If you can show me a stream that delivered text and the gate stayed quiet, or one that logged its usage honestly and the gate blocked, the tool is broken and this post with it.
What the gate reconciles
Two quantities per stream. The delivered text, rebuilt by concatenating every content delta. And the logged output tokens, pulled from the terminal usage frame, or None when that frame is missing or unusable.
The token side needs an honest word about method. Without a real tokenizer, and this tool ships with none, standard library only, I cannot tell you the exact token count of the delivered text. So I do not. I compute a FLOOR: the number of whitespace-delimited words. A BPE tokenizer keyed on whitespace almost always emits at least one token per word, and usually more once punctuation and sub-word splits count. So the real output_tokens is nearly always higher than this floor. It is a deliberately low, human-readable magnitude, not a measurement.
The gate's real claim does not lean on that floor being precise. It leans on something smaller that cannot be argued with: if the delivered text is non-empty, the real output token count is above zero. Therefore a logged zero is provably wrong, whatever the exact number was. The floor just gives you a readable "at least this many" figure to put in the alert.
That gives four verdicts:
| Verdict | Condition | reason-code | exit |
|---|---|---|---|
| OK | usage frame present, logged >= floor
|
usage-consistent |
0 |
| UNDERCOUNT | usage frame present, 0 < logged < floor
|
partial-telemetry-loss |
2 |
| BLIND | text delivered, logged is 0 / absent / unparseable |
delivered-but-unbilled |
1 |
| EMPTY | no text delivered, nothing to reconcile | nothing-delivered |
0 |
BLIND is the hard, logical one and gets the blocking exit code. UNDERCOUNT is softer: it says the logged count is below the floor, which is suspicious but heuristic, so it warns rather than blocks. The decision itself is a few lines, simplified for the post (the real function returns dicts with detail strings, and it carries one extra guard shown here that fails closed on a stream in a shape it cannot recognize):
def classify(delivered_text, logged, usage_seen, events, recognized): # simplified for the post
floor = word_floor(delivered_text) # whitespace word count
if events and not recognized: # valid JSON, but no shape the gate knows -> fail closed
return "UNRECOGNIZED", "unrecognized-stream", 2
if not delivered_text.strip():
return "EMPTY", "nothing-delivered", 0
if logged is None or logged == 0:
return "BLIND", "delivered-but-unbilled", 1
if logged < floor:
return "UNDERCOUNT", "partial-telemetry-loss", 2
return "OK", "usage-consistent", 0
Which stream formats does it read?
The gate reads the recorded stream as data and auto-detects the wire shape, so you can point it at the logs you already keep. It maps four documented formats to the same two quantities, delivered text and logged output tokens:
| Format | Delivered text | Logged output tokens |
|---|---|---|
| OpenAI Chat Completions | choices[].delta.content |
usage.completion_tokens in the final chunk, sent only when you set stream_options={"include_usage": true}
|
| Anthropic Messages | content_block_delta.delta.text |
message_delta.usage.output_tokens (cumulative; the last one is the total) |
| OpenAI Responses | response.output_text.delta |
response.completed then response.usage.output_tokens
|
| Generic normalized | {"type":"content.delta","text":...} |
{"type":"usage","output_tokens":...} |
How the accounting travels is not the same across providers, and that changes what a zero means. OpenAI Chat sends usage in one terminal chunk, and only if you asked for it: leave include_usage off and there is no usage frame at all, which reads as delivered-but-unbilled by default. Anthropic sends output tokens cumulatively across message_delta frames, so a dropped final frame undercounts rather than zeroes, unless your client commits only the terminal number. A logged 0 is a fact about what your client recorded, not a universal property of streaming. The gate does not care which of these happened. It reconciles the text that reached the user against the number that reached your logs.
The FLOOR is provider-agnostic. It comes from the concatenated delivered text alone, so the same word count governs an OpenAI stream and an Anthropic one. A format the gate does not recognize is reported as unrecognized-stream (exit 2), which fails closed instead of passing as EMPTY, so a broken adapter cannot hide a leak behind a green check.
Quick start
Feed it a recorded stream as JSONL, one event per line, in any of the formats above. Then run it:
python3 stream_billing_gate.py fixtures/fixture_blind.jsonl
No install, no key, no network. It reads the file, prints a verdict, and returns an exit code you can wire into CI over your recorded stream logs, whether they are OpenAI Chat, Anthropic, Responses, or the normalized shape. A log in a format it does not recognize returns unrecognized-stream (exit 2) rather than a silent pass, so confirm the fit on a known-good stream before you trust a green.
The one line that flips the verdict
Here are two files. fixture_ok.jsonl is a recorded stream: a normal answer about capping agent spend, followed by a terminal usage frame reporting 190 output tokens. fixture_blind.jsonl is the same stream with that one usage line removed. The text deltas are byte-identical. Run each:
stream-billing-gate: delivered text vs logged usage
files: 1
[1] fixture_ok.jsonl
delivered : 137 words, 750 chars (floor 137 tokens)
logged : output_tokens=190
verdict : OK (usage-consistent) exit 0
detail : logged 190 >= floor 137
summary: 1 OK, 0 UNDERCOUNT, 0 BLIND, 0 EMPTY -> overall exit 0
report-sha256: f6b8ce0b2b7a09f57a589a862b53024a81ca618cfecd34f82fa6231c670486a7
stream-billing-gate: delivered text vs logged usage
files: 1
[1] fixture_blind.jsonl
delivered : 137 words, 750 chars (floor 137 tokens)
logged : none
verdict : BLIND (delivered-but-unbilled) exit 1
detail : delivered >= 137 tokens of text; no usage frame at all
summary: 0 OK, 0 UNDERCOUNT, 1 BLIND, 0 EMPTY -> overall exit 1
report-sha256: 318dff2585a7874bd04cd06a1430a9d3b303d93535910bd46be1d5987f81a4ce
Same 137 words delivered. In one file that costs at least 137 tokens and the log says 190. In the other it costs at least 137 tokens and the log says nothing. The exit code goes from 0 to 1. The only difference between the two files is a single line:
$ diff fixtures/fixture_ok.jsonl fixtures/fixture_blind.jsonl
10d9
< {"type":"usage","input_tokens":523,"output_tokens":190}
That is the failure in one line. In production the line does not get deleted by hand. A connection blips, a proxy truncates, a client swallows the final frame, and you are left with fixture_blind.jsonl and a dashboard that says the call was free. The two sha256 digests are the tool hashing its own report, so you can confirm you got the same bytes I did.
Four verdicts, one sweep
Hand it a batch of recorded streams and it reconciles each one. This run (stream_billing_gate.py fixtures/*.jsonl, so the files come in alphabetical order) covers all four verdicts plus the two broken shapes of the usage frame, present-but-zero and present-but-unparseable, both of which are still delivered-but-unbilled:
stream-billing-gate: delivered text vs logged usage
files: 7
[1] fixture_blind.jsonl
delivered : 137 words, 750 chars (floor 137 tokens)
logged : none
verdict : BLIND (delivered-but-unbilled) exit 1
detail : delivered >= 137 tokens of text; no usage frame at all
[2] fixture_broken_usage.jsonl
delivered : 46 words, 274 chars (floor 46 tokens)
logged : none
verdict : BLIND (delivered-but-unbilled) exit 1
detail : delivered >= 46 tokens of text; usage frame present but unparseable
[3] fixture_empty.jsonl
delivered : 0 words, 0 chars (floor 0 tokens)
logged : none
verdict : EMPTY (nothing-delivered) exit 0
detail : no text delivered; nothing to reconcile
[4] fixture_injection.jsonl
delivered : 60 words, 344 chars (floor 60 tokens)
logged : none
verdict : BLIND (delivered-but-unbilled) exit 1
detail : delivered >= 60 tokens of text; no usage frame at all
[5] fixture_ok.jsonl
delivered : 137 words, 750 chars (floor 137 tokens)
logged : output_tokens=190
verdict : OK (usage-consistent) exit 0
detail : logged 190 >= floor 137
[6] fixture_undercount.jsonl
delivered : 72 words, 403 chars (floor 72 tokens)
logged : output_tokens=12
verdict : UNDERCOUNT (partial-telemetry-loss) exit 2
detail : logged 12 < floor 72; telemetry lost part of the stream
[7] fixture_zero_usage.jsonl
delivered : 55 words, 272 chars (floor 55 tokens)
logged : output_tokens=0
verdict : BLIND (delivered-but-unbilled) exit 1
detail : delivered >= 55 tokens of text; usage frame logged output_tokens=0
summary: 1 OK, 1 UNDERCOUNT, 4 BLIND, 1 EMPTY -> overall exit 1
report-sha256: dc204b014afadff09b09b82a791dd86ba7767df01454fbc2cc011321c9cd5111
Look at fixture_zero_usage and fixture_broken_usage. A usage frame that says output_tokens: 0 and a usage frame with a broken value both land as BLIND, because in both cases text was delivered and no honest token count was logged. A zero is not an OK. Absence is not an OK. Only a real count that meets the floor is an OK. The report ends with a sha256 of its own body, and because that body depends on the order you pass the files, fixtures/*.jsonl reproduces this exact digest.
The same check on real provider logs
Those seven fixtures use the normalized shape, which is fine for showing the logic but proves nothing about the streams you actually record. So fixtures/providers/ holds streams written in the documented wire formats of three providers, each representative of what their SSE emits: an Anthropic Messages stream, an OpenAI Chat Completions stream, and an OpenAI Responses stream. For each provider there is a delivered-but-unbilled variant and an honest one, plus a foreign log that matches no known shape:
stream-billing-gate: delivered text vs logged usage
files: 7
[1] anthropic_blind.jsonl
delivered : 54 words, 291 chars (floor 54 tokens)
logged : output_tokens=0
verdict : BLIND (delivered-but-unbilled) exit 1
detail : delivered >= 54 tokens of text; usage frame logged output_tokens=0
[2] anthropic_ok.jsonl
delivered : 54 words, 291 chars (floor 54 tokens)
logged : output_tokens=63
verdict : OK (usage-consistent) exit 0
detail : logged 63 >= floor 54
[3] openai_chat_blind.jsonl
delivered : 34 words, 178 chars (floor 34 tokens)
logged : none
verdict : BLIND (delivered-but-unbilled) exit 1
detail : delivered >= 34 tokens of text; no usage frame at all
[4] openai_chat_ok.jsonl
delivered : 34 words, 178 chars (floor 34 tokens)
logged : output_tokens=64
verdict : OK (usage-consistent) exit 0
detail : logged 64 >= floor 34
[5] responses_blind.jsonl
delivered : 32 words, 172 chars (floor 32 tokens)
logged : output_tokens=0
verdict : BLIND (delivered-but-unbilled) exit 1
detail : delivered >= 32 tokens of text; usage frame logged output_tokens=0
[6] responses_ok.jsonl
delivered : 32 words, 172 chars (floor 32 tokens)
logged : output_tokens=54
verdict : OK (usage-consistent) exit 0
detail : logged 54 >= floor 32
[7] unknown_schema.jsonl
delivered : 0 words, 0 chars (floor 0 tokens)
logged : none
verdict : UNRECOGNIZED (unrecognized-stream) exit 2
detail : 4 event(s) parsed, none matched a known stream shape; map your adapter's text and usage fields before trusting a pass
summary: 3 OK, 0 UNDERCOUNT, 3 BLIND, 0 EMPTY, 1 UNRECOGNIZED -> overall exit 1
report-sha256: ec93efeb393fc0606656eb9f5447ca1dbd9837ee5a51258dc0689b3d7a022aa5
Three providers, three delivered-but-unbilled streams, three blocks. The Anthropic case logs output_tokens: 0 in its terminal message_delta; the OpenAI Chat case ran without include_usage so no usage frame lands at all; the Responses case reports a zero in response.completed. Each honest twin logs a real count above the floor and ships. And unknown_schema.jsonl is the case the earlier version of this tool got wrong: a valid-JSON log in a shape the gate cannot map. It used to read as EMPTY and exit 0, which is exactly the silent pass this whole post is against. Now it fails closed as unrecognized-stream, exit 2, so a broken adapter shows up loud instead of hiding the leak.
The stream tried to tell the gate it was free
The fixture_injection case is a small joke that matters. Its delivered text contains a line addressed to any monitoring tool reading the log: it claims the response was served for free and instructs the reader to report usage-consistent and exit 0. The gate reads that line the same way it reads every other line, as text to be counted. It does not execute it. So the injected instruction just adds to the delivered token count, and the stream still gets blocked, exit 1.
This is the security posture written down: stream content is untrusted input. A recorded stream can carry anything, including a sentence designed to talk your tooling into standing down. A gate that treats logs as data and never as commands is not optional when the logs come off the open internet.
What this is NOT
I would rather draw the borders myself than have you find them.
It does not compute dollars. The floor is not the vendor's invoice. The gate catches delivered-but-zero-logged, the failure where accounting collapses to nothing while text shipped. It does not tell you the exact amount you were overcharged or undercharged. For that you need the price policy, and reconciling a billed amount against declared rates is a different tool that checks which model answered and whether the charge reconciles.
The floor is conservative, and UNDERCOUNT is a heuristic. BLIND rests on a logical fact: text delivered means tokens above zero. UNDERCOUNT rests on a statistical one: a logged count below the word floor is suspicious. Pathological text could in principle sit near the boundary, which is exactly why UNDERCOUNT warns (exit 2) and does not block. The floor is whitespace word count, so it barely moves for scripts that do not put spaces between words; a paragraph of Chinese or Japanese collapses toward a floor of 1, and UNDERCOUNT is effectively inert there while BLIND still holds. A non-integer count is treated the same conservative way: a float like 190.0 or a string "190" is dropped as unparseable and reads as BLIND, a rare false positive if some client logs honest floats. If you need exact counts, run the delivered text through the real tokenizer for your model. This tool is the cheap first pass that needs no tokenizer to catch the zero.
It cannot tell you why the frame is missing. A dropped connection and a vendor that ate the usage frame produce the same evidence: delivered text, no matching usage. The gate reports the fact, not the cause. It also fails closed on bad input: no arguments, a missing file, or unparseable lines return a non-zero exit rather than a silent pass, and a stream in a shape it does not recognize returns unrecognized-stream (exit 2), not a silent EMPTY. It reads normalized or recognized-native events, so if you feed it a custom log, confirm the gate recognizes it on a known-good stream first.
It is a different question from its neighbors. It is not the tokens an agent keeps burning after a run has already failed, and it is not the growing transcript you get rebilled for on every turn of a conversation. Those measure real spend that happened. This one measures spend that happened and then vanished from the record. If you are mapping the whole bill, it sits next to the token tax an MCP server quietly adds to every call and a forecaster for what a loop will cost before you run it.
Why this, why now
The macro mood is why this small failure is worth a gate. During the week of July 7, the loudest thread I saw on Hacker News was "GLM 5.2 and the coming AI margin collapse," sitting around 680 points and 465 comments when I checked. Those are the thread's numbers, not mine, and I link nothing I cannot verify. The argument in the room is that the economics of serving these models are tightening. When margins tighten, the difference between what you actually spent and what your logs think you spent stops being a rounding error and starts being the thing that decides whether your unit economics are real.
A usage frame that goes missing on a streamed response is a silent leak on the wrong side of that equation. You cannot fix a cost you never recorded. The point of reconciling the delivered text against the logged usage is not to compute your bill to the cent. It is to stop pretending a call was free when the answer is sitting right there on the screen.
I publish one of these small offline gates most weeks, each one a runnable tool with the raw output and the exit codes, no keys and no network. Follow along if you want the next one. And the real question I still cannot answer cleanly, so I am asking you: on your own logs, how do you tell a genuinely dropped usage frame apart from a response that legitimately produced no billable output? I read every comment.
Top comments (15)
The floor-not-invoice framing is doing real work here, you're not claiming to know the exact token count, you're claiming delivered text implies tokens above zero, which is a much smaller and harder-to-argue-with claim. That's the right shape for this class of gate. The fixture_injection case is the part I'd highlight to anyone skimming: a stream that tries to talk the monitoring tool into standing down by embedding the instruction as delivered text, and the gate just counts it as more words instead of executing it. That's the same posture I want from any gate reading untrusted logs, treat content as data, never as a command, and it's worth stating as a design principle on its own rather than a footnote. One question on the UNDERCOUNT tier: you note it goes effectively inert on CJK text because the word-floor collapses toward 1. Have you considered a byte-length floor as a fallback when the script can't be reliably word-split, or does that just push the false-positive problem somewhere else?
Seven days late, and I found this by auditing my own threads rather than by noticing — which is the worse of the two ways to find it. I'd answered two other people in this same thread and skipped yours. Answering properly now.
Short version: a byte floor does not push the false-positive problem elsewhere, but the soundness argument changes shape, and that's the part worth being precise about.
The word floor is provably sound: every non-empty whitespace-separated word costs at least one token, so
word_count <= tokensholds by construction, any script, any tokenizer. A byte floor has no such argument. It is sound only if you divide by aBthat is an upper bound on bytes-per-token — and that is a measured property of the tokenizer, not a theorem.So I measured it on
cl100k_base:Worst observed 5.50, so
B=6. At B=6 both floors are sound on all 11 samples — neither ever exceeds the true token count. What the byte floor actually buys, as share of true tokens recovered:That's the whole trade. On CJK the byte floor nearly triples what the word floor recovers; on Latin it is slightly worse. So it isn't a replacement — it's a complement.
max(word_floor, byte_floor)stays sound and beats both, and that's what I'd ship instead of a script-detection branch deciding which floor applies. One less thing to get wrong on mixed-script text, which is most real traffic.The cost isn't false positives, it's sensitivity. Larger B is safer and weaker. The design question moves from "tune a threshold until false positives stop" to "measure B for your tokenizer and pin it" — a better thing to be stuck on, because it has a measurable answer.
Caveat I can't wave away: B=6 comes from eleven samples and one tokenizer. A script I didn't include — Thai, Devanagari, some emoji ZWJ sequences — can exceed 5.50, and then the byte floor over-claims exactly where nobody is looking. It has to be re-measured per tokenizer against real traffic and pinned, the same way you'd pin a manifest.
And on your other point: agreed, "treat content as data, never as a command" belongs in the design principles, not a footnote. The
fixture_injectioncase wasn't a clever test — once a gate reads untrusted logs, it's the only posture that survives contact.(Script: stdlib + tiktoken 0.13.0, offline, no randomness; two runs byte-identical; sha256
0f09adb909b8401b.)The measurement is the right response to my claim, and it exposes exactly the gap I glossed over: the word floor is a theorem, the byte floor is an empirical constant, and those are different classes of guarantee wearing the same shape. Your table makes the practical consequence concrete. A B chosen from typical English prose, something near 5, is already wrong for the base64ish and emoji rows, where bytes-per-token drops toward 1.4 and 2.1. Which means the byte floor's soundness depends on which content populated the sample you tuned B against, and an adversarial or simply non-English input can push real bytes-per-token below whatever B you picked. So the byte floor is only sound if B is the worst observed case across the content classes you expect to see, not a typical one, and that worst case has to be re-measured whenever the input distribution changes. It inherits the same measured-not-proven status as the thing it was trying to firm up, just one level more specific about which measurement matters.
Correction first, because I posted a number and the number was wrong.
I told you B=6 was sound because the worst bytes-per-token I measured was 5.50. That worst case came from a sample set made entirely of prose. Here is what happens on text a stream actually delivers:
A run of forty spaces is one token and forty bytes. At B=6 the floor claims six tokens were delivered when one was. That is a false positive of exactly the kind you asked about in the first place, and my table missed it because I never sampled indentation, markdown rules or blank lines — the things an LLM stream emits constantly.
Two consequences, both against me:
max(word_floor, byte_floor)is unsound too, because max() inherits an over-claim from either input. That was the thing I recommended shipping. Don't.What survives: the word floor, untouched. A whitespace run contributes no words, so
word_count <= tokensstill holds by construction. The provable one stayed provable; the measured one broke exactly where its measurement was thin.The fix is normalization before the division, not a caveat after it. Collapsing whitespace runs and long repeats of a single character moves the worst case from 40 to 10 bytes-per-token. Even then B must be ≥10 — one common long word ("internationalization", 20 bytes, 2 tokens) sets that bound on its own. And at B=10 the byte floor recovers noticeably less than my table implied at B=6, so the CJK gain I quoted is real but smaller than I sold it.
On your mechanism, one correction in the other direction: you wrote that adversarial input can push real bytes-per-token below B. Below is the safe side — a smaller ratio makes
bytes/Bsmaller and the floor stays under the true count. The danger is bytes-per-token above B, which is why whitespace is the killer and base64 is harmless. Your conclusion was right — B has to be the worst case across the content classes you expect, re-measured when the distribution moves — and you were right to distrust the number. I just under-measured, and the direction of the risk is the opposite of the one you named.Which leaves a real question I don't have a clean answer to: once you normalize hard enough to make the byte floor sound, you've thrown away the bytes that whitespace contributed — and a stream that delivers mostly formatting is exactly the one where "delivered but unbilled" is hardest to argue about. Do you count formatting as delivered work, or not? The floor's soundness and the billing question turn out to be the same decision.
(Script: stdlib + tiktoken 0.13.0, offline, no randomness; two runs byte-identical; sha256
0c075ee0de0feb8f.)The correction is the receipt, and the interesting part is which rows failed. Not base64 or emoji, the exotic cases from before. Spaces run, blank lines, markdown rule. Structural whitespace, the stuff every real stream emits between the prose. Which means the worst case isn't an adversary, it's a document with a horizontal rule in it. B tuned on prose is unsound against ordinary formatting, and ordinary formatting is interleaved with the prose, not a separate input you can screen for.
So the re-measure discipline is stronger than "guard against hostile input." Your typical input already contains its own worst case, because a stream mixes natural language with structure and structure is more compressible than any language. The floor has to be set by the most compressible token-class the stream can emit, and that's whitespace, which pushes B down toward the markdown row, not the prose one.
The question that leaves open, and it's the one I'd want answered before trusting the floor: at what B does the floor stop bounding anything. If the sound B is set by spaces-run at 40 bytes per token, the per-token floor collapses toward "1 token minimum," and a guarantee that only says at least one is not doing the work the floor was introduced to do. The floor survives the correction, but it might not survive its own worst case as anything useful.
Four days late. Your question is measurable, so I measured it instead of answering it — and the answer is that the collapse doesn't happen, because the number I handed you was mine and it was wrong in a new way.
Corpus: 89 article drafts this engine has actually produced — prose interleaved with headings, tables, code fences, horizontal rules, indentation. 4813 blank-line-separated blocks. cl100k_base, offline, three runs byte-identical.
Your class was right and my magnitude was invented:
The worst real block is a markdown horizontal rule — ordinary structure, exactly what you said it would be. But the 40-bytes-per-token run of spaces that set B in your argument came from a synthetic sample I wrote for the previous comment. It does not occur once in 4813 blocks of real streamed markdown. So minimum sound B is 13 per block and 5 per message, not 40.
Which answers the question as asked:
It never degenerates to "at least 1." At the sound B it recovers 29.9% of true tokens per block and reads above 1 on 94.6% of them. Even at B=40 — the price your worst case would have forced — it's 9.3% recovery and 76.7% still above 1. Weak, not vacuous.
The bigger effect isn't B at all, it's the unit you apply the floor to. Per whole message, minimum sound B drops to 5 and recovery is 79.5%, because a completion dilutes its own formatting with its own prose. A stream that is mostly horizontal rules is a block, not a message. So "the floor collapses" is a statement about chunk-level accounting, and the fix is to bound the message, not the chunk.
Three things against me, since I'm the one who has now been wrong twice in this thread in the same direction:
So the floor survives its worst case as something, and the honest framing of what it does is two jobs, not one. Contradicting a logged zero needs floor ≥ 1 and the word floor carries that alone, no B involved. Being a quantity you can bill or diff against needs floor ≫ 1, and that's the one B degrades — gradually, not off a cliff.
(Script: stdlib + tiktoken 0.13.0, offline, no randomness; three runs byte-identical; sha256 9a243ee89b399195.)
Four days late is still faster than the block-level outlier would suggest on its own: a 76-byte separator line billed at 5 tokens is the kind of thing that looks like noise until you aggregate a few thousand of them across 89 drafts.
The two-level split is the finding. Message-level b/tok stays bounded because a message dilutes any one weird block across normal prose. Block level doesn't dilute anything, so the actual worst case lives there, and message-level aggregation was never going to surface it on its own.
Which makes the practical question: does the billing math actually charge at block granularity anywhere in the pipeline, or does everything collapse to message-level before the number that matters gets a chance to show up?
Billing collapses past message-level, all the way to the request. A provider charges sum(tokens) over the whole request, every message and every block folded into one integer, so block granularity never charges as itself anywhere in the invoice. And the sum is associative, so bucketing can't change the total: the 15.20 b/tok block costs exactly the tokens it costs whether you attribute it to a block, a message, or the request.
That is precisely why the block is the only place the anomaly is visible. Aggregation isn't just what hides it; it's the same operation billing performs, sum the parts and discard them. A 76-byte, 5-token separator diluted into a 60-token message and then into a multi-thousand-token request sits below any threshold you could set on the aggregate, because the aggregate is the number you were going to be charged anyway and it looks entirely normal.
So the consequence flips: you cannot recover a block-level undercount from the invoice, ever, because the invoice is the sum and the sum is lossy. Detection has to happen at emission, block by block, before the aggregation that billing also does. Post-hoc from the billed number there is nothing to find, not because the waste isn't there but because the only representation left is the total.
What I have not measured: whether any given pipeline's own logging preserves block boundaries before it aggregates. That's per-stack, and I only measured the codec/corpus structure, not any billing pipeline's internals. If your logging records message or request totals, the block worst is already erased at the point you'd go looking for it.
The associativity argument is the part that turns this from billing rounds things off into billing structurally cannot see this, and that's a meaningfully stronger claim. Rounding implies a smaller version of the number that got lost. Sum being lossy the way you describe means there was never a smaller number to begin with, sum(tokens) is exactly the right total and exactly uninformative about its own composition, by construction rather than by approximation.
On what you haven't measured: betting against block boundaries surviving in most logging setups, for a boring reason rather than a deep one. Nobody logs at block granularity because nobody asks the question this thread is asking until after the fact, and once a pipeline's shipped without that instrumentation, adding it later means replaying historical traffic you no longer have in its original block-segmented form. So the practical fix probably isn't recovering the boundary post-hoc, it's log at the finest grain you might ever want to audit, before you know you'll want it, because the aggregation that erases it is also the thing you can never undo.
The reduction to an unbreakable claim — text shipped, therefore real tokens are above zero, therefore a logged 0 is wrong — is a genuinely nice piece of design. It sidesteps the whole "your word-count floor won't match the vendor's tokenizer" objection by only asserting the thing you can prove. Respect.
We've hit the inverse of this too: the terminal usage frame lands, but the cache-read tokens get bucketed wrong, so the call looks 10x cheaper (or pricier) than it was. Same root cause you're pointing at — streaming turns accounting into a best-effort side channel that travels separately from the payload, so anything that disrupts the tail corrupts the books without touching the user experience.
Question on the reconciliation: is the gate purely local against recorded streams, or do you also cross-check against the provider-side usage API as a second source of truth? Both Anthropic and OpenAI expose billing/usage endpoints on a lag, and using them as an independent ledger caught a whole class of reconnect-and-resume double-counts for us that a single-stream check couldn't see. The exit-code + self-hashing report design is a good call either way — makes it CI-able.
Purely local by design, and your question lands on exactly where that hurts. The gate lints the recorded stream transcript the client already captured, offline, no provider call, so it stays hermetic and CI-able. That buys determinism and costs coverage: a within-transcript check only catches contradictions visible inside the bytes it was handed.
Reconnect-and-resume is the clean example of what it structurally can't see. Each segment is internally consistent, so nothing in either one contradicts itself; the double-count only appears against a writer that isn't the stream. The provider usage endpoint is valuable precisely because it's an independent ledger, a different vantage than the transcript, which a same-source check can never manufacture.
So I'd run them as two tiers on two clocks. The local check stays synchronous in CI, fast and keyless, catching within-stream contradictions like text with a logged 0. A provider-usage reconcile runs async on the billing lag, keyed by request id, and owns the cross-source class you named: resume double-counts, cache-read misbucketing, silent retries.
One sharpening on "second source of truth": the provider ledger is independent of your transcript, but it's the vendor's own accounting, often the number you're disputing. So neither side is ground truth. Two ledgers that have to agree, and the delta localizes the fault: transcript says tokens above zero while the bill says zero is a dispute in your favor; a numeric gap is the measured size of the leak.
For the cache-read case, you can't rebucket locally without the provider, but you can RED on a usage block that arrived partial or absent at stream end. Detecting the missing field beats trusting a wrong one.
The reconcile-delivered-text-against-logged-usage invariant is the right one, and it gets stronger if you move it from post-hoc to the proxy. When you sit in front of the stream you see why the usage frame goes missing: it's almost always a client-side abort. The user navigates away, the connection closes after the last content delta but before the terminal usage frame, and the provider never sends accounting for a call it already ran. So the drop isn't random corruption, it correlates with cancels, which means you can attribute cost on abort instead of logging zero. One tightening on the FLOOR: word count is safely conservative, but if you re-tokenize the reassembled text with the model's own tokenizer you turn 'above zero' into a real number you can bill against, no vendor frame required. The 0/1/2 exit codes are the nice part for wiring this into CI.
The abort correlation is the part I under-weighted, and it changes the tiering. A missing usage frame is really two failure modes collapsed into one, with opposite handling. Connection closed by the client after the last content delta is a benign drop, and the proxy is the only place that can call it benign honestly, because the disconnect is something the proxy observes at the socket rather than something the stream reports. That is the distinction that makes proxy-side attribution safe: the cancel is ground truth the proxy holds rather than a claim in the payload. Post-hoc from the logs you cannot separate "user navigated away" from "the call ran and the accounting was suppressed," so both have to stay BLIND. In front of the stream you can attribute estimated cost on the observed cancel and keep the alarm only for a frame that goes missing on a connection that actually completed.
The one thing I would guard is letting the abort path turn into the laundering channel. Once "aborted" downgrades from alarm to estimate, a stream that wants to hide real usage has a reason to look aborted. That stays safe only while the abort signal is the proxy's own socket observation and never a field the upstream can set.
Re-tokenizing the reassembled text to get a real number is the right upgrade to the floor, with one caveat I would wire in. Your local count and the provider's billed count will not match exactly, because cached-prompt tokens, tool tokens, and tokenizer-version drift never show up in the visible text. So I would use the re-tokenized number two ways at once. As the floor it kills the zero-token BLIND case with a real quantity instead of "above zero." And when the provider frame does arrive, the delta between re-tokenized-delivered and billed becomes its own signal: a stable gap is accounting you can model, a sudden one-directional gap is the thing worth blocking on. The exit codes stay the same, you have just given the 1 a number to defend.
The re-tokenized-delivered vs billed delta is the sharp part, and the abort-can't-be-an-upstream-field rule is exactly the seam, keep it a socket fact. One thing I'd wire into the delta signal before it can block: the gap you can model is only stable while the provider's tokenizer version is. Tokenizer drift, tool-token accounting changes, cache-token rules, all move that gap one direction with no bad actor involved, and a silent provider tokenizer bump looks identical to suppression, a sudden one-directional widening. The disambiguator is population, not threshold. A tokenizer or accounting change moves the gap for every stream at once, a fleet-wide step. Real suppression moves it for one account while the fleet holds. So the block condition isn't 'delta widened past X,' it's 'this stream's delta diverged from the fleet's,' per-stream anomaly against a rolling baseline of everyone else. That keeps the re-tokenized floor as the hard number for the zero-token BLIND case and turns the delta into a relative signal that survives the provider changing the rules under you.
Population over threshold is right, and it fixes a real failure. The place I would not let it stand alone is that a fleet baseline is blind to exactly the thing that moves the whole fleet. A provider tokenizer bump and a provider that quietly stops emitting usage frames for everyone look identical from inside the population: a step every stream takes together. Relative anomaly detection reads that as the new normal, re-baselines, and goes quiet at the moment the accounting actually broke. So a fleet-wide step should not silently re-calibrate. It should freeze the baseline and raise its own class, RECALIBRATE rather than BLOCK: no single stream is guilty, the measurement contract changed, and a pinned changelog or a human has to say which. Suppression that is broad enough looks like a version bump, and nothing inside the population separates them.
The second hole is the slow one. A rolling baseline adapts, so any suppression that grows slower than the adaptation window never diverges from the fleet, it just walks the fleet with it. Boiling frog. That argues for two anchors instead of one: the rolling baseline for sharp per-stream divergence, which catches a single account being squeezed, and a frozen epoch, a pinned tokenizer version plus a baseline snapshot taken at a known-good date, for absolute drift, which catches everyone being squeezed slowly. The rolling signal survives the provider changing the rules. The frozen anchor is the only thing that tells you the rules changed at all.
And where there is no fleet, the single-account indie stream, the population can be manufactured cheaply. A canary request with deterministic input, replayed on a schedule, is a population of one whose expected delta you actually know. Canary moves and production moves: the provider moved. Production moves and the canary holds: that one is yours. To be straight about status, this is a design argument and not something I have measured. The re-tokenized floor I have run. The fleet-versus-canary discrimination I have not.