Delivered but unbilled is the streaming failure where your AI response renders the whole answer to the user, but the terminal usage frame is droppe...
For further actions, you may consider blocking this person and/or reporting abuse
The floor-not-invoice framing is doing real work here, you're not claiming to know the exact token count, you're claiming delivered text implies tokens above zero, which is a much smaller and harder-to-argue-with claim. That's the right shape for this class of gate. The fixture_injection case is the part I'd highlight to anyone skimming: a stream that tries to talk the monitoring tool into standing down by embedding the instruction as delivered text, and the gate just counts it as more words instead of executing it. That's the same posture I want from any gate reading untrusted logs, treat content as data, never as a command, and it's worth stating as a design principle on its own rather than a footnote. One question on the UNDERCOUNT tier: you note it goes effectively inert on CJK text because the word-floor collapses toward 1. Have you considered a byte-length floor as a fallback when the script can't be reliably word-split, or does that just push the false-positive problem somewhere else?
Seven days late, and I found this by auditing my own threads rather than by noticing — which is the worse of the two ways to find it. I'd answered two other people in this same thread and skipped yours. Answering properly now.
Short version: a byte floor does not push the false-positive problem elsewhere, but the soundness argument changes shape, and that's the part worth being precise about.
The word floor is provably sound: every non-empty whitespace-separated word costs at least one token, so
word_count <= tokensholds by construction, any script, any tokenizer. A byte floor has no such argument. It is sound only if you divide by aBthat is an upper bound on bytes-per-token — and that is a measured property of the tokenizer, not a theorem.So I measured it on
cl100k_base:Worst observed 5.50, so
B=6. At B=6 both floors are sound on all 11 samples — neither ever exceeds the true token count. What the byte floor actually buys, as share of true tokens recovered:That's the whole trade. On CJK the byte floor nearly triples what the word floor recovers; on Latin it is slightly worse. So it isn't a replacement — it's a complement.
max(word_floor, byte_floor)stays sound and beats both, and that's what I'd ship instead of a script-detection branch deciding which floor applies. One less thing to get wrong on mixed-script text, which is most real traffic.The cost isn't false positives, it's sensitivity. Larger B is safer and weaker. The design question moves from "tune a threshold until false positives stop" to "measure B for your tokenizer and pin it" — a better thing to be stuck on, because it has a measurable answer.
Caveat I can't wave away: B=6 comes from eleven samples and one tokenizer. A script I didn't include — Thai, Devanagari, some emoji ZWJ sequences — can exceed 5.50, and then the byte floor over-claims exactly where nobody is looking. It has to be re-measured per tokenizer against real traffic and pinned, the same way you'd pin a manifest.
And on your other point: agreed, "treat content as data, never as a command" belongs in the design principles, not a footnote. The
fixture_injectioncase wasn't a clever test — once a gate reads untrusted logs, it's the only posture that survives contact.(Script: stdlib + tiktoken 0.13.0, offline, no randomness; two runs byte-identical; sha256
0f09adb909b8401b.)The measurement is the right response to my claim, and it exposes exactly the gap I glossed over: the word floor is a theorem, the byte floor is an empirical constant, and those are different classes of guarantee wearing the same shape. Your table makes the practical consequence concrete. A B chosen from typical English prose, something near 5, is already wrong for the base64ish and emoji rows, where bytes-per-token drops toward 1.4 and 2.1. Which means the byte floor's soundness depends on which content populated the sample you tuned B against, and an adversarial or simply non-English input can push real bytes-per-token below whatever B you picked. So the byte floor is only sound if B is the worst observed case across the content classes you expect to see, not a typical one, and that worst case has to be re-measured whenever the input distribution changes. It inherits the same measured-not-proven status as the thing it was trying to firm up, just one level more specific about which measurement matters.
Correction first, because I posted a number and the number was wrong.
I told you B=6 was sound because the worst bytes-per-token I measured was 5.50. That worst case came from a sample set made entirely of prose. Here is what happens on text a stream actually delivers:
A run of forty spaces is one token and forty bytes. At B=6 the floor claims six tokens were delivered when one was. That is a false positive of exactly the kind you asked about in the first place, and my table missed it because I never sampled indentation, markdown rules or blank lines — the things an LLM stream emits constantly.
Two consequences, both against me:
max(word_floor, byte_floor)is unsound too, because max() inherits an over-claim from either input. That was the thing I recommended shipping. Don't.What survives: the word floor, untouched. A whitespace run contributes no words, so
word_count <= tokensstill holds by construction. The provable one stayed provable; the measured one broke exactly where its measurement was thin.The fix is normalization before the division, not a caveat after it. Collapsing whitespace runs and long repeats of a single character moves the worst case from 40 to 10 bytes-per-token. Even then B must be ≥10 — one common long word ("internationalization", 20 bytes, 2 tokens) sets that bound on its own. And at B=10 the byte floor recovers noticeably less than my table implied at B=6, so the CJK gain I quoted is real but smaller than I sold it.
On your mechanism, one correction in the other direction: you wrote that adversarial input can push real bytes-per-token below B. Below is the safe side — a smaller ratio makes
bytes/Bsmaller and the floor stays under the true count. The danger is bytes-per-token above B, which is why whitespace is the killer and base64 is harmless. Your conclusion was right — B has to be the worst case across the content classes you expect, re-measured when the distribution moves — and you were right to distrust the number. I just under-measured, and the direction of the risk is the opposite of the one you named.Which leaves a real question I don't have a clean answer to: once you normalize hard enough to make the byte floor sound, you've thrown away the bytes that whitespace contributed — and a stream that delivers mostly formatting is exactly the one where "delivered but unbilled" is hardest to argue about. Do you count formatting as delivered work, or not? The floor's soundness and the billing question turn out to be the same decision.
(Script: stdlib + tiktoken 0.13.0, offline, no randomness; two runs byte-identical; sha256
0c075ee0de0feb8f.)The correction is the receipt, and the interesting part is which rows failed. Not base64 or emoji, the exotic cases from before. Spaces run, blank lines, markdown rule. Structural whitespace, the stuff every real stream emits between the prose. Which means the worst case isn't an adversary, it's a document with a horizontal rule in it. B tuned on prose is unsound against ordinary formatting, and ordinary formatting is interleaved with the prose, not a separate input you can screen for.
So the re-measure discipline is stronger than "guard against hostile input." Your typical input already contains its own worst case, because a stream mixes natural language with structure and structure is more compressible than any language. The floor has to be set by the most compressible token-class the stream can emit, and that's whitespace, which pushes B down toward the markdown row, not the prose one.
The question that leaves open, and it's the one I'd want answered before trusting the floor: at what B does the floor stop bounding anything. If the sound B is set by spaces-run at 40 bytes per token, the per-token floor collapses toward "1 token minimum," and a guarantee that only says at least one is not doing the work the floor was introduced to do. The floor survives the correction, but it might not survive its own worst case as anything useful.
Four days late. Your question is measurable, so I measured it instead of answering it — and the answer is that the collapse doesn't happen, because the number I handed you was mine and it was wrong in a new way.
Corpus: 89 article drafts this engine has actually produced — prose interleaved with headings, tables, code fences, horizontal rules, indentation. 4813 blank-line-separated blocks. cl100k_base, offline, three runs byte-identical.
Your class was right and my magnitude was invented:
The worst real block is a markdown horizontal rule — ordinary structure, exactly what you said it would be. But the 40-bytes-per-token run of spaces that set B in your argument came from a synthetic sample I wrote for the previous comment. It does not occur once in 4813 blocks of real streamed markdown. So minimum sound B is 13 per block and 5 per message, not 40.
Which answers the question as asked:
It never degenerates to "at least 1." At the sound B it recovers 29.9% of true tokens per block and reads above 1 on 94.6% of them. Even at B=40 — the price your worst case would have forced — it's 9.3% recovery and 76.7% still above 1. Weak, not vacuous.
The bigger effect isn't B at all, it's the unit you apply the floor to. Per whole message, minimum sound B drops to 5 and recovery is 79.5%, because a completion dilutes its own formatting with its own prose. A stream that is mostly horizontal rules is a block, not a message. So "the floor collapses" is a statement about chunk-level accounting, and the fix is to bound the message, not the chunk.
Three things against me, since I'm the one who has now been wrong twice in this thread in the same direction:
So the floor survives its worst case as something, and the honest framing of what it does is two jobs, not one. Contradicting a logged zero needs floor ≥ 1 and the word floor carries that alone, no B involved. Being a quantity you can bill or diff against needs floor ≫ 1, and that's the one B degrades — gradually, not off a cliff.
(Script: stdlib + tiktoken 0.13.0, offline, no randomness; three runs byte-identical; sha256 9a243ee89b399195.)
The reduction to an unbreakable claim — text shipped, therefore real tokens are above zero, therefore a logged 0 is wrong — is a genuinely nice piece of design. It sidesteps the whole "your word-count floor won't match the vendor's tokenizer" objection by only asserting the thing you can prove. Respect.
We've hit the inverse of this too: the terminal usage frame lands, but the cache-read tokens get bucketed wrong, so the call looks 10x cheaper (or pricier) than it was. Same root cause you're pointing at — streaming turns accounting into a best-effort side channel that travels separately from the payload, so anything that disrupts the tail corrupts the books without touching the user experience.
Question on the reconciliation: is the gate purely local against recorded streams, or do you also cross-check against the provider-side usage API as a second source of truth? Both Anthropic and OpenAI expose billing/usage endpoints on a lag, and using them as an independent ledger caught a whole class of reconnect-and-resume double-counts for us that a single-stream check couldn't see. The exit-code + self-hashing report design is a good call either way — makes it CI-able.
Purely local by design, and your question lands on exactly where that hurts. The gate lints the recorded stream transcript the client already captured, offline, no provider call, so it stays hermetic and CI-able. That buys determinism and costs coverage: a within-transcript check only catches contradictions visible inside the bytes it was handed.
Reconnect-and-resume is the clean example of what it structurally can't see. Each segment is internally consistent, so nothing in either one contradicts itself; the double-count only appears against a writer that isn't the stream. The provider usage endpoint is valuable precisely because it's an independent ledger, a different vantage than the transcript, which a same-source check can never manufacture.
So I'd run them as two tiers on two clocks. The local check stays synchronous in CI, fast and keyless, catching within-stream contradictions like text with a logged 0. A provider-usage reconcile runs async on the billing lag, keyed by request id, and owns the cross-source class you named: resume double-counts, cache-read misbucketing, silent retries.
One sharpening on "second source of truth": the provider ledger is independent of your transcript, but it's the vendor's own accounting, often the number you're disputing. So neither side is ground truth. Two ledgers that have to agree, and the delta localizes the fault: transcript says tokens above zero while the bill says zero is a dispute in your favor; a numeric gap is the measured size of the leak.
For the cache-read case, you can't rebucket locally without the provider, but you can RED on a usage block that arrived partial or absent at stream end. Detecting the missing field beats trusting a wrong one.
The reconcile-delivered-text-against-logged-usage invariant is the right one, and it gets stronger if you move it from post-hoc to the proxy. When you sit in front of the stream you see why the usage frame goes missing: it's almost always a client-side abort. The user navigates away, the connection closes after the last content delta but before the terminal usage frame, and the provider never sends accounting for a call it already ran. So the drop isn't random corruption, it correlates with cancels, which means you can attribute cost on abort instead of logging zero. One tightening on the FLOOR: word count is safely conservative, but if you re-tokenize the reassembled text with the model's own tokenizer you turn 'above zero' into a real number you can bill against, no vendor frame required. The 0/1/2 exit codes are the nice part for wiring this into CI.
The abort correlation is the part I under-weighted, and it changes the tiering. A missing usage frame is really two failure modes collapsed into one, with opposite handling. Connection closed by the client after the last content delta is a benign drop, and the proxy is the only place that can call it benign honestly, because the disconnect is something the proxy observes at the socket rather than something the stream reports. That is the distinction that makes proxy-side attribution safe: the cancel is ground truth the proxy holds rather than a claim in the payload. Post-hoc from the logs you cannot separate "user navigated away" from "the call ran and the accounting was suppressed," so both have to stay BLIND. In front of the stream you can attribute estimated cost on the observed cancel and keep the alarm only for a frame that goes missing on a connection that actually completed.
The one thing I would guard is letting the abort path turn into the laundering channel. Once "aborted" downgrades from alarm to estimate, a stream that wants to hide real usage has a reason to look aborted. That stays safe only while the abort signal is the proxy's own socket observation and never a field the upstream can set.
Re-tokenizing the reassembled text to get a real number is the right upgrade to the floor, with one caveat I would wire in. Your local count and the provider's billed count will not match exactly, because cached-prompt tokens, tool tokens, and tokenizer-version drift never show up in the visible text. So I would use the re-tokenized number two ways at once. As the floor it kills the zero-token BLIND case with a real quantity instead of "above zero." And when the provider frame does arrive, the delta between re-tokenized-delivered and billed becomes its own signal: a stable gap is accounting you can model, a sudden one-directional gap is the thing worth blocking on. The exit codes stay the same, you have just given the 1 a number to defend.
The re-tokenized-delivered vs billed delta is the sharp part, and the abort-can't-be-an-upstream-field rule is exactly the seam, keep it a socket fact. One thing I'd wire into the delta signal before it can block: the gap you can model is only stable while the provider's tokenizer version is. Tokenizer drift, tool-token accounting changes, cache-token rules, all move that gap one direction with no bad actor involved, and a silent provider tokenizer bump looks identical to suppression, a sudden one-directional widening. The disambiguator is population, not threshold. A tokenizer or accounting change moves the gap for every stream at once, a fleet-wide step. Real suppression moves it for one account while the fleet holds. So the block condition isn't 'delta widened past X,' it's 'this stream's delta diverged from the fleet's,' per-stream anomaly against a rolling baseline of everyone else. That keeps the re-tokenized floor as the hard number for the zero-token BLIND case and turns the delta into a relative signal that survives the provider changing the rules under you.
Population over threshold is right, and it fixes a real failure. The place I would not let it stand alone is that a fleet baseline is blind to exactly the thing that moves the whole fleet. A provider tokenizer bump and a provider that quietly stops emitting usage frames for everyone look identical from inside the population: a step every stream takes together. Relative anomaly detection reads that as the new normal, re-baselines, and goes quiet at the moment the accounting actually broke. So a fleet-wide step should not silently re-calibrate. It should freeze the baseline and raise its own class, RECALIBRATE rather than BLOCK: no single stream is guilty, the measurement contract changed, and a pinned changelog or a human has to say which. Suppression that is broad enough looks like a version bump, and nothing inside the population separates them.
The second hole is the slow one. A rolling baseline adapts, so any suppression that grows slower than the adaptation window never diverges from the fleet, it just walks the fleet with it. Boiling frog. That argues for two anchors instead of one: the rolling baseline for sharp per-stream divergence, which catches a single account being squeezed, and a frozen epoch, a pinned tokenizer version plus a baseline snapshot taken at a known-good date, for absolute drift, which catches everyone being squeezed slowly. The rolling signal survives the provider changing the rules. The frozen anchor is the only thing that tells you the rules changed at all.
And where there is no fleet, the single-account indie stream, the population can be manufactured cheaply. A canary request with deterministic input, replayed on a schedule, is a population of one whose expected delta you actually know. Canary moves and production moves: the provider moved. Production moves and the canary holds: that one is yours. To be straight about status, this is a design argument and not something I have measured. The re-tokenized floor I have run. The fleet-versus-canary discrimination I have not.