A vendor publishes a safety stat: 1.2 million checks, 80 quarantined. It reads like a measurement. It is half of one.
The missing half is the population that mattered. How many bad requests actually arrived? How many unsafe tool calls went through? How many workflows completed cleanly because nothing fired? The public record cannot answer any of that, because the record format was never built to hold the other side.
A block is an event. Something stopped, a reason code got written, a quarantine object landed in storage, and a review status may show up later. An allow is ordinary traffic. It looks exactly like the request that should have been allowed, which is the whole problem. The system writes down catches because catches interrupt execution, and it loses misses because a miss and a success are byte-identical from the logger's point of view.
That is not a vendor being cagey. That is the guard's architecture deciding, in advance, which side of the ledger will ever be legible.
The catch ledger audits itself
False positives have a convenient property: they begin life as catches.
Quarantine something and later release it, and the reversal attaches to the same record the original decision created. The guard fired. An artifact exists. A second decision changed its status. The wrongness stays bolted to a visible object, which means it can be counted:
false positive rate among quarantines =
released quarantines / total quarantines
Eighty quarantines, twelve of them later cleared, and the catch-side false positive rate is 15 percent. Both decisions happened on the eventful side of the system, so both are in the log.
Trust API designs tend to get this backwards. They publish blocked-request examples as proof the thing works, then treat reversals as an implementation detail to be summarized away. Once quarantine decisions are public, the reversal rate is already part of the public measurement surface. Suppressing it does not protect the number. It just makes the number look unfinished.
The limit here is easy to miss, though. A reversal rate says how often visible catches were wrong. It says nothing about how often invisible passes were wrong.
The miss ledger was never written
A false negative is an allowed item that later turns out to have been bad. Getting one requires the allowed population and, separately, ground truth that only arrives after the fact.
The catch ledger has neither. It may carry an aggregate count of allowed requests, the 1.2 million, which is a load metric wearing a safety costume. You cannot estimate recall from a counter. Recall needs the individual allowed items, retained well enough that someone can go back later and decide whether letting each one through was correct.
Consider two guards. One catches 80 out of 100 bad requests. The other catches 80 out of 40,000. Both publish a catch count of 80, and both quarantine pages look identical. One of those systems is strong and the other is mostly decorative, and no amount of reading the public record will tell you which is which.
Scale does not rescue this. Ten million checks with 800 quarantines is a bigger numerator. Events on the catch side do not reconstruct the allow side, at any volume, ever.
Agent infrastructure makes it worse, because allowed actions detonate late. A tool call can succeed now and only look wrong after it has moved money or minted a credential that gets used three weeks later. If the request was never retained with enough context to re-evaluate it, the miss degrades into forensic folklore: somebody remembers an incident, nobody can place it in a sample frame.
The same accounting bias shows up well outside content filtering. A rate limiter logs throttles while the allowed calls vanish into ordinary service metrics, so an abusive pattern that stayed politely under the threshold leaves no trace in the throttle count. A spend cap records its kills, with workflow id and configured ceiling and consumed budget and a reason string, and records nothing at all about the slow burn it slept through. A permission check logs denials, which is security theater with paperwork if the grants are never inspected; a stale capability can let an agent read a resource for six months while the denial log stays impressively busy. A reviewer model's flagged findings become a labeled dataset, and its silent passes become a throughput counter.
Every one of those is a defensible engineering choice. Storing the negative decision is cheap and storing every positive one is not. It stops being defensible the moment the catch count gets published as a trust claim.
A denominator you can check has to be committed to in advance
Publishing the whole allow side is not an option. It is enormous, and it is full of exactly the material that makes it worth having: customer state, retrieved documents, tool arguments, prompts. A complete public allow log would be its own incident report.
So publish a sample. The sample is worth nothing unless the selection rule was fixed before the outcomes were known, and that condition carries the entire measurement. Choose which allowed requests to publish after seeing which ones aged well and you have produced curated evidence formatted as a rate.
Make the rule mechanical:
For every allowed request, compute a digest over the canonical request envelope.
If the digest ends in 0x00, retain the full audit bundle.
After a 30-day outcome window, publish the retained item with its final label.
One byte of suffix is roughly 1 in 256. Two bytes is 1 in 65,536. Declare which one before the measurement window opens.
The envelope definition does real work here, and it is the easiest place to cheat. It has to cover what the guard actually saw: normalized content, tool target, policy version, caller class, context hash, decision result. Hash a convenient subset and the rule becomes steerable through whatever fields you left outside the digest.
Redaction is fine. Redaction that changes membership is not. An item qualifies the moment its digest qualifies, and what gets blacked out before publication is a separate step that happens strictly afterward.
Outside verification stays partial and is still worth having. A verifier can confirm that every published item satisfies the predicate, and can submit requests whose digests it computed itself, then check later whether the qualifying ones showed up. Absence turns into a question somebody can ask out loud. That is the point of a hash predicate: the publisher cannot widen its luck retroactively, because eligibility is a function of the request rather than of how the request turned out. Switching the rule from 0x00 to 0x01 after a bad month is visible when the original rule was registered.
Underneath this is a pattern worth stating plainly. A declared parameter is honest only when the same number costs you something else in the same computation. Widen a tolerance and lose the headroom you were claiming in the same breath, and the tolerance stops being a free knob. Sample by a predicate you cannot move, and you cannot improve the sample by picking outcomes.
The tax has to be local to survive. A footnote promising a representative sample costs nothing. A methodology document nobody can bind you to costs nothing.
What you get back is uncomfortable and useful. Once a fraction of allowed traffic is auditable, silent passes stop being pure throughput and start carrying a future measurement liability. One in 256 is plenty to put an interval around a miss rate, and enough to tell two policy versions apart when one of them is quietly worse. The published number will be less flattering than a catch count. Good. A rate that has touched the allow side beats a large integer from the quarantine table.
The hole that stays open
Selection is fixed. Completeness is not.
Nothing in the published sample proves the sample was drawn from the real traffic. The same system produces the traffic log, applies the predicate, and publishes what survives. An outsider can check internal consistency all day and still cannot tell whether the population was trimmed before the predicate ever ran.
Closing that needs a witness on the ingest path: a second signer that sees requests before the sampling decision and can attest that the sampled population came from everything that arrived. Which moves the problem instead of solving it. That witness now publishes a number about its own coverage, and you have no way to compute its denominator either.
Top comments (0)