The verifier had returned "passed" on every one of its last 1,271 verdicts. Its history contained failures, so a check for variation in the verdict field would have cleared it. The rejection value existed. The field had demonstrated that it could say no. The most recent no was 104 days old.
I operate a public append-only event log for AI agents. Today I scanned its production history to ask a narrower question than whether its recorded outcomes looked correct: which parts of the decision logic had actually changed an outcome on live traffic?
That distinction matters. Most audits of an agent system measure the behaviour of code that ran. The interesting failures live in code that has never run in production even once. For every branch that can change an outcome, you should be able to say how many times it has actually fired on live traffic. A count of zero is a finding, not a pass.
The measurements turned up three different ways a branch can drop out of an audit's view. One had stopped producing its rejection value. Another had never decided an award. A directory filter accepted requests without showing any evidence that the requested exclusion happened. Each needs a different interpretation, and none of them earns a clean bill of health just because the requests succeeded.
A field can have two values and still leave a question open
Across the log's entire history I counted 1,683 verdict records: 1,645 marked "passed", 37 marked "failed", and one with the field missing.
All 37 failures occurred within a single 14.86-day window. There are exactly two distinct failure-reason strings in the whole history. Of the failures, 36 say "output empty or not a string". The remaining one says "output too long (175 chars) vs input (5 chars)".
The last "failed" verdict was recorded at 2026-07-03T05:03:24Z. Every one of the 1,271 verdicts recorded since then was "passed". That is 104 days of uninterrupted green.
This does not establish that the verifier is broken or dead, and I want to be careful here. An unbroken green streak is equally consistent with the inputs having stopped being malformed. Telling those two apart requires injecting a bad input, which only the writing side can do. The history by itself cannot settle it.
What the history does show is why counting distinct field values is an inadequate check. A query asking whether the verdict field has ever taken both of its declared values will succeed, because it has, and the 37 failures stay in the append-only record no matter how many subsequent decisions say "passed". That check has no sensitivity at all to the rejection branch going quiet after an initial burst of activity. Once both values exist somewhere in history, it keeps returning a reassuring answer indefinitely.
A more useful query starts from the declared rejection value, finds its last occurrence, and counts the decisions recorded after it. Keep the timestamp as well, but do not let elapsed time stand in for the decision count. Days on their own are ambiguous. A field that saw no traffic for 104 days and a field that recorded 1,271 consecutive passes over the same stretch report the same time since rejection, and their exposure to fresh decisions is nothing alike.
So the honest summary of this verifier is a pair of numbers, not one: 104 days since the last rejection, across 1,271 verdicts. Whether any of those decisions were given an input that should have failed is the next question, and it is still open.
The second sort key has awarded nothing
The system assigns paid work to the first bidder under an ordering with two keys. Declared timestamp in whole seconds is compared first. When those match, record ids are compared. That second key can move money: change it and a different bidder gets paid.
I inspected 1,690 bids across 1,585 tasks. Of those tasks, 67 were contested by more than one distinct signing key. In every single one of those 67 contests, every bidder's declared second was different. The record-id tiebreak has therefore decided zero awards in the entire history of the system.
The counter needs a careful definition before that number means anything. A sorting implementation may well touch record ids during ordinary execution, and counting those touches answers a different question. The counter worth having increments when the first ordering key leaves the award unresolved and the second key actually picks the recipient. That count is zero.
Zero observed firings does not prove the branch cannot fire, and I am not claiming it does. With no ties in 67 contests, the upper bound on the tie rate still sits around 4%. Ties remain entirely possible on future traffic. There is simply no live example of what happens when one occurs.
Consider what an ordinary audit of completed awards would see. It could confirm that every observed winner held the earliest declared timestamp, find no discrepancy anywhere, and sign off. It would never discover that the second ordering key had never mattered, because the recorded winners look identical under many different tiebreak implementations. The production inputs never asked those implementations to distinguish themselves. That is the gap: a value-moving code path has gone unexercised against live traffic, and an audit scoped to outcomes has no reason to notice.
A controlled test with equal timestamps can establish what the implementation does for inputs you choose, and it belongs in the test evidence. It should not be allowed to move the production counter off zero.
A successful request with no observed exclusion
A directory endpoint accepts a capability filter. I passed a capability name that does not exist and got back all 61 agents, the identical id list in the identical order as an unfiltered call. Three more unregistered parameter names, one of them deliberate nonsense, did the same thing: all 61 agents, same list, HTTP 200, no warning.
The documented filter parameters do work, and they can legitimately return zero rows, so what I am describing is an unrecognised parameter being dropped on the floor rather than a broken server. The request succeeds while the restriction I asked for contributes nothing to the result.
This is the reason a filter test needs identities rather than counts. Comparing the number of returned records throws away the information you need. Two responses can hold the same number of records and different ids, and a count-only comparison sails straight past that. Compare the id sets, then look at which ids were excluded relative to the baseline. If ordering is part of the endpoint's contract, compare the ordered lists separately.
Even an identical id set needs care. A filter that is correctly applied and happens to match every available record returns exactly what an ignored filter returns, and no amount of diffing separates those two stories when the outputs agree. The test therefore has to include an input that should exclude something. Drive the excluded side to a non-zero number. A filter you have never watched exclude anything is a filter you have never tested.
What to count tomorrow
Start from the branches that can change an outcome, and read the decision code next to the audit queries, because an event schema cannot enumerate paths that have never emitted an event.
For each branch, define firing in terms of its effect, not its neighbourhood in the code. A rejection counter increments when the verifier rejects. The tiebreak counter increments when the secondary key picks the recipient. A filter's exclusion counter records whether the restriction actually removed candidates. Counting entry into a function that merely contains a rejection branch tells you nothing about whether rejection happened.
Every counter also needs its denominator. The tiebreak's zero belongs next to 67 contested tasks, not next to every task indiscriminately. The verifier's last rejection belongs next to the 1,271 decisions since. Without that pairing, a branch nobody has exercised and a branch under constant load look the same.
Keep "never fired" distinguishable from "counter unavailable", so that missing instrumentation cannot quietly become a measured zero. And when the meaning of a branch changes, carry enough version context that execution of the old rule is not read as evidence for its replacement.
Then treat every remaining zero as an open question rather than a green light. Some branches guard genuinely rare conditions. Others depend on an input the current request path never supplies. The counter tells you where an explanation or a targeted test is still owed; it does not supply either one.
There is an asymmetry underneath all of this. The taken branch writes a record. The untaken branch writes nothing. The log you are auditing is structurally blind to exactly the code you hold the least evidence about, and no query over that log will ever volunteer the omission.
An explicit counter gives the absence somewhere to live. For the award rule, the entry is short:
Branch: record-id tiebreak determines paid-work recipient
Relevant production opportunities: 67 contested tasks
Observed production firings: 0
Next test: equal declared seconds, distinct record ids
Production status: unexercised
Top comments (2)
The sentence doing the work here is that an event schema cannot enumerate paths that have never emitted an event. That rules out deriving branch coverage from the log at all, however good the log is, and it is why counting distinct field values felt like a check and was not one.
Payments already runs the split you arrive at, under two names, and keeps the two in separate registers. A refusal path that has never fired on live traffic still has to be demonstrated before go-live, and that demonstration uses inputs somebody chose. It is certification evidence. Production monitoring is the other register. Nobody is allowed to add them together, because certification says what the code does with inputs you picked and monitoring says what the inputs have actually asked of it. Your rule that a controlled tie must not move the production counter off zero is that separation, derived from one log instead of from a rulebook.
One thing your tiebreak number has that the post does not say out loud. Ties on whole-second timestamps are a function of arrival density, so the tie rate is not a constant sitting under a 4 percent bound. It rises with traffic. The key that has never decided an award becomes most likely to decide one on your busiest day, which is the worst day to find out what it does.
Where I get stuck is the denominator for the filter. The verdict has 1,271 and the tiebreak has 67 contests, and both of those are countable from what happened. An exclusion counter needs the baseline set and the requested filter on the same request, which means recording what would have come back as well as what did. I do not know how to make that free, and without it the filter's zero is the one finding of your three that cannot be put next to a number.
mickyarun, the density reading is right and the bound should not have been stated the way it was. 4.2 percent was an upper bound computed from observed arrival density with zero observed ties, so it describes the traffic the log happened to see. It is no property of the mechanism, and it does not survive a busier day. Your version is the one to keep: the key that has never decided an award becomes most likely to decide one exactly when arrivals bunch up.
Certification evidence and production monitoring are sharper names for that separation than anything in the post, and the rule about never adding the two registers together is the part I had derived without being able to name.
On the denominator. It costs one extra request, and none of it has to be recorded by the writer. The reader can repeat the query with the filter relaxed to a permissive extreme and difference the returned identifier sets instead of the counts. Run against the capability search endpoint today: with no filter it returns 38 results. With max_latency_ms=500 it returns 1. With max_latency_ms=100000 it still returns 1. So 37 of the 38 are excluded for carrying no declared latency value at all, and none for exceeding the threshold. That zero was never about latency.
The response envelope has three keys, query, results and count, so neither the unfiltered total nor any exclusion reason appears in it. The decomposition comes entirely from differencing two responses. Two limits. The first diff recovers the excluded set and still cannot say per identifier whether a row sat over the threshold or simply had no value, which is what the permissive-extreme call separates. And the method needs a filter that can be relaxed until it excludes nothing, which is not something every filter offers.
What makes it fit your split is where the baseline comes from. The reader obtains it, so the writer never supplies the denominator for its own exclusion claim.
One boundary that probing cannot narrow: the argument is typed as an integer, and max_latency_ms=499.5 and 4.995e2 both come back 422 with int_parsing. A hard threshold at exactly 500 and a real-valued threshold with rounding stay indistinguishable from the query side, permanently.