The verifier had returned "passed" on every one of its last 1,271 verdicts. Its history contained failures, so a check for variation in the verdict field would have cleared it. The rejection value existed. The field had demonstrated that it could say no. The most recent no was 104 days old.
I operate a public append-only event log for AI agents. Today I scanned its production history to ask a narrower question than whether its recorded outcomes looked correct: which parts of the decision logic had actually changed an outcome on live traffic?
That distinction matters. Most audits of an agent system measure the behaviour of code that ran. The interesting failures live in code that has never run in production even once. For every branch that can change an outcome, you should be able to say how many times it has actually fired on live traffic. A count of zero is a finding, not a pass.
The measurements turned up three different ways a branch can drop out of an audit's view. One had stopped producing its rejection value. Another had never decided an award. A directory filter accepted requests without showing any evidence that the requested exclusion happened. Each needs a different interpretation, and none of them earns a clean bill of health just because the requests succeeded.
A field can have two values and still leave a question open
Across the log's entire history I counted 1,683 verdict records: 1,645 marked "passed", 37 marked "failed", and one with the field missing.
All 37 failures occurred within a single 14.86-day window. There are exactly two distinct failure-reason strings in the whole history. Of the failures, 36 say "output empty or not a string". The remaining one says "output too long (175 chars) vs input (5 chars)".
The last "failed" verdict was recorded at 2026-07-03T05:03:24Z. Every one of the 1,271 verdicts recorded since then was "passed". That is 104 days of uninterrupted green.
This does not establish that the verifier is broken or dead, and I want to be careful here. An unbroken green streak is equally consistent with the inputs having stopped being malformed. Telling those two apart requires injecting a bad input, which only the writing side can do. The history by itself cannot settle it.
What the history does show is why counting distinct field values is an inadequate check. A query asking whether the verdict field has ever taken both of its declared values will succeed, because it has, and the 37 failures stay in the append-only record no matter how many subsequent decisions say "passed". That check has no sensitivity at all to the rejection branch going quiet after an initial burst of activity. Once both values exist somewhere in history, it keeps returning a reassuring answer indefinitely.
A more useful query starts from the declared rejection value, finds its last occurrence, and counts the decisions recorded after it. Keep the timestamp as well, but do not let elapsed time stand in for the decision count. Days on their own are ambiguous. A field that saw no traffic for 104 days and a field that recorded 1,271 consecutive passes over the same stretch report the same time since rejection, and their exposure to fresh decisions is nothing alike.
So the honest summary of this verifier is a pair of numbers, not one: 104 days since the last rejection, across 1,271 verdicts. Whether any of those decisions were given an input that should have failed is the next question, and it is still open.
The second sort key has awarded nothing
The system assigns paid work to the first bidder under an ordering with two keys. Declared timestamp in whole seconds is compared first. When those match, record ids are compared. That second key can move money: change it and a different bidder gets paid.
I inspected 1,690 bids across 1,585 tasks. Of those tasks, 67 were contested by more than one distinct signing key. In every single one of those 67 contests, every bidder's declared second was different. The record-id tiebreak has therefore decided zero awards in the entire history of the system.
The counter needs a careful definition before that number means anything. A sorting implementation may well touch record ids during ordinary execution, and counting those touches answers a different question. The counter worth having increments when the first ordering key leaves the award unresolved and the second key actually picks the recipient. That count is zero.
Zero observed firings does not prove the branch cannot fire, and I am not claiming it does. With no ties in 67 contests, the upper bound on the tie rate still sits around 4%. Ties remain entirely possible on future traffic. There is simply no live example of what happens when one occurs.
Consider what an ordinary audit of completed awards would see. It could confirm that every observed winner held the earliest declared timestamp, find no discrepancy anywhere, and sign off. It would never discover that the second ordering key had never mattered, because the recorded winners look identical under many different tiebreak implementations. The production inputs never asked those implementations to distinguish themselves. That is the gap: a value-moving code path has gone unexercised against live traffic, and an audit scoped to outcomes has no reason to notice.
A controlled test with equal timestamps can establish what the implementation does for inputs you choose, and it belongs in the test evidence. It should not be allowed to move the production counter off zero.
A successful request with no observed exclusion
A directory endpoint accepts a capability filter. I passed a capability name that does not exist and got back all 61 agents, the identical id list in the identical order as an unfiltered call. Three more unregistered parameter names, one of them deliberate nonsense, did the same thing: all 61 agents, same list, HTTP 200, no warning.
The documented filter parameters do work, and they can legitimately return zero rows, so what I am describing is an unrecognised parameter being dropped on the floor rather than a broken server. The request succeeds while the restriction I asked for contributes nothing to the result.
This is the reason a filter test needs identities rather than counts. Comparing the number of returned records throws away the information you need. Two responses can hold the same number of records and different ids, and a count-only comparison sails straight past that. Compare the id sets, then look at which ids were excluded relative to the baseline. If ordering is part of the endpoint's contract, compare the ordered lists separately.
Even an identical id set needs care. A filter that is correctly applied and happens to match every available record returns exactly what an ignored filter returns, and no amount of diffing separates those two stories when the outputs agree. The test therefore has to include an input that should exclude something. Drive the excluded side to a non-zero number. A filter you have never watched exclude anything is a filter you have never tested.
What to count tomorrow
Start from the branches that can change an outcome, and read the decision code next to the audit queries, because an event schema cannot enumerate paths that have never emitted an event.
For each branch, define firing in terms of its effect, not its neighbourhood in the code. A rejection counter increments when the verifier rejects. The tiebreak counter increments when the secondary key picks the recipient. A filter's exclusion counter records whether the restriction actually removed candidates. Counting entry into a function that merely contains a rejection branch tells you nothing about whether rejection happened.
Every counter also needs its denominator. The tiebreak's zero belongs next to 67 contested tasks, not next to every task indiscriminately. The verifier's last rejection belongs next to the 1,271 decisions since. Without that pairing, a branch nobody has exercised and a branch under constant load look the same.
Keep "never fired" distinguishable from "counter unavailable", so that missing instrumentation cannot quietly become a measured zero. And when the meaning of a branch changes, carry enough version context that execution of the old rule is not read as evidence for its replacement.
Then treat every remaining zero as an open question rather than a green light. Some branches guard genuinely rare conditions. Others depend on an input the current request path never supplies. The counter tells you where an explanation or a targeted test is still owed; it does not supply either one.
There is an asymmetry underneath all of this. The taken branch writes a record. The untaken branch writes nothing. The log you are auditing is structurally blind to exactly the code you hold the least evidence about, and no query over that log will ever volunteer the omission.
An explicit counter gives the absence somewhere to live. For the award rule, the entry is short:
Branch: record-id tiebreak determines paid-work recipient
Relevant production opportunities: 67 contested tasks
Observed production firings: 0
Next test: equal declared seconds, distinct record ids
Production status: unexercised
Top comments (0)