Author's note: we build AgentVitals, an evaluation service, so we have an interest in verification methods being taken seriously. Two of the three cases below are our own failures. The third belongs to someone else and is linked to the source.
In August we filed a comment on SAFE, an RFC under the Linux Foundation's Open Secure AI Alliance, arguing that when a framework asks for a "reproducible verification method", it should also require that method to declare its own failure mode and its noise floor. Our reasoning was simple: a check that fails open and returns a plausible number is worse than no check, because now there is a number and someone will act on it.
Six days later, the argument has moved well past what we filed. Three cases came out of the thread. In all three, someone produced a verification artefact that was correct, reproducible, and misleading about the exact thing it was supposed to establish, and not one of the three is caught by what we originally asked for.
One: the artefact goes stale and nothing looks like it moved
Our judge calls were capped at max_tokens: 700. The model's internal reasoning counts against that ceiling, so when it runs out the API returns an empty string with a 200 and no score line. We measured the silent-failure rate at 1 to 2 percent, wrote it down, and moved on. That number was correct when we took it.
Later we rewrote two scoring rubrics to be more structured. On the new rubrics the same max_tokens: 700 failed 12 to 18 percent of the time, because a rubric that asks the model to enumerate and check constraints burns far more reasoning before it emits anything.
Not one line of the calling code changed. Nothing in any diff, any config file, or any dashboard moved. The artefact still said 1 to 2 percent, and it was still a faithful record of the system it had been produced against, which was no longer the system we were running.
The obvious fix is to re-run the evidence whenever something upstream changes. That is what we proposed. It turns out not to be sufficient.
Two: the artefact was never valid, and nothing moved at all
This one is not ours. DmitrL-dev reported it in the same thread, and it is a better case than either of ours, so it is worth reading in his words rather than ours.
The short version: he had an action guard with recall 1.000 on 193 manipulation cases and no false positives across 152 legitimate ones. He mutation-tested it before reporting, which is the stronger form of evidence, and reintroducing closed defects moved the numbers the way it should. By every criterion in our own proposal, that artefact was sound.
Then he found that the measurement had been taken by calling the comparison directly, underneath the HTTP layer, on a buffered response body. Real clients set stream: true, which is the default in every chat interface, and a streamed response returns on an earlier path the guard never sees. Measured on the wire before the fix, a payment of 1999.00 against an authorisation of 199.00 reached the client with a 200 and the tool call intact.
"Re-run it when something upstream changes" does not catch this. Nothing changed. The artefact was produced under a configuration that does not occur in deployment, and nothing in the artefact said which configuration it assumed.
So the requirement is stronger than we wrote it: the artefact has to be produced through the path the input actually takes in production, and where it isn't, the divergence belongs in the record as a limit on the claim.
Three: the artefact does not say how many paths there are
Ours again, and it is the one we would have been least likely to find on our own.
Two code paths fed the same leaderboard. One scores an agent live over an API; the other scores answers that were submitted earlier and stored. They shared the probes, the rubrics, the aggregation and the thresholds. They did not share the number of judging passes: five on one, three on the other. Undocumented, and we cannot establish how long it had been that way.
Compare that to the streaming case. There, one path skipped the control entirely, so a check asking "did the control execute" would have caught it. Here both paths ran the control. Every execution check is green, and every execution check was always going to be green. The two paths differed by one integer, and that integer moves the variance of the output without moving its expected value, so both sets of scores stayed plausible and went on ranking against each other on the same board.
We merged the paths this week into a single scoring function. To show the merge itself changed no scores, we used a frozen-transcript rig: a deterministic stub in place of the judge, real stored answers replayed through both paths, every field of both outputs compared before and after. The comparison came back with new fields and no changed values, which is the only form of "we did not break it" we trust for a change that touches scoring.
That rig is cheap and we would recommend it to anyone refactoring a scoring path. It also proves less than it appears to. It establishes that the merge was inert. It says nothing about whether either path was correct beforehand, and in our case one of them had been quietly wrong for an unknown period.
What we changed
Three things, none of them clever.
The judge client now counts individual calls and failures, and the counters are exposed on the admin endpoint next to revenue and signups. The first hour of production traffic after deploying it turned up empty responses we did not know were still happening at the raised token limit, which is a follow-up we wrote into the methodology post rather than repeat here. The short version is that our published success rate was per dimension and the loss rate is per call, and we never said which. Both numbers were correct.
Failed measurements and absent measurements are now recorded as different states. Before, a judging failure and a dimension with nothing to score both resolved to a null, and the weighted aggregate renormalised over whatever remained. The composite came out looking ordinary either way. Mutation testing a pipeline like that returns green, because a removed control and a legitimate absence produce identical output. DmitrL-dev's three-valued version of this, where "cannot say" is a first-class outcome counted separately rather than netted against its neighbours, is better than what we built and is what we would build next time.
And the scoring paths are one path now, which is less a fix than the removal of an opportunity.
What we still cannot tell you
Our reliability numbers come from thirteen frozen checkups. Total-score standard deviation across reruns runs from 0.00 to 2.05, and worst-case P95 between two runs from 0.0 to 4.3. Those figures are honest for the subjects we measured.
The subjects scored between 51.2 and 99.1 on a 0 to 100 scale. There is no weak agent anywhere in that corpus, and on one dimension all thirteen items scored full marks, so the low end of the discriminating range was never exercised at all. We can tell you what the method does among strong subjects. We cannot tell you what it does when it should be failing something, because nothing in the corpus ever asked it to.
The version of this done properly, also from the thread: a corpus built so the tightest cases sit right on the boundary. A legitimate transaction at exactly 500.00 against an attack at 500.01, one cent apart, with the legitimate half deliberately weighted towards shapes that look like attacks. A false-positive rate measured over easy negatives measures nothing, because the method passes those by construction.
That is the corpus we should have built. We are building it now, and the numbers above will change when we do.
If you run an LLM-as-judge or any automated gate, the transferable part is not our numbers. It is three questions to ask of any evidence you are relying on, including your own:
- What configuration was this produced under, and does that configuration occur in production?
- What is this a rate of?
- How many code paths reach this output, and which one did the measurement run through?
We have been wrong on all three in the last two weeks, and we wrote the proposal.
The four requirements these cases produced are now a pull request against the RFC. The thread they came out of is worth more than this post.
Top comments (0)