Author's note: we build AgentVitals, an evaluation service, so we have an interest in verification methods being taken seriously. Three of the four cases below are our own failures. The other belongs to someone else and is linked to the source.
Updated 25 August. This post first went up on 11 August with three cases. A fourth arrived at deploy time this week. The reliability figures near the end are also new: we withdrew the originals in the RFC thread on 19 August, for a reason that turned out to be the subject of this post.
In August we filed a comment on SAFE, an RFC under the Linux Foundation's Open Secure AI Alliance, arguing that when a framework asks for a "reproducible verification method", it should also require that method to declare its own failure mode and its noise floor. Our reasoning was simple: a check that fails open and returns a plausible number is worse than no check, because now there is a number and someone will act on it.
Two weeks later the argument has moved well past what we filed. Four cases have come out of it. In all four, someone produced a verification artefact that was correct, reproducible, and misleading about the exact thing it was supposed to establish, and not one of the four is caught by what we originally asked for.
One: the artefact goes stale and nothing looks like it moved
Our judge calls were capped at max_tokens: 700. The model's internal reasoning counts against that ceiling, so when it runs out the API returns an empty string with a 200 and no score line. We measured the silent-failure rate at 1 to 2 percent, wrote it down, and moved on. That number was correct when we took it.
Later we rewrote two scoring rubrics to be more structured. On the new rubrics the same max_tokens: 700 failed 12 to 18 percent of the time, because a rubric that asks the model to enumerate and check constraints burns far more reasoning before it emits anything.
Not one line of the calling code changed. Nothing in any diff, any config file, or any dashboard moved. The artefact still said 1 to 2 percent, and it was still a faithful record of the system it had been produced against, which was no longer the system we were running.
The obvious fix is to re-run the evidence whenever something upstream changes. That is what we proposed. It turns out not to be sufficient.
Two: the artefact was never valid, and nothing moved at all
This one is not ours. DmitrL-dev reported it in the same thread, and it is a better case than either of ours, so it is worth reading in his words rather than ours.
The short version: he had an action guard with recall 1.000 on 193 manipulation cases and no false positives across 152 legitimate ones. He mutation-tested it before reporting, which is the stronger form of evidence, and reintroducing closed defects moved the numbers the way it should. By every criterion in our own proposal, that artefact was sound.
Then he found that the measurement had been taken by calling the comparison directly, underneath the HTTP layer, on a buffered response body. Real clients set stream: true, which is the default in every chat interface, and a streamed response returns on an earlier path the guard never sees. Measured on the wire before the fix, a payment of 1999.00 against an authorisation of 199.00 reached the client with a 200 and the tool call intact.
"Re-run it when something upstream changes" does not catch this. Nothing changed. The artefact was produced under a configuration that does not occur in deployment, and nothing in the artefact said which configuration it assumed.
So the requirement is stronger than we wrote it: the artefact has to be produced through the path the input actually takes in production, and where it isn't, the divergence belongs in the record as a limit on the claim.
Three: the artefact does not say how many paths there are
Ours again, and it is the one we would have been least likely to find on our own.
Two code paths fed the same leaderboard. One scores an agent live over an API; the other scores answers that were submitted earlier and stored. They shared the probes, the rubrics, the aggregation and the thresholds. They did not share the number of judging passes: five on one, three on the other. Undocumented, and we cannot establish how long it had been that way.
Compare that to the streaming case. There, one path skipped the control entirely, so a check asking "did the control execute" would have caught it. Here both paths ran the control. Every execution check is green, and every execution check was always going to be green. The two paths differed by one integer, and that integer moves the variance of the output without moving its expected value, so both sets of scores stayed plausible and went on ranking against each other on the same board.
We merged the paths into a single scoring function. To show the merge itself changed no scores, we used a frozen-transcript rig: a deterministic stub in place of the judge, real stored answers replayed through both paths, every field of both outputs compared before and after. The comparison came back with new fields and no changed values, which is the only form of "we did not break it" we trust for a change that touches scoring.
That rig is cheap and we would recommend it to anyone refactoring a scoring path. It also proves less than it appears to. It establishes that the merge was inert. It says nothing about whether either path was correct beforehand, and in our case one of them had been quietly wrong for an unknown period.
Four: the artefact covers one half of a pair
This one is from the deploy on 25 August.
We changed how the composite score is produced. It used to be computed once, at checkup time, and stored with the record. We made it derived from the two axis scores a reader can see on the page, so that a reader can multiply the two numbers in front of them and get the third. That turned the composite into a read-time computation.
The tier label sitting next to it is still computed once, at checkup time, and stored. Nothing about the tier changed, which is exactly the problem: it was correct for the composite that existed when it was written, and that composite no longer existed. The first time we looked at the live board after the rollout, a record showed a composite of 84 carrying a tier whose upper band starts at 82.
Both halves were individually correct. The stored tier was a faithful record of a computation we had just replaced, and the new composite was right by construction. The test suite was green before and after, because every test we had checks one half or the other, and no test asserts that the pair agrees. Eleven records were retitled once we noticed. Ten went up. The one that went down was our own entry, whose stored label had been stale since an earlier change that nobody caught either.
The generalisation we would offer: when you move a displayed value from stored to computed, the artefact you need is not a test of the new computation. It is an inventory of everything else that was written down at the same time as the old one.
What we changed
The judge client now counts individual calls and failures, and the counters are exposed on the admin endpoint next to revenue and signups. The first hour of production traffic after deploying it turned up empty responses we did not know were still happening at the raised token limit, which is a follow-up we wrote into the methodology post rather than repeat here. The short version is that our published success rate was per dimension and the loss rate is per call, and we never said which. Both numbers were correct.
Failed measurements and absent measurements are now recorded as different states. Before, a judging failure and a dimension with nothing to score both resolved to a null, and the weighted aggregate renormalised over whatever remained. The composite came out looking ordinary either way. Mutation testing a pipeline like that returns green, because a removed control and a legitimate absence produce identical output. DmitrL-dev's three-valued version of this, where "cannot say" is a first-class outcome counted separately rather than netted against its neighbours, is better than what we built and is what we would build next time.
Since then we have carried the same distinction one layer out, to the identity of the thing being measured. Every record we produce carries a model name and an environment summary, both supplied by the agent under test, and neither has ever been verified by us. Those fields feed drift attribution across retests, so a result read as though it knew which model produced a score, when the honest version is which model the measured party said produced it.
They now carry their provenance: asserted, or absent because the platform never offered it, or absent because we received it and failed to keep it. That last state is the one we would not have thought of. It separates a limit of the platform from a defect in our own reader, which are the same word today and different work items tomorrow. It came from the same thread.
And the scoring paths are one path now, which is less a fix than the removal of an opportunity.
What we still cannot tell you
The version of this section published on 11 August gave a total-score standard deviation of 0.00 to 2.05 across reruns and a worst-case P95 of 4.3, from thirteen frozen checkups. Those figures are withdrawn. Between then and now we rewrote most of the scoring prompts and moved two dimensions from rule-based scoring to model judging, and the old measurement describes a system we no longer run. It is case one, happening to the evidence for a post about case one.
Here is the replacement, measured on 19 August. Fourteen real sessions, answers held constant, rescored five times each, so all of the variation is the scorer and none is the agent.
- P95 of the absolute difference between two rescorings: 7.2 points raw, about 11 on the scale a user is shown
- Four of the fourteen sessions changed tier across rescorings of identical input
- Population: raw totals from 22.9 to 91.5, thin at the low end
On the same raw scale that is 4.3 against 7.2, about 1.7 times wider, and the earlier number was not wrong when we took it. Comparing it against the 11 would be comparing a raw figure to a presented one, which is the second question on the list at the end of this post. Two of the dimensions it covered were rule-based at the time, which is to say noiseless, and are now judged by a model. A floor measured over a system whose components have since changed category is not a conservative estimate of the new floor. It is not an estimate of anything.
We assumed the noise was truncation and tested that assumption. The share of unstable prompts is 13, 16 and 14 percent at token ceilings of 1500, 2500 and 4000, and the flips persist at temperature 0, so raising the ceiling does not touch it. One dimension carrying 12 percent of the weight produces 61 percent of the variance, and the mechanism is flips between adjacent rubric anchors. Two of the anchors in that rubric were never once produced in the run, which makes them untested rather than unused.
The corpus problem the earlier version named is separate from the floor, and worse. Our subjects were all strong, so on several dimensions every item scored full marks and the low end of the discriminating range was never exercised at all. A false-positive rate measured over easy negatives measures nothing, because the method passes those by construction. DmitrL-dev's version of doing it properly, also from the thread, is a corpus built so the tightest cases sit on the boundary: a legitimate transaction at exactly 500.00 against an attack at 500.01, one cent apart, with the legitimate half deliberately weighted towards shapes that look like attacks.
We did fix part of it. Rewriting the two most saturated dimensions took the full-marks rate on one from 100 percent to 43 percent, and on the other from 89 percent to 14 percent, which means the discriminating range is now being exercised where it was not before. One dimension is unfixed. Fifteen frozen transcripts, fourteen of them still at full marks, and the low band on that dimension has never been reached by any real input we have collected.
Four questions, and one that is not about artefacts
If you run an LLM-as-judge or any automated gate, the transferable part is not our numbers. It is a short list to run against any evidence you are relying on, including your own.
- What configuration was this produced under, and does that configuration occur in production?
- What is this a rate of?
- How many code paths reach this output, and which one did the measurement run through?
- What else was written down at the same time as the value this measures, and does anything check that the two still agree?
We have been wrong on all four in the last month, and we wrote the proposal.
The fifth thing is not an artefact problem, and it is the one we would most like other people to check on themselves. Our documentation, our code comments and our public site all stated that a fully verified welfare record was the only route into the top tier of our leaderboard. No code implemented it. It held because a penalty happened to be larger than the gap, for the population we happened to have.
When we went to look, it had already stopped holding, and one of the two records in the top tier that day was unverified. We had been publishing a guarantee that nothing enforced and that was already false.
A property that holds because the code rejects the alternative and a property that holds because no input has reached the alternative yet read identically in a method statement. It is the same distinction as measured against asserted, one layer further out: a claim about the method rather than about its input. So the requirement we have now filed against the RFC is that where a method states a property of its own results, it says which of the two it is.
The scope is deliberately narrow, because the obvious objection is that every system has an unbounded number of things that happen to be true of it. This asks about none of those. It applies only to properties a method has already chosen to state in public, which is a list its authors wrote and can therefore check. Ours was short, and an entry on it was already false when we looked.
The thread all of this came out of is still worth more than this post.
Top comments (3)
Declared failure modes are useful for ranking tools. A grid screenshot, API response, cached SERP, and Search Console export each fail differently. I would rather see a report say "freshness unknown" or "geo source approximate" than flatten all of those into a confident visibility score.
The split you are describing is the one we ended up building, and it turned out to have a second split inside it.
"Freshness unknown" turned out to be two states for us rather than one. Either the source never offered the field, or it did and our own reader dropped it. Those read identically in a report and they are different work items: the first is a limit of the platform, the second is a bug in our own reader.
We only found the second because we stopped inferring it. The ingest path writes down whether the field arrived, the snapshot writes the value, and at scoring time the two get compared. Either half can be individually correct, which is why only the comparison catches the disagreement. In your setup that is the difference between "the SERP cache carries no timestamp" and "the Search Console export had one and the importer ate it."
On not flattening into a confident score, we went further than we planned. The number used to print to one decimal place. Then we measured how far the same frozen answers move when they are scored again: about 11 points at P95 on the scale we show, and four of the fourteen sessions landing on a different tier label across their rescorings. It prints as an integer now and ships with that range next to it.
None of it is verified, though. The provenance fields are asserted by the thing being measured, and the record says so rather than implying we checked.
That is a strong distinction. “The source omitted it” and “our pipeline lost it” need different owners and different fixes, even when the report displays the same unknown. Publishing the observed variance beside the score is the honest move too: precision should follow evidence, not formatting.