I put a preprint on arXiv this week. It is about forging evidence bundles that my own verifier calls clean. I mutated the sites where the verifier is supposed to refuse, counted how many mutations survived, and reported the number. The cheapest forgery that survived was four bytes.
Two people commented. Within twenty four hours they had pressed on the same weak point from opposite directions, and one of them was not talking about my paper at all.
The comment
Nguyen Thanh Vinh:
A refusal-site score is bounded by the refusals that already exist. A comparison that never runs because the field was deleted is the visible version of that, since there is still a site to mutate. The version that stays invisible is an obligation in the spec that no site was ever written for, because it never enters the denominator at all.
He is right, and it is worse than he could have known from the outside. Four forgeries survived. Three landed in classes I had sites for. The fourth landed in an obligation with nothing to mutate, so it could never have appeared in the score regardless of how the sweep went. The number counted the places I had already thought to look.
He asked for the one figure that bounds it: how many spec obligations have no refusal site mapped to them.
I do not have it. SPEC.md carries 39 normative obligations, 37 MUST and 2 MUST NOT, and there is no artifact mapping any of them to sites. "Obligations with no refusal site" is unmeasured, not small. That is the honest state and it is the next thing I am building.
The same argument, aimed somewhere else
The same day, Sarvex Jatasra published a correction to his own earlier piece on pricing agent work, crediting Vinh with finding the hole:
every price and every score quoted for agent work is measured on the subset that was kept, which makes each of them a lower bound on the real cost rather than an estimate of it
Vinh's comment on that article and his comment on my paper are one argument in two costumes. On Sarvex's post the missing denominator is the runs a person abandoned and rewrote. On mine it is the obligations nobody wrote a check for. Both quote a number measured on survivors. Neither publishes the denominator.
Sarvex left a comment on the paper too, and it is the same shape a third time: my counts of 37 through 143 sites are totals over a population re-enumerated on every run, so a site that flipped to surviving cannot be told apart from one deleted while another arrived. Also correct. I had already pulled that per-site claim from the paper for exactly that reason and the v2 with the corrected wording is queued.
The part I got to feel smug about for one minute
I run a public drift board that re-tests 17 models against a frozen 35 task suite daily. In August, Google's endpoint started rate limiting, and a model scored 0.0286. One task out of thirty five.
That is not a bad model. That is an absent measurement wearing a score. Average it in and you publish a false regression. Drop it silently and you publish a stale standing with no trace anything was missing. Both are wrong and only one looks wrong.
The board already handled this. There is a reliability floor at 0.5, and low-reliability points are excluded from the accuracy line while being kept in the Reliability metric, because reliability genuinely did drop and that is the true signal. I shipped that on 2026-07-21, the same day I published a post here called "Distinguishing wrong from absent."
So when I read Sarvex's prescription:
Tag every invocation with whether it ended in an artifact somebody kept, then publish runs per accepted change beside your cost per completed task, on the same slide, for the same window.
my first reaction was that I had been doing that for six weeks.
The minute ended
My board was right. My published evidence could not demonstrate it.
The board ships a VAC bundle: hash-pinned artifacts that an offline verifier re-derives from the committed rows, so a reader can check the claim without trusting me. That bundle did not publish the floor. It did not publish which run had been excluded. It published the standing and nothing about the population the standing was computed over.
Which means the verifier, recomputing honestly from the same data, got different answers for three Google models and refused the bundle. It was correct to refuse. Two of the verdicts it recomputed were inverted against what the board displayed. The registry entry sat PENDING for eight days with a refusal message that read like the board was broken.
The board was not broken. The evidence was incomplete in exactly the way Vinh described: the excluded runs never entered the published denominator, so nothing downstream could reconstruct the decision.
The fix, yesterday, was to publish the floor as a pinned value and require both numbers: the current qualifying standing, and the latest observed run with its reliability, whether or not it qualifies. Three of seventeen models currently carry a disqualified latest run. An outage now reads as an outage in the evidence, not only on the chart.
And then, writing this post
I went to check my own claim before publishing it, because a post asserting "the board publishes X" should be verified against the board rather than against my memory of it.
REL_FLOOR = 0.5 is defined twice. Once in modeldrift/policy.py, once as a JavaScript literal in the dashboard. They agree today. Nothing makes them agree tomorrow, and they are not redundant: the Python one decides the qualifying standing, the JS one decides what the chart draws. If they drift, the table and the chart disagree about the same model on the same day, each internally consistent, with no error raised anywhere.
Yesterday's work taught the verifier to cross-check the floor against two surfaces. The chart is the fourth surface and it was left out. It is filed now.
What it adds up to
Knowing a principle and having it enforced everywhere are different states, and the distance between them is invisible until something in the world goes down or a stranger asks the right question. I had the floor in the chart, then in Python, then in the evidence, and I still found a fourth copy outside the check while writing a post about it.
The mutation score has the same shape and I have not closed it. Until an artifact maps all 39 obligations to sites, that score answers "how many of the checks I wrote can be defeated" while readers will hear "how many ways this can be defeated." Those are different questions. Only one of them currently has a denominator.
Paper: https://arxiv.org/abs/2608.26183
Board: https://egnaro9.github.io/model-drift/
Verifier and registry: https://github.com/egnaro9/vac-protocol
Thanks to Nguyen Thanh Vinh and Sarvex Jatasra, who between them cost me a claim and handed me the next two pieces of work.
Top comments (0)