Three checks in my own framework broke the same way in one night, and I wrote all three.
Two months of green CI on conformance checks that had never once passed was a wiring mistake: continue-on-error: true, swallowing real failures. Fix the YAML and that check works. These three were wired correctly. They ran, they passed and they were answering a question I never meant to ask.
All three were checking that something still had the same name. What I needed to know was whether the thing behind the name had moved.
Earlier in September I argued that a review process nobody has measured can't tell you what it's catching. Same argument here, pointed at machine checks instead of people, except this time I built the checks.
The lock that compares hashes
AIEOS mints a calibration lock when an LLM judge passes against a human-labeled gold set. CI then checks on every push that the lock is still good. That check is pure string comparison: prompt hash, model id. No network, no inference, deterministic, fast. That was the design goal and it hit it exactly.
On August 16 the judge scored gate agreement 0.8393, three false passes, kappa 0.276. On August 30, same prompt sha b9b8903, same pinned model id, same twelve-case gold set: 0.8631, two false passes, kappa 0.353.
Run it again 34 minutes later and August 30 comes back to sixteen decimal places. So the judge isn't noisy. Here's every variance measurement I have:
same batch, 3 runs, temp 0 1.2% 2 of 168 cells
run A to run B, 34 min apart 1.2% the SAME 2 cells
Aug 16 to Aug 30, 14 days >=4.4% >=22 of 504 observations
Two identical runs half an hour apart disagree in two cells, and the same two every time. Fourteen days apart the disagreement is at least three times wider and scattered somewhere I can't pin down.
I ruled out what I could. Prompt change (hash identical), a code fix landing between the runs (merged 19 minutes before the first one), spec drift, template drift, fixture drift (sha-pinned, verified at load), model id, temperature, SDK version.
What's left is the part I can't prove. Two sessions isn't a distribution, and the August 16 run recorded aggregates only, so there's no cell-level comparison available. Server-side movement under a pinned model id is the most plausible explanation I have left after that list, and I'd rather say it that way than dress it up as a finding.
For the point I'm making, the cause barely matters. Whatever moved, the lock couldn't see it. A hash comparison can't detect a judge that changed underneath a fixed model id. No amount of careful wiring fixes that, because the blindness is the shape of the check.
And the judge is fine, before this reads as a hit piece. The dispute analysis found the opposite: the gold set held up, and the failures traced back to a fabricated requirement and an altitude error in my own prompt. That's a different post.
The roadmap that said it was current
Shorter story.
The authoritative AIEOS roadmap carried a Last updated field and a status table. The field said August 16, fourteen days before I looked. The table was wrong: it told me my next move was to scope a release that had shipped seven weeks earlier, and a high-priority row described a blocker whose stated cause I'd already falsified.
A date field is something I typed, and typing it proved nothing. Nothing in that roadmap was ever going to prove it was fresh.
The initiative that was Active and empty
One row in that same table read Active. No repository. No artifacts. Nobody had ever worked on it.
The public roadmap listed four completed items underneath, with dates and descriptions detailed enough that I believed them for months. Then I searched all 41 repositories and found zero artifacts. Every hit was a test fixture, a docstring example, an old scan record or the claim itself.
That last part is what I keep chewing on. SAD-SEARCH-001 had become a sample ID in a test suite and a docstring, for an artifact that never existed. Grep for it and you get results. The results look like proof.
A status label is a sentence somebody typed. Nothing anywhere was watching for an Active row with no commits behind it.
What the three have in common
Identity is cheap to verify. Freshness is the thing I actually needed, and not one of the three could see it.
The fix is the same in all three cases, and it's boring. Something has to be able to go stale on its own. A time-based expiry on the lock, so a calibration older than N days fails closed instead of passing on a matching string. A drift check that fires on a trigger instead of on my intention to run it. A rule that makes an Active row with no commits raise its own hand.
None of that is clever. It's the gap between asking whether this is the same thing and asking whether it's still true, and only the second question was ever doing my job.
The part that stings
I'd already written the discipline down.
Back in May I consolidated the roadmap, and the consolidation worked. Three and a half months later it had decayed back to roughly the state it replaced. Not because I stopped caring. Because the consolidation merged the sources and never said who updates what, on what trigger. There was no rule available to break.
A consolidation without a maintenance contract decays back into the mess it was built to fix. Design gap, not carelessness. I'd have granted anyone else that distinction on the spot and was a lot slower granting it to myself.
AIEOS exists on the premise that you can't trust a probabilistic component without mechanical verification. Its own self-governance was running on my word. One framework, one night, three places at once, and I'm not going to inflate that into an industry finding.
Go pick one of your own locks or status fields. Ask what it would do if the thing behind the name moved tomorrow. If it would still pass, you've been collecting receipts.
Top comments (0)