DEV Community: Erik Hill

I wrote down which test would catch each bug. The one aimed straight at it was blind.

Erik Hill — Sat, 10 Oct 2026 02:53:27 +0000

I have three checks over the same game engine and they all look like determinism checks.
That raises a question the checks cannot answer about themselves: do they cover different
failure modes, or the same one three times?

So I wrote down, before running anything, which check would catch which bug. Then I
injected the bugs.

Two rows came out as predicted. The third did not, and the third is the only one worth
writing about.

Three checks that look alike

S, theSameSeedProducesTheSameRun: runs the engine twice in one runtime, compares.
G, ParityTraceTest: asserts a 600-frame, 11-checkpoint trace against a committed
golden file.
D, :web:parityCheck: diffs the JVM trace against the shipped TeaVM artifact's trace.

And a fourth column as a footnote, U, InterstitialCadenceTest: a plain unit test on
how often an interstitial cue fires.

Context: this is the rules engine of a small Android game, compiled a second time to the
browser with TeaVM. Same Java, two compilers. The determinism story is its own piece; this one is about the checks that came out of it.

Real patches, real rebuilds, real exit codes

matrix/run_matrix.sh, 134 lines: apply patch, rebuild both artifacts, run S, G, D and
U independently, revert under a trap.

They have to run separately, because :web:parityCheck declares dependsOn :engine:test.
A golden failure would stop the differential from ever running, and then I could not
attribute the columns: a blank cell would mean "did not run" rather than "did not fire".
ParityTraceTest writes its traces before it asserts, which is what makes the split
possible.

Every cell below is an exit status, not a judgement.

Three faults:

F1: Rng delegates to java.util.Random instead of the written-out LCG. On the JVM that is behaviour-identical, because it is the documented algorithm. TeaVM's implementation is not the JDK's. This is the original bug, reproduced from source.
F2: a shared constant, PLAYER_BOTTOM_MARGIN, 120f to 121f. Both builds compile it.
F3: INTERSTITIAL_EVERY, 3 to 4. Behaviour the trace did not record.

The first run

fault	S	G	D	U
baseline	pass	pass	pass	pass
F1 runtime-supplied RNG	pass	pass	FAIL	pass
F2 shared constant	pass	FAIL	pass	pass
F3 untraced cadence	pass	pass	pass	pass, predicted FAIL

Read the two that worked first, because they are the boring half and they are the reason
the exercise was worth doing.

F1 is caught only by the differential. S compares the JVM to itself, and the golden
file's JVM side never moves, so neither can see a fault that only changes the browser.

F2 is caught only by the golden. Both builds move together, so the differential sees
two artifacts agreeing perfectly on the wrong number.

Neither check subsumes the other. Good.

The row I got wrong

F3 moving nothing in S, G or D was half expected. The trace recorded frame, score, streak,
run state, player position and obstacles, but not runsSinceInterstitial. Blind by schema.
Fine.

The half I did not expect was column U. InterstitialCadenceTest exists for exactly this
behaviour. It passed.

Its assertions referenced GameEngine.INTERSTITIAL_EVERY on eight lines. When the fault
moved that constant from 3 to 4, the test's expectation moved with it. It looped
for (run = 1; run < INTERSTITIAL_EVERY; run++) and then asserted "no cue after
INTERSTITIAL_EVERY runs". Both halves defined by the thing under test.

It checked the predicate against the constant, not against an answer key. A test written
that way cannot notice the constant changing, which is the only interesting way that
constant can be wrong.

So F3 had no catching check at all. Three trace-based checks blind by schema, and the one
test aimed directly at the behaviour blind because it had no independent expectation.

Fixing it, and the part I nearly got wrong twice

The obvious fix is both halves: give the test a literal answer key, and put the field in the
trace.

// before: expectation derived from the thing under test
for (int run = 1; run < GameEngine.INTERSTITIAL_EVERY; run++) { ... }

// after: expectation is an answer key
private static final boolean[] CUE_AFTER_RUN = {false, false, true};

The test now mentions INTERSTITIAL_EVERY exactly once, in a comment explaining why it must
not use it. The trace gained runsSinceInterstitial on both sides of the differential.

Here is the part worth the whole article. Before re-running, I wrote down what I expected:

F3 flips to U=FAIL only, and G and D stay green. The scripted run dies once, so the
counter reads 0 then 1 and never approaches any threshold. Recording it cannot make the
trace-based checks see a threshold change the run never exercises.

That prediction is the interesting one, because the naive expectation is the opposite. I had
just added the missing field to the trace. Surely now the trace-based checks can see it?

The re-run

fault	S	G	D	U
baseline	pass	pass	pass	pass
F1 runtime-supplied RNG	pass	pass	FAIL	pass
F2 shared constant	pass	FAIL	pass	pass
F3 untraced cadence	pass	pass	pass	FAIL

The prediction held. U fires. G and D stayed green even though the field they were
missing is now in the trace.

Because the scripted run only ever dies once. The counter reads 0, then 1. Changing the
threshold from 3 to 4 cannot move a number that never gets near either value.

Three ways a check is built blind

The first run showed two. The second showed the third, and the third only became visible
after the first two were fixed.

The schema. A trace cannot report a field it does not carry.
The expectation. If the expected value in an assertion is imported from the code under test, the assertion measures internal consistency, not correctness. Grep your suite for assertions that reference the constant they are testing.
The input plan. A check is only sensitive to behaviour its inputs actually exercise. Coverage of a field is not coverage of a behaviour. This is the one I would have missed entirely if I had stopped at the first fix and declared the residual closed.

The caveats this needs, in the body rather than a footer

Three faults is not a coverage measurement. It is three worked examples chosen to separate
four checks. A real matrix would keep going until it stopped finding new residual rows, and
mine stopped when I ran out of faults I could think of.

The predictions were written in a commit message and a chat log, not a pre-registration
anyone can audit. Take the first miss as the evidence they were genuinely made in advance,
since it is the row I would have quietly dropped otherwise.

And the defect was in a test, not in shipped behaviour. The game's cadence is correct. What
shipped broken was the confidence.

A matrix without a residual row is advertising. The row where nothing fires is the
deliverable.

Harness, patches and full output: github.com/egnaro9/tapdodge-engine, matrix/
Write-up: docs/field-note-fault-matrix.md

I shipped a linter for checks that cannot fail. It had four of them.

Erik Hill — Thu, 08 Oct 2026 12:30:00 +0000

Where this landed, 9 October 2026. This post opens with 40%. That number is not
wrong so much as it is not the quantity you would take it for, and I only found that out
by being argued with in the comments. It has taken a week of measuring to say what it
should have been, and the tool has changed more than the number has. This block replaces
the two corrections that were stacked here, because three corrections in a row is not
honesty, it is sediment.

The sampling was measuring the wrong unit. The samples behind every figure were
stratified with a cap of two findings per repository, to stop one repo dominating. But
the tool emits one finding per (job, upstream) pair, and the false positives cluster in
jobs with many dependencies: one reporting job produced eleven by itself. The cap deletes
exactly that clustering. Simulated against a population I later labelled completely, the
capped sampler is an unbiased estimator of the per-job rate and runs about 16 points
low for the per-finding rate. So every percentage in this post understates, all in the
same direction. @arhancanli called this in the thread before
I measured it.

What the numbers are now, all hand-labelled, no estimate of an estimate. Four
detectors are two. D4 went on 0 defensible of 18 sampled, D2 on 0 true of 27 censused.
Corpus output went 1986 findings to 1026.

basis false

D3, all 653 complete census, every finding read 36.1% [32.5, 39.9]

D1 gated on --repo, and 0 findings with it see below

The severity tiers are gone, and that is the result I did not expect. Censusing all
three strata gave HIGH 30.9%, LOW 31.6%, MEDIUM 52.8%. HIGH's interval overlaps the
merged rest. The only real separation is MEDIUM being worse, and it came from the
severity function short-circuiting on whether the workflow runs on pull_request before
it looked at the finding at all. The axis measured the trigger, not correctness. It
hid the better findings behind --all and shouted the worse ones, so collapsing it
raised the default output from 238 findings to 653 and lowered the false rate a reader
sees, from 44.1% to 36.1%.

D1 is gated rather than removed, and the gate reports nothing. Its text says a
skipped upstream leaves the dependent running without it; GitHub documents that the
dependent is skipped too.
68% of its findings are that case. So it now says nothing
unless --repo confirms the job is a required check. Then I ran that against live branch
protection for all 373 findings: 10 resolved to REQUIRED, all in one repository, and all
10 vanish on a complete checkout where the tool can read the script that job calls. On
275 repositories, --repo has never reported a finding. It is kept because the gate is
correct, not because it is productive, and that distinction is the whole post in one line.

Three numbers I published and had to take back, every one caught by checking my own
claim rather than by anyone complaining. The "65% false" I quoted for D1 came from a
census covering 12% of the population it was quoted against. The "37.0%, 245 of 662" in
my own 0.3.0 release notes added a stratum censused at 153 to a population taken
after a suppression had already removed 9 of them; every commit since emits 653. And the
54.9% in the table that used to sit right here kept 3 findings labelled true from a
sample taken before the insight that refutes them, which is holding both positions at
once: I shipped a suppression on the grounds that cmd | grep -q X is not a defect
while still counting three of those lines as defects. The arithmetic carries its own
check, and I had not run it. The census was 68 true and 85 false of 153, so removing 9
false findings has to leave true at 68, and 68 + 76 = 144. A correction is a new claim
and needs verifying as hard as the original.

The part of this post I would still defend is the part about taste: the corpus average
hid the problem, and pointing the tool at two repositories I respected is what found it.
The part I would not defend is any specific percentage in it. The working is in the
comment thread, including the fixes measurement killed before they shipped. The latest
was a filter that genuinely improved precision, 31% false down to 17%, and I deleted it
anyway: it bought those 14 points by discarding two thirds of the true findings, and
on that population it turned out to be a shape classifier wearing a semantic test.

	basis	false
D3, all 653	complete census, every finding read	36.1% `[32.5, 39.9]`
D1	gated on `--repo`, and 0 findings with it	see below

Yesterday I published a tool called deadgate to PyPI. It reads GitHub Actions workflows and finds checks that cannot fail: a required gate satisfied by a skipped job, an exit status swallowed by a pipe, a fan-in job that never reads whether its upstream passed.

Then I pointed it at Arize-ai/openinference, 1.2k stars, 22 workflows. It reported 13 HIGH findings.

Twelve of them were false.

Not borderline. Not "technically true but low impact." Their CI was correct and my tool could not see it. The version doing the reporting had been on PyPI for about an hour.

What it could not see

GitHub's documented way to ask "did anything upstream fail" is the wildcard form:

if: always()
run: |
  if [[ "${{ contains(needs.*.result, 'failure') }}" == "true" ]]; then exit 1; fi

My detector tested for that with needs\.[A-Za-z0-9_-]+\.(result|outputs), which cannot match *. So the one spelling the documentation recommends was the one spelling the tool was blind to, and every correctly written fan-in gate came back as a gate that cannot fail.

The finding's own message read "never reads needs.*.result" while failing to match that exact string.

I fixed it, shipped 0.1.1, and pointed the fixed version at astral-sh/ruff. Twenty-one HIGH findings. Eighteen false. Ruff writes its gate a third way:

env:
  NEEDS_JSON: ${{ toJSON(needs) }}
run: |
  failing=$(echo "$NEEDS_JSON" | jq -r 'to_entries[]
    | select(.value.result != "success" and .value.result != "skipped")')

That reads every upstream at once and the word result never appears in any workflow expression. There is nothing for my pattern to match.

The fourth one was not a spelling at all. scikit-learn has a job that calls a reusable workflow, so it has uses: and no steps:, and it passes needs.check-sdist.result through job-level with:. That expression was one I already understood perfectly. It just lived somewhere I was not looking, because I scanned step-level with and not job-level with.

Four classes. The first two shipped to PyPI. I found the third and fourth only because I kept pointing the "fixed" version at new repositories instead of declaring victory.

The part I think is actually generalizable

Here is the number that matters, measured on a 275 repository, 4543 file corpus with both versions reading identical bytes:

version	findings	HIGH
before	4562	1104
after	2917	658

Forty percent of HIGH findings were false. That sounds bad, and it undersells it, because 40% is an average and the errors were not spread evenly.

Only 15 of those 275 repositories use the wildcard idiom at all. The false positives piled onto the repositories that write their gates carefully. On openinference the false rate was 92%. On ruff, 86%. On a repository with no fan-in gate at all, it was zero, because there was nothing to misread.

My linter was least accurate on the best code.

If a tool is noisy on bad code, the noise and the signal arrive together and you read both. If it is noisy specifically on good code, the people who did the work correctly are the ones punished, and the obvious response is to stop running it. A corpus average cannot surface that, because the corpus is mostly ordinary repositories where the tool happens to be right.

I had the corpus. I ran it. It said 40%, and I would have shipped that as the headline. What actually found this was pointing the tool at two repositories whose CI I respected.

The mutation that survived

To stop reporting jobs whose failure a gate already catches, I walked the needs graph transitively: if a gate covers ci, and ci needs changes, then changes is covered too.

I mutation tested the fix, which for me means planting the defect each test claims to catch and checking that something goes red. One mutation survived: the one that deleted my transitive walk.

My first instinct was a missing test. It was not. The mutation survived because the walk was wrong: needs.*.result reports only direct needs, and a failed grandparent makes the parent skip, and skipped is not failure, so the gate never sees it. Walking transitively would have suppressed real findings.

A surviving mutant has two meanings and they are easy to confuse. Either a test is missing, or the code does nothing. If I had written a test to kill it, I would have pinned a bug in place and called it coverage.

The small one that is somehow the worst

While fixing all this I noticed five of my repositories advertised a stale test count in the README. I corrected them by hand and added a CI check so they could not drift again.

The check went red immediately. On a count I had just "corrected."

I had changed agent-graph's README from 42 to 58 and written a public commit message saying the previous commit overstated the count at 67. CI collects 67. My local venv was missing an optional extra, and tests/test_rag.py opens with pytest.importorskip, so nine tests vanished at collection with no error, no warning, and no mention of the file. pytest said "58 tests collected" and meant it.

The original number was right. I corrected a correct claim, publicly, using a measurement from an environment that was not the real one.

The check I shipped an hour earlier had the same blind spot and agreed with 58. It now fails when any test file on disk contributes zero collected tests, because that is the only evidence an environment is incomplete.

What I am not showing

The sweeps above are driven by my own harness, and the agent scaffolding, gate implementations and prompts behind it are not in the repository and are not going in it. deadgate itself is small and MIT and you can read all of it. The thing that pointed it at 275 repositories and classified the survivors is the part I am keeping.

The question I actually have

I do not know how to test a linter for this class of error without a sample of good code, and "good code" is not a property I can detect automatically. A corpus gives you volume and volume gives you the average, which is exactly the statistic that hid this.

The only thing that worked was taste: picking two repositories I already believed were well engineered and reading the output by hand. That does not scale and it is not reproducible, and I would like to be talked out of it.

If you maintain a static analysis tool, how do you measure your false positive rate on the code that is already correct? I would rather hear that I am missing something obvious than keep doing it by hand.

deadgate is at pypi.org/project/deadgate and github.com/egnaro9/deadgate, MIT, no model in the loop.

I work on AI evaluation and testing, open to relocation above $120K, portfolio at erikhill.dev.

The gate could not see what it was gating

Erik Hill — Sat, 03 Oct 2026 14:37:55 +0000

Originally published at erikhill.dev. The numbers below are checked against the repository they come from.

The gate could not see what it was gating

This is a finding about my own tooling. The gate it describes is mine, and so is the hole.

What happened

I keep a pre-commit hook on my portfolio repo that refuses a commit touching a
claim-bearing file unless a receipts file carries a receipt for it. It never judges
whether a claim is true. It checks that a receipt exists and that its pointer
resolves: a local path has to exist, an https pointer has to answer 2xx. Prose alone
cannot satisfy it.

For four days, any file that had been reviewed once was exempt from review forever.

The defect is one line

LINE=$(grep -F -- "$f ::" "$ACK" 2>/dev/null | head -1)

head -1 on a pathname. The first line in the receipts file that mentioned the path
won, whatever it said and whenever it was written. A receipt recorded for an earlier
version of a file kept satisfying every later version of it.

So the gate reported success along a path where it never examined the staged
content. That is the precise failure I build tooling to find, sitting in the tooling
I built to find it.

The evidence

claim	how it was checked
the defect line existed	`git show 47f82a1^:.githooks/pre-commit` line 61, the `head -1` shown above
the window was four days	the hook's full history is four commits: installed `c9af511` 2026-08-23, fixed `47f82a1` 2026-08-27
selection is now by blob	the fix reads `BLOB=$(git rev-parse ":$f")` and greps `":: blob $BLOB"`, so line order decides nothing
a stale receipt is now refused by name	the hook prints `no receipt carries the staged blob $BLOB`
six tests did not catch it	`tools/test_claim_receipt_blob.py` has 7 tests; the seventh is the one that kills the mutant
I re-ran the mutant while writing this	baseline 7 passed. Truncating the comparison to `${BLOB:0:4}`: 1 failed, 6 passed, and the failure is `test_a_blob_prefix_is_not_a_blob`

What found it, and what did not

I wrote six tests for the fix. A receipt for the previous blob is refused. The
refusal names the blob. A legacy path-only receipt satisfies nothing. An honest
receipt passes. A stale line sitting first does not shadow a correct one below it. A
stale override is refused too, because an escape hatch that never expires is the
same hole.

All six passed. Then I mutated the guard: compare only the blob's first four hex
characters.

All six still passed. The hole was still open.

The seventh test is the one that kills that mutant, and its docstring says so:

Found by mutation, not by inspection: truncating the comparison to the blob's
first four hex characters survived every other test in this file.

Six is not a coverage number. It is how many tests I wrote before I felt finished,
and my sense of finished ran out exactly one test before the bug did.

I had already written the symptom down and filed it wrong

Two days before the fix, in a different commit, I described this:

passing needed two receipts for one file in a specific order, because the generic
loop reads the first matching line only. That ordering was load-bearing and
recorded nowhere.

I noticed that line order was deciding outcomes. I filed it as a documentation gap.

It was the bug. Inspection had the evidence in hand and did not convert it into a
failing test, because inspection asks what I believe the code does. Mutation does
not ask. It breaks the code and reports which assertion noticed.

What I might have wrong

The hook's own header states the residual and I am not going to soften it: git commit --no-verify skips every client-side hook, so no local hook is a complete
policy boundary. This gate stops an accident. It does not stop a decision.

I know the window and not the damage. Four days is a fact about dates. I did not go
back and check whether any claim actually shipped through the hole while it was
open, so I cannot tell you the blast radius, only its duration.

The receipts file is gitignored, so my own migration state is not independently
checkable by a reader. Take that part on my word or leave it.

And the mutation was hand-applied by me to a shell script, which means the operator
set is whatever I happened to think of. That is the same limitation that produced
the six-test blind spot in the first place, one level up.

If you have run mutation testing against your guard code rather than your product
code, I would like to know what it killed that you were sure was fine.

I build deterministic evaluation and verification tooling for LLM systems, and I am looking for my first full-time role in AI evaluation or QA engineering. Remote US Eastern, or Charleston SC. https://erikhill.dev

The gate, its tests and the fix: https://github.com/egnaro9/egnaro9.github.io

Verified by re-running the mutant against the committed hook while writing this. The judgement of whether it was worth saying is mine.

My arithmetic grader reads the first number in the sentence, not the answer

Erik Hill — Wed, 30 Sep 2026 15:55:43 +0000

Originally published at erikhill.dev. The numbers below are checked against the repository they come from.

My arithmetic grader reads the first number in the sentence, not the answer

This is a finding about my own harness. No model got the arithmetic wrong.

What happened

A week ago I published a post saying three of my factual-recall tasks were scoring
format rather than facts, and I ended it with a guess I had not tested. The guess
was that models read Answer with only the city name as a constraint on the output
and Two letters only as a description of the answer, and that the difference
decides whether you get a bare token or a sentence.

This week the same suite's arithmetic grader failed Claude Opus 4.8 on sixty
consecutive clean runs of a task that asks what 0.3 minus 0.1 is.

So I went and looked, and the guess held in a place I did not plant it.

The defect

def _number(expected: float, tol: float = 1e-6):
    def g(out: str) -> bool:
        m = re.search(r"-?\d+(?:\.\d+)?", out.replace(",", ""))
        return m is not None and abs(float(m.group()) - expected) <= tol
    return g

re.search returns the first match. So the grader reads the first number that
appears anywhere in the output and compares that to the expected answer. If a model
restates the question before answering it, the first number in the output is a
number from the question.

Seven of the eight tasks graded this way put a number in the prompt that is not the
answer. Compute 3 + 4 * 5 expects 23 and leads with 3. What is 15% of 200
expects 30 and leads with 15. What is 0.3 minus 0.1 expects 0.2 and leads with
0.3.

The evidence

claim	how it was checked
the grader takes the first number, not the answer	`modeldrift/suite.py:64`, `re.search(r"-?\d+(?:\.\d+)?", ...)`
7 of its 8 tasks lead with a number that is not the answer	enumerated over `SUITE`: math-order 3 vs 23, math-percent 15 vs 30, reason-older 30 vs 25, multi-step-math 3 vs 8, unit-minutes 2.5 vs 150, compare-negatives -0.5 vs -0.05, float-subtract 0.3 vs 0.2. Only compare-decimals leads with its own answer
it bites on runs where nothing errored	`float-subtract` fails on 9 models across runs with `reliability: 1.0`, worst Claude Opus 4.8 at 60 of 60
every model had the right answer	live calls through the probe's own path: gpt-4o-mini `'0.3 minus 0.1 equals 0.2. To one decimal place, the answer is 0.2.'`; opus-4-8 `'0.3 minus 0.1 is 0.2.'`; sonnet-5 `'0.3 minus 0.1 equals 0.2'`
changing only the instruction flips the outcome	same arithmetic, three wordings, three models, nine calls: as-shipped FAIL x3 with first=0.3; `Give only the number` and `Number only` PASS x3 with first=0.2
the same models pass the other number tasks in the same call	all three answered math-order `'23'`, math-percent `'30'`, unit-minutes `'150'`, bare, no restatement

What separates the task that fails from the seven that do not

This is the part that makes it more than a bug report.

The tasks that pass say "Give only the number" and "Number only". Those are
instructions about the shape of the output. The task that fails says "Give the
number to one decimal place". That is an instruction about the format of the
value, and it appears to invite the model to show that it has formatted it, which
means saying what it started from.

Last week's post guessed exactly that distinction from nine calls on three
factual-recall tasks, and said plainly that it was one story fitting nine calls. It
now fits twenty one calls, across two different grader families, with the same
split: an output-shape instruction gets a bare token, a description of the answer
gets a sentence.

I would rather have found it the other way around. A guess that survives a test I
did not design for it is worth more than a guess I go looking to confirm.

So I ran the experiment

The paragraph above used to say this was a pattern I had observed twice rather than
a mechanism I had demonstrated, and that the test was one sweep away. Here is the
sweep. Same arithmetic, same three models, three instruction wordings, nine calls.
The frozen suite was not edited, because its hash travels in published claims; the
variants were sent directly.

as-shipped  "Give the number to one decimal place"   FAIL x3   first=0.3
            opus-4-8     '0.3 minus 0.1 equals **0.2**.'
            sonnet-5     '0.3 - 0.1 = **0.2**'
            gpt-4o-mini  '0.3 minus 0.1 equals 0.2. To one decimal place, the answer is 0.2.'

"Give only the number"                             PASS x3   first=0.2   all answered '0.2'
"Number only"                                      PASS x3   first=0.2   all answered '0.2'

Nine for nine. The only thing that changed was the instruction, and it flipped every
row.

It also kills the alternative I could not rule out. I had written that 0.3 minus 0.1 is the canonical floating point example and a model might restate it because it
recognises the exercise rather than because of the wording. If that were the cause,
the restatement would survive Give only the number. It does not survive it once.

What I might have wrong

The experiment is three models and one task. It demonstrates that the wording
controls the restatement here; it does not establish how far that generalises, and I
would not carry the claim to a model family I have not called.

And the count needs a caveat. Seven of eight tasks are vulnerable in principle,
because their prompts lead with the wrong number. Only one of them is observed
failing at scale. The others pass because these models happen to answer them
tersely, which is not a property of my grader, it is a property of this month's
models. The grader is one style change away from failing all seven.

The fix is not obvious either, and I have not made it. Reading the last number
instead of the first would pass '0.3 minus 0.1 equals 0.2' and would also pass a
model that shows its work and lands on the wrong final value. Requiring a bare
number would be honest but would turn an arithmetic task into a formatting task,
which is the exact complaint this post is making.

If you grade numeric answers with a regex: check which end of the string you are
reading, and check it against an output that restates the question. Mine had been
reading the question back to itself for months, and the board recorded the result as
an arithmetic failure.

Live board: https://egnaro9.github.io/model-drift/
Suite, graders and runner: https://github.com/egnaro9/model-drift

The finding was surfaced by the drift detector. The live calls, the enumeration and the judgement of whether it was worth saying are mine.

Twice the data, less power: my stability rule got blinder the harder I looked

Erik Hill — Wed, 23 Sep 2026 23:16:26 +0000

Originally published at erikhill.dev. The numbers below are checked against the repository they come from.

Twice the data, less power: my stability rule got blinder the harder I looked

This is a finding about a measurement rule, not about a model. The models did not change.

What happened

I was comparing two models on a frozen 159 task suite and I had a problem most eval
code does not admit to having: the same model, asked the same question three times,
sometimes disagrees with itself. I had written a rule for that. A task where a
config does not agree with itself is discarded, not counted. Call it strict.

On the first three repetitions strict gave me Haiku ahead 7 to 1, eight
informative tasks, p equal to 0.070. Not significant.

So I looked at why, found that strict was throwing away thirteen tasks, and wrote
a second rule that tolerates a minority disagreement. Call it rate. It gave Haiku
ahead 13 to 2, fifteen informative tasks, p equal to 0.0074. Significant.

I had run the conservative rule, got a null, and then built the rule that returned
the result I wanted. That is also exactly what everyone who p hacks believes about
themselves.

The evidence

claim	how it was checked
the two rules disagreed on the same data	reps 1 to 3, suite `combined-159.json`: `strict` 7 to 1, informative 8, p 0.070; `rate` 13 to 2, informative 15, p 0.0074
the second rule was written after seeing the first result	stated in `runs/PREREGISTRATION.md`, committed before reps 4 to 6 existed, so the git timestamp is checkable against the run files' mtimes
the replication was pre-registered, analysis included	predictions, refutation conditions and fixed parameters in `runs/PREREGISTRATION.md`; the analysis in `tools/replicate.py`, written while the sweeps were still running
the prediction I got wrong, I got wrong in public	predicted `strict` would again fail to reach significance; on fresh data it reached p 0.0215
more repetitions raised the discard count	2 reps: about 8.7 discarded, about 10.3 informative. 3 reps: 13 discarded, 8 informative
pooling all six repetitions lost the result	reps 4 to 6 alone: informative 10, discards 13, p 0.0215, decisive. Reps 1 to 6 pooled: informative 8, discards 17, p 0.0703, not significant

What the extra data did not buy

This is the part I did not expect and the reason the post exists.

Every extra repetition is another chance to observe a config disagreeing with
itself, and strict discards a task the moment it sees one. So the discards climb
with the data. Thirteen became seventeen. Informative tasks fell from ten to eight.
The p value went from 0.0215 to 0.0703, which is to say a decisive result became a
null by adding measurements to it.

In the limit the rule throws away every genuinely stochastic task. More measurement
cannot fix that. Only a different rule can.

A conservative rule is not a free choice. It will never report noise as signal, and
the price is that it grows blinder the harder you look. That price is measurable,
and until I measured it I had been describing the conservatism as pure upside.

What the pre-registration was actually for

The rule I trust reached significance on data it had never seen. strict on reps
4 to 6 alone: Haiku 9 to 1, ten informative tasks, p 0.0215.

I had predicted it would fail again. The pre-registration says in advance what to
do about that: "If 1 holds and 2 fails, that is stronger than predicted, and I will
say the prediction was too conservative rather than claiming I called it." So that
is what I am saying. I did not call it.

The parameter is the part worth stealing. rate_margin was fixed at 0.5 in
advance, and afterwards I removed the flag from the command line entirely, along
with --alpha, because a threshold a reader can dial after seeing the result is
not a threshold. The registered value did not change. The knob did.

What I might have wrong

The weakest link is that this is one suite and two models. Haiku 4.5 beating Sonnet
4.6 here is a claim about a suite made of trap questions, not a capability ranking.
A suite built to catch specific slips measures susceptibility to those slips, and I
would not carry that sentence anywhere else.

The reading I cannot rule out is that rate is simply the better rule and strict
was never worth defending, in which case the whole discard analysis is an
elaborate defence of a mistake. The replication is evidence against that, since
strict reached significance on fresh data, but one replication at two models is
not much.

And I should be honest that the pooled result has an innocent reading. Reps 1 to 3
and reps 4 to 6 were run at different times, so pooling them mixes two sampling
windows. I attribute the power loss to the discard mechanism because the discard
count is the thing that visibly moved, thirteen to seventeen, but I have not run
the version of this that would separate those two explanations.

If you maintain an eval suite with a stability or flakiness filter: count what it
discards, and count it again after you add data. If the discard count rises faster
than your informative count, your filter is spending your sample size, and the
direction of that trade is not obvious from the code.

Suite, rules, pre-registration and replication: https://github.com/egnaro9/pi-eval

The numbers above are quoted from FINDINGS.md and runs/PREREGISTRATION.md in that repository. The judgement of whether this was worth saying is mine.

My factual-recall tasks were scoring format, not facts

Erik Hill — Wed, 23 Sep 2026 23:15:35 +0000

Originally published at erikhill.dev. The numbers below are checked against the repository they come from.

My factual-recall tasks were scoring format, not facts

This is a finding about my own harness. The suspect is the probe, not the models it measures.

What happened

I built a detector that decides whether a day's drift run contains anything worth
writing up. The first thing it did was accuse my own suite. One task,
fact-element, had failed on three or more providers on the same day, on 46
separate days.

The task is this:

What is the chemical symbol for gold? Two letters only.

Graded by exact match against au, after stripping whitespace and lowercasing.

Claude Sonnet 5 failed it 60 times out of 60. GPT-4o mini failed it 59 out
of 59. Llama 3.1 8B, 23 out of 23. Those are not knowledge results. A model
that scores 83% across a 35-task suite does not have a 100% failure rate on
whether gold is Au.

The evidence

claim	how it was checked
`fact-element` failed on 3 or more providers on the same day, 46 times	cross-provider flip analysis over every stored run in `dashboard/drift_board.json`
those were graded failures, not outages	every run counted had `reliability: 1.0`, so no call errored and none was truncated
the models know the answer	`gpt-4o-mini` answered `'The chemical symbol for gold is Au.'`; `claude-sonnet-5` answered `'Au\n\nThat's the chemical symbol for gold, derived from the Latin word aurum.'`
the same models pass a near-identical task	all three answered `fact-capital` with exactly `'Tokyo'`, 0 failures in 59 and 60 runs
one trailing period is the whole difference	`gpt-4o-mini` answered `fact-planet` with `'Mercury.'` and was recorded wrong 59 times out of 59
a model that formats tersely passes cleanly	`claude-opus-4-8` answered `'Au'`, `'Mercury'`, `'Tokyo'`, and fails `fact-element` on 1 of 60 runs

The answers above are not from the board. They came from nine live calls made
through the probe's own call path, printed with repr() so punctuation and
markdown are visible rather than inferred.

What stayed green anyway

fact-capital is the control, and it is the reason this is a finding rather
than a theory. It asks for the capital of Japan and every model in the table
answers 'Tokyo', exactly, on every run. Same grader, same strictness, same
models, same day. Nothing about exact matching is broken in general.

The three tasks differ in one way. fact-capital says "Answer with only the
city name", which is an instruction about the shape of the output.
fact-element says "Two letters only" and fact-planet says "One word",
which are descriptions of the answer. Models appear to read the first as a
constraint they must obey and the other two as a hint about what is being asked,
and then answer in their house style: a full sentence, a bolded token with an
etymology, a word with a period after it.

What this run could and could not have seen

The suite has 35 tasks, so the smallest accuracy change it can print is
100/35 = 2.86 points. fact-element and fact-planet together are 5.7 points
of every affected model's score, permanently, for reasons that have nothing to
do with the models.

Worse than the points: these tasks are tagged factual-recall, and that tag
feeds a per-category breakdown on the public board. GPT-4o mini's
factual-recall score has been reading as a knowledge number when two thirds of
it is a formatting number. Anyone comparing models on that column, including
me, was comparing how terse they are.

What I might have wrong

The part I am least sure of is the explanation, not the measurement. The numbers
are solid and the live answers are quoted verbatim, but "models treat answer
with only X as a constraint and X only as a hint" is one story that fits nine
calls. I have not tested it. The honest version is that I know the format
differs and I am guessing at why.

Two things genuinely argue against my reading. Grok 4.5 fails fact-element on
26 of 49 runs, which is close to a coin flip, and a rigid house style should not
produce that. And Claude Sonnet 5 failed fact-planet on 38 of 60 historical
runs but answered 'Mercury' cleanly when I called it today, so at least some
of this moves over time and is not a fixed property of the model.

There is also a real question about what the fix should be, and I do not think
it is obviously "loosen the grader". A grader that accepts 'Mercury.' also
accepts a model that ignores the instruction, and the instruction was part of
the task. Making the grader lenient would convert a visible measurement problem
into an invisible one. The alternative is to say plainly that these are
instruction-following tasks and retag them, which changes what the board has
been reporting for three months.

If you maintain an eval suite with exact-match graders on short answers: check
what your passing models actually return, not just whether they passed. The
board I built recorded which tasks failed for three months and never once
recorded what the model said, which is why this took a live call to see. That
gap is the more embarrassing half of this post.

Live board: https://egnaro9.github.io/model-drift/
Suite, graders and runner: https://github.com/egnaro9/model-drift

The finding was surfaced automatically by the drift detector. The investigation, the live calls and the judgement of whether it was worth saying are mine.

My mutation score was 1.000 in two places and the denominators disagreed

Erik Hill — Wed, 23 Sep 2026 01:24:56 +0000

I published a number, went to confirm it, and the confirmation disagreed with it. The number was fine. The confirmation was broken, and it was broken in a way I had already written a warning about.

The number

My evidence-bundle verifier has one metric I care about more than coverage: for every place in the code where the tool can refuse, is there a test that notices if I delete that refusal? So the sweep replaces each refusal site with a pass, one at a time, and reruns the suite. A site nobody catches is a gate that is not a gate.

CI, on the pushed commit:

baseline clean; 168 refusal sites (4 excluded, see EXCLUDE)
MUTATION SCORE: 168/168 = 1.000

The build fails under 0.99, so this is a gate rather than a dashboard.

The confirmation

I ran the same script locally, expecting 168:

MUTATION SCORE: 170/170 = 1.000

Same tool, same command, same score, two different denominators. 168 and 170 are both "everything", which is exactly why it is easy to miss. If the local run had said 0.94 I would have investigated in seconds. It said 1.000, so it looked like agreement.

Why

My working tree had 28 files that were not committed: 15 modified, 13 untracked. The verifier source itself was clean, so no new refusal sites had appeared in the thing being measured. What had changed was the suite and the fixtures around it, which decides which mutants get caught and which sites get excluded.

So the population moved. The sweep did not measure a different quality of the same system; it measured a different system.

The part I would rather not write

Here is a comment sitting in my own CI config, committed months ago, above the sibling checkouts:

# The score is only comparable if the suite is. Without these checkouts
# the real-bundle tests SKIP, four mutants stop being caught, and the
# denominator changes -- a mutation score that depends on what happens
# to be on the machine is the same host-dependence class this verifier
# exists to refuse. Check them out so CI measures what a developer does.

I wrote that. It names the failure precisely. And I still ran a local sweep to check a published figure, because checking felt like the careful thing to do.

Knowing a rule is not the same as being governed by it. The gate that actually held here was not my knowledge, it was CI, because CI had no choice about which tree it ran on.

What I changed

The number I put in front of other people is the one CI computed on the commit they can check out. Not the one my laptop computed on a tree only I have.

That sounds obvious written down. It was not obvious while I was doing it, because the local run felt like more diligence rather than less.

Two questions I would genuinely like answers to:

If you publish a metric out of your own repository, is its denominator stable across a clean clone and your working tree? Have you checked, or does it just feel stable?
Does your number get quoted from a CI log, or from whatever you last ran locally?

If you check and the two disagree, I would like to hear what moved. I suspect this is common and mostly invisible, because agreement on the headline figure hides disagreement underneath it.

Three tools said my resume was clean. One of them was not installed.

Erik Hill — Fri, 18 Sep 2026 02:26:19 +0000

Today I audited my own public claims and found a false one. My resume said the Android game I shipped, Tap Dodge Rush, was built with "Kotlin / Jetpack Compose". The app has never contained a line of Compose. The build file says so in my own words:

plugins {
    alias(libs.plugins.android.application)
    alias(libs.plugins.kotlin.android)
}

// The game is a custom View painting to a Canvas, driven by an Activity with
// an XML layout. Compose came from the project template and was never used.

The claim was falsifiable only by me. That repo is private, so for months the one person who could check it was the person who wrote it. That is most of why it sat there.

Fixing the HTML and the markdown took a minute. Then I went to check the PDFs, and that is where the real story is.

Three tools, three clean results

$ for f in Erik_Hill_Resume*.pdf; do
    printf '%s %s\n' "$(pdftotext "$f" - 2>/dev/null | grep -ci jetpack)" "$f"
  done
0 Erik_Hill_Resume.pdf
0 Erik_Hill_Resume_DRAFT.pdf
0 Erik_Hill_Resume_VARIANT_A.pdf
0 Erik_Hill_Resume_VARIANT_B.pdf

Clean. For a second opinion I ran a small extractor I had written earlier that pulls the PDF content streams, inflates them with zlib, and searches the literal text. Zero hits. Third pass, the blunt one:

$ strings Erik_Hill_Resume.pdf | grep -ci jetpack
0

Three independent tools, three clean results. I was one command away from calling it done.

The check that broke it

Before closing it out I searched for a string that has to be in the file.

$ pdftotext Erik_Hill_Resume.pdf - 2>/dev/null | grep -ci erik
0

Zero hits for "erik" in Erik Hill's resume. That cannot be true.

$ command -v pdftotext
$ pdftotext Erik_Hill_Resume.pdf -
zsh: command not found: pdftotext

pdftotext was never installed on this machine. In a pipeline the error goes to stderr, grep reads an empty stdin, and grep honestly reports zero matches in nothing at all. The exit status I was reading belonged to grep. Every clean result in that sweep was a measurement of the empty string.

I run sweeps like this with an agent driving the shell, and the agent reported the same clean result. It was reading the same zero I was. The tooling did not fail quietly because it was an agent. It failed quietly because a pipe swallows stderr and nobody asked the pipeline to prove it could find anything.

The trap inside the fix

Here is the part I did not expect. strings also passes a positive control:

$ strings Erik_Hill_Resume.pdf | grep -i erik
/URI (https://linkedin.com/in/erik-hill-98895575)>>
/URI (https://erikhill.dev/)>>

Two hits. A green control, on a tool that cannot read a word of the body text. Those are link annotations, which PDF stores uncompressed, while the body text sits in Flate-compressed content streams that no byte search will ever reach.

So the control passed in a layer the target does not live in. A positive control only proves the tool can read the layer the control sits in.

The zlib extractor failed for a third reason: these PDFs use subsetted fonts and hex strings, so scanning for literal parenthesized text finds nothing. That file has 27 streams, all 27 inflate cleanly, zero literal-paren text operators and 4721 hex-string ones.

The tool that actually reads a PDF here is pypdf, in a throwaway venv:

$ python3 -c "
from pypdf import PdfReader
t=''.join((p.extract_text() or '') for p in PdfReader('Erik_Hill_Resume.pdf.bak-20260825-1859').pages)
print(len(t), t.lower().count('erik'), t.lower().count('jetpack'))"
4812 5 1

That is a stale copy that still carries the claim, so the transcript is reproducible today. Nine PDFs had it, including the two copies in ~/Downloads that get attached to applications and the one my site serves.

The two files every sweep missed

An adversarial pass afterward found two more, and they are the best part. Both are real PDFs whose names do not end in .pdf, so every *.pdf glob had skipped them:

$ find ~/Desktop ~/Downloads ~/.Trash ~/.claude -type f ! -name '*.pdf' |
  while read -r f; do
    [ "$(head -c 5 "$f" 2>/dev/null)" = '%PDF-' ] && echo "$f"
  done
.../.Trash/Erik_Hill_Resume.pdf.bak-20260825-1859
.../uploads/Erik_Hill_Resume.pdf.old-0719

The second sits in the uploads directory that LinkedIn Easy Apply pulls from. A stale resume, one filename character away from invisible, in the exact folder an application form reaches into.

A second checkout of the same Android app is also still on disk with the full Compose stack in its build file, so a future grep that strays one directory sideways will conclude the app really is a Compose app.

The fix, and one number

The line now reads "Kotlin / Java, Android Views + Canvas, AdMob". Kotlin alone would have replaced a false claim with a misleading one: of 1204 lines in the main source sets, 710 are Java and 494 are Kotlin. The live page and the PDF it serves both carry the corrected line, checked with curl and pypdf rather than trust.

Limits

pypdf is not a universal answer either. It returns nothing on a scanned PDF, which needs OCR, and it will hand you the same confident zero. A magic-byte sweep is slower than a glob, which is why nobody runs one until a glob has already lied to them.

This is one finding from a wider audit of my own published claims. I am not listing the rest yet, because several are still being verified, and shipping a half-verified claim audit would be the same mistake in a new costume.

The rule I am left with: a search that returns nothing is not evidence of absence until you have shown the search can return something, using a control that lives in the same layer as the thing you are hunting.

So run one tonight. Take the sweep command you actually trust, over a format you actually search, and point it at a string you know is in the body of that file. If it comes back zero, which layer of the file was your tool reading?

I build evaluation infrastructure for LLM systems and I am looking for work. contact@erikhill.dev

My board never scored an outage as a regression. My evidence couldn't prove it.

Erik Hill — Mon, 31 Aug 2026 05:53:52 +0000

I put a preprint on arXiv this week. It is about forging evidence bundles that my own verifier calls clean. I mutated the sites where the verifier is supposed to refuse, counted how many mutations survived, and reported the number. The cheapest forgery that survived was four bytes.

Two people commented. Within twenty four hours they had pressed on the same weak point from opposite directions, and one of them was not talking about my paper at all.

The comment

Nguyen Thanh Vinh:

A refusal-site score is bounded by the refusals that already exist. A comparison that never runs because the field was deleted is the visible version of that, since there is still a site to mutate. The version that stays invisible is an obligation in the spec that no site was ever written for, because it never enters the denominator at all.

He is right, and it is worse than he could have known from the outside. Four forgeries survived. Three landed in classes I had sites for. The fourth landed in an obligation with nothing to mutate, so it could never have appeared in the score regardless of how the sweep went. The number counted the places I had already thought to look.

He asked for the one figure that bounds it: how many spec obligations have no refusal site mapped to them.

I do not have it. SPEC.md carries 39 normative obligations, 37 MUST and 2 MUST NOT, and there is no artifact mapping any of them to sites. "Obligations with no refusal site" is unmeasured, not small. That is the honest state and it is the next thing I am building.

The same argument, aimed somewhere else

The same day, Sarvex Jatasra published a correction to his own earlier piece on pricing agent work, crediting Vinh with finding the hole:

every price and every score quoted for agent work is measured on the subset that was kept, which makes each of them a lower bound on the real cost rather than an estimate of it

Vinh's comment on that article and his comment on my paper are one argument in two costumes. On Sarvex's post the missing denominator is the runs a person abandoned and rewrote. On mine it is the obligations nobody wrote a check for. Both quote a number measured on survivors. Neither publishes the denominator.

Sarvex left a comment on the paper too, and it is the same shape a third time: my counts of 37 through 143 sites are totals over a population re-enumerated on every run, so a site that flipped to surviving cannot be told apart from one deleted while another arrived. Also correct. I had already pulled that per-site claim from the paper for exactly that reason and the v2 with the corrected wording is queued.

The part I got to feel smug about for one minute

I run a public drift board that re-tests 17 models against a frozen 35 task suite daily. In August, Google's endpoint started rate limiting, and a model scored 0.0286. One task out of thirty five.

That is not a bad model. That is an absent measurement wearing a score. Average it in and you publish a false regression. Drop it silently and you publish a stale standing with no trace anything was missing. Both are wrong and only one looks wrong.

The board already handled this. There is a reliability floor at 0.5, and low-reliability points are excluded from the accuracy line while being kept in the Reliability metric, because reliability genuinely did drop and that is the true signal. I shipped that on 2026-07-21, the same day I published a post here called "Distinguishing wrong from absent."

So when I read Sarvex's prescription:

Tag every invocation with whether it ended in an artifact somebody kept, then publish runs per accepted change beside your cost per completed task, on the same slide, for the same window.

my first reaction was that I had been doing that for six weeks.

The minute ended

My board was right. My published evidence could not demonstrate it.

The board ships a VAC bundle: hash-pinned artifacts that an offline verifier re-derives from the committed rows, so a reader can check the claim without trusting me. That bundle did not publish the floor. It did not publish which run had been excluded. It published the standing and nothing about the population the standing was computed over.

Which means the verifier, recomputing honestly from the same data, got different answers for three Google models and refused the bundle. It was correct to refuse. Two of the verdicts it recomputed were inverted against what the board displayed. The registry entry sat PENDING for eight days with a refusal message that read like the board was broken.

The board was not broken. The evidence was incomplete in exactly the way Vinh described: the excluded runs never entered the published denominator, so nothing downstream could reconstruct the decision.

The fix, yesterday, was to publish the floor as a pinned value and require both numbers: the current qualifying standing, and the latest observed run with its reliability, whether or not it qualifies. Three of seventeen models currently carry a disqualified latest run. An outage now reads as an outage in the evidence, not only on the chart.

And then, writing this post

I went to check my own claim before publishing it, because a post asserting "the board publishes X" should be verified against the board rather than against my memory of it.

REL_FLOOR = 0.5 is defined twice. Once in modeldrift/policy.py, once as a JavaScript literal in the dashboard. They agree today. Nothing makes them agree tomorrow, and they are not redundant: the Python one decides the qualifying standing, the JS one decides what the chart draws. If they drift, the table and the chart disagree about the same model on the same day, each internally consistent, with no error raised anywhere.

Yesterday's work taught the verifier to cross-check the floor against two surfaces. The chart is the fourth surface and it was left out. It is filed now.

What it adds up to

Knowing a principle and having it enforced everywhere are different states, and the distance between them is invisible until something in the world goes down or a stranger asks the right question. I had the floor in the chart, then in Python, then in the evidence, and I still found a fourth copy outside the check while writing a post about it.

The mutation score has the same shape and I have not closed it. Until an artifact maps all 39 obligations to sites, that score answers "how many of the checks I wrote can be defeated" while readers will hear "how many ways this can be defeated." Those are different questions. Only one of them currently has a denominator.

Paper: https://arxiv.org/abs/2608.26183
Board: https://egnaro9.github.io/model-drift/
Verifier and registry: https://github.com/egnaro9/vac-protocol

Thanks to Nguyen Thanh Vinh and Sarvex Jatasra, who between them cost me a claim and handed me the next two pieces of work.

Five Wrong Root Causes, and the Fifth Was in the Post

Erik Hill — Sat, 22 Aug 2026 09:29:27 +0000

I wrote most of a post about how a failing system's logs seduce you into confident wrong root causes. Then I committed one, in the post itself. Within an hour of publishing, the maintainer of the Wine packages I had written about corrected me in the open.

This is the whole arc, including the part where the instrument I trusted most was the one that lied.

The setup

The Wolf Among Us: the macOS build is 32-bit, dead since Catalina. The Windows build runs under Wine, but it is wrapped in Steam DRM, so a logged-in Steam client has to run in the same prefix. The real boss fight is therefore: log into Steam, under Wine, on a Mac, in 2026.

Stock Wine could not do it. The login page rendered sometimes, polled forever, and died:

Login: Failed to poll auth session. Result 2. Transport Error: 2

Four wrong answers I caught myself

Every one of these was supported by the failing system's logs:

Winsock. Unknown error 10045 (WSAEOPNOTSUPP) sat right next to the failing poll. Obvious. Wrong.
The CEF sandbox. Disabling it changed nothing, and one flag deleted the UI, which I briefly celebrated as a new bug.
Missing Gecko. I "confirmed" Wine's HTML runtime was absent by searching only the working system for it. It was present in both. A one-sided search dressed up as a comparison.
GL/ANGLE. Real errors, and forcing software rendering removed them completely. Login still failed at the same rate.

The instrument that broke the tie felt authoritative: diff a working system against the failing one. CrossOver logs the same winsock noise and logs in fine.

signal	stock Wine (fails)	CrossOver (works)
`WSALookupServiceBegin` failures	30	5
`10045` errors	15	2
failed auth polls	24	0

Anything present in both is noise. What remained on only the failing side was two fixme lines I had scrolled past all night:

fixme:bcrypt:export_asymmetric_key unsupported blob type L"OpaqueKeyBlob"
fixme:ncrypt:map_ntstatus unhandled status 0xc0000002

Upstream Wine dropped its gnutls-backed bcrypt after 11.6, and the rewrite never implemented opaque export for asymmetric keys. It is still unimplemented in master. That gap is real. I filed it: WineHQ 60214.

The part that actually worked

I got a Steam-DRM game running on a fully free stack: a GPTk-based Wine (which ships Apple's D3DMetal) plus an archived early-2025 Steam client old enough to predate the heavier client. Login by phone confirmation, library loaded, game ran. Verified, witnessed, screenshotted. I also revived the discontinued Whisky app from its archived runtime, which is the one piece of this whole night nobody has disputed.

So I published. Wrote it up, filed the bug, posted the thread. Felt good.

The fifth wrong root cause

Within the hour, Gcenx, who maintains the official WineHQ macOS packages, replied. Two of my claims did not hold up.

One: I said only Apple's D3DMetal could solve the rendering wall, that the community DXMT layer was a dead end because I hit a cross-process swapchain error. Wrong. Gcenx and YupItzAfi run the current Steam client with a rendering fix alone: a small steamwebhelper wrapper forcing --in-process-gpu. Gcenx does it on the MoltenVK path; DXMT is a separate self-compiled-Wine route. I had hit the same wall and quit early, because I passed the flag to Steam.exe instead of into steamwebhelper.

Two, and this is the one that matters: I framed the bcrypt gap as the cause of the login failure. It is not proven. Gcenx has the current client working via the rendering fix alone, which means rendering, not the crypto gap, may be the real blocker. My own attempt to isolate the two was inconclusive. The API gap is real; its role as the Steam login cause is not.

Sit with that. The differential diff, the instrument I had just praised for cutting through four wrong theories, produced a fifth wrong theory. It felt like the answer because it was cleaner than the others, not because it was proven. Clean is not the same as correct.

What I did about it

Corrected every surface the same hour. Retitled the WineHQ bug to describe the verifiable API gap and stop implying it was the Steam cause. Refocused and renamed the repo to what it can honestly stand on, the bug and this post-mortem, and pointed anyone who wants to actually run Steam at the maintained path: the official WineHQ packages plus the steamwebhelper wrapper. Credited Gcenx and YupItzAfi, who were right and generous with their time.

Nothing had to be redacted, because the verified core held: the game did run, Whisky did revive, the API gap is real. Only the interpretation was wrong, and interpretation is exactly what a differential diff cannot certify. It tells you what differs. It does not tell you which difference is the cause.

What I am keeping

A failing system's logs support every theory you bring to them. Four of mine, equally.
The instrument you trust most is the one to distrust hardest. Mine handed me a confident wrong answer wearing the costume of rigor.
Being right was never the measure. Four wrong causes I caught; the fifth I shipped, and someone better-informed caught it. What I could control was how fast and completely I fixed it.

That last part is the whole discipline, and it is the only reason this is a post I am glad I wrote rather than one I wish I could delete.

One capital letter made my verifier call a failing score clean

Erik Hill — Sun, 16 Aug 2026 06:41:51 +0000

I build a thing that checks whether AI evaluation claims are true. Someone broke it, I fixed it, and then I broke it four more times myself.

Here's the cheapest one.

A bundle carries security test results. Each case has a severity, and the score weighs failures by it:

_WEIGHTS = {"none": 0, "low": 1, "med": 2, "high": 4, "critical": 8}
total_w  = sum(_WEIGHTS.get(c.get("severity"), 0) for c in graded)
failed_w = sum(_WEIGHTS.get(c.get("severity"), 0) for c in graded if not c["passed"])

Take a real bundle with three failing cases. Change critical to Critical, on the failing rows only. Their weight drops to zero and leaves the numerator. The passing rows keep theirs. The score divides cleanly:

severities: [critical, critical, high, med, med]
        ->  [critical, Critical, high, Med, Med]

vulnerability_score:  0.0    (honest value: 0.4545)
sha256:               re-pinned honestly
verifier:             exit 0, "structural verification: PASS"

No exception. No fallback branch. Ordinary arithmetic, all the way down. .get(sev, 0) treats an unknown label as harmless, so a weight table I called "frozen" was actually one the issuer controlled.

And my own spec had written the hole down as a decision: "an unknown severity weighs 0."

Three more, same shape

A lie typed as a string. The summary comparison walked the JSON and returned early on anything that wasn't a number. Retype every headline as "9999" and nothing is ever compared. No artifact touched, no hash re-pinned.
Delete the check instead of breaking it. Nothing required a listed artifact to be covered by a check. Earlier hardening made breaking a check a named failure; deleting one stayed free.
Delete the stamp instead of faking it. Four comparisons were guarded on the artifact-side key existing. Corrupt those fields and you get refused by name. Remove them and nothing happens.

That last one generalizes: a comparison guarded on both operands existing is not a check, it's a suggestion. The party supplying one operand decides whether the comparison happens at all.

Then I stopped collecting anecdotes

Four bugs is a story, not a measurement. So I asked a different question: of every place this verifier can say no, how many are actually protected by a test?

The verifier rejects bundles by appending named reasons. I disabled them one at a time, replacing each failures.append(...) with pass, and asked whether anything noticed: the unit suite, a liveness control, or any of the 16 committed tamper fixtures.

112 refusal sites
 37 caught
 75 SURVIVED, deletable in silence, everything still green
mutation score 0.330

Two thirds of the ways this thing could say no were unprotected.

Worse: I had a CI job named invalidation-liveness, written specifically to prove the verifier can block. It caught zero of the 75. Each fixture exercises one refusal path, so deleting any other refusal leaves all sixteen verdicts unchanged. The gate built to prove the gate works had, against this operator, a score of zero.

The number I didn't expect

I fixed the four bugs. Added a regression fixture for each, the responsible post-audit move. Re-measured:

before fixing   37/112 = 0.330
after fixing    39/119 = 0.328

Fixing the four defects I'd actually found, each pinned against return, did not meaningfully improve coverage. It closed four real holes, which is real value, but it left the rest of the gates exactly as unprotected as before, because four regressions sample the failures you discovered rather than the population you own.

Testing every refusal instead:

0.330  ->  0.941  ->  1.000

The defects you find are a sample. The gates you own are the population.

The part where my tools lied

Six times, during this work. Twice inside scripts I wrote to hunt this bug. The worst:

The first run after hardening scored a perfect 1.000. I almost wrote that down. It was false. The harness ran pytest -x against a baseline that was already red, so pytest exited nonzero for every mutant, every mutant scored "caught," and nothing was measured at all.

A tool built to detect checks that pass without checking produced a check that passed without checking. And it presented as the best possible result.

The fix is the rule the whole exercise is about, turned on itself: the sweep now aborts unless the clean baseline is green. And when it finally did reach a real 1.000, I planted a refusal on an unreachable branch and confirmed the sweep still reported SURVIVED before believing the number.

What I'm not claiming

I sent this to an outside reviewer specifically to get the framing attacked, and several claims came back too big:

1.000 is not "the verifier is correct." It's a complete score against one narrow operator (disable a refusal) over an enumerated population. Mutated comparisons, boundaries, and control flow are unmeasured.
The honest name is refusal-append liveness coverage. The denominator is a source-level proxy that misses raises, early returns, and every fail-open behavior that happens before a refusal is reached.
"Vacuous pass" is a label, not a discovery. This is well-trodden ground: vacuity detection in model checking, the oracle problem, mutation adequacy, fail-open validation.
Every number here is self-measured on a system I wrote, verifying a registry whose entries are all my own repos. That's one multi-repo fixture, not independent evidence.

The one that surprised me

The closure fix, every listed artifact must be read by some check, was written to close one hole. Pointed at my live registry, it refused every remaining issuer I had.

And in every single case, the unchecked artifact was the human-readable one. The rendered report. The contract. The .md file. The machine-readable JSON was always bound. One contract announced "6/6 seeded defects fixed". True, and verified by nothing.

I don't have the sample to call that a law. Three repos, one author. But as a hypothesis it's cheap to test on your own work: find the document your users actually read, and ask what binds its numbers to the data.

Everything's public: verifier, spec, the mutation tool, and the write-up with the parts that came back wrong. github.com/egnaro9/vac-protocol

If you run something similar: what does your suite do if you delete one of your gates? Not break it. Delete it. I'd genuinely like to know whether 0.330 was unusual or ordinary.

I built an answer key for eval suites: six models broken on purpose, exactly.

Erik Hill — Fri, 14 Aug 2026 13:17:27 +0000

This week I launched evalmut — mutation testing for eval suites. Its weakness, stated in the paper: I chose the mutations, so a suite author can call them unrepresentative. Fair.

So I built the version of the argument you can't dismiss: reference-fleet, six deterministic models, each broken in exactly one documented way, at a stated seeded rate. Not trained — constructed. The defect count over a fixed request set is a constant, not a sample, and the certificate is a test file you can run:

citation-hallucinator — fabricates well-formed, on-topic, nonexistent references (the Mata v. Avianca failure)
constraint-dropper — honors instructions 1..N-1, silently drops the last
refuse-then-comply — refusal preamble, full compliance after
tool-arg-swapper — schema-valid tool call, first two argument values transposed; executes cleanly into the wrong place
sycophancy-flip — endorses whatever the user asserts
stale-cutoff — confidently unhedged past its knowledge boundary

The framing is metrology's certified reference materials: NIST ships samples of known composition so you can prove your instrument reads correctly. This is that, for eval suites.

Then I pointed real suites at the fleet. The audit protocol is paired: a defect counts as detected only if the suite fails the defective response AND passes its clean twin — a suite that fails both isn't detecting anything.

The board's first result, measured (paired protocol, zero false alarms):

naive-contains — the "did it answer, is it on topic, did it not refuse" suite most teams start with — detects 1 of 6 defect classes (refuse-then-comply). Citation fabrication, dropped constraints, swapped tool arguments, sycophancy, and confident staleness all pass it invisibly, at detection rate 0.000, n=200 each.
A diligent gradecore suite and a real promptfoo run with docs-style asserts both detect 6 of 6 at rate 1.000 — with a disclosure that matters: I wrote those suites knowing the task's ground truth. Their perfect rows prove the ceiling is REACHABLE (every fleet defect is catchable by a suite that checks everything checkable), not that typical suites reach it.

The gap between those two rows is the product: the distance between the suite you have and the suite the task admits.

One honesty note, because the whole project is about honesty notes: what's audited is a suite archetype executed by its real engine — a real promptfoo run, real gradecore graders — for this task format. No named public benchmark's own dataset is being scored. The claim is narrower and therefore checkable: "a suite built this way misses this defect class at this rate." And the reproducibility claim is CI-enforced: the runner refuses to stamp results from a dirty tree, and a CI job re-runs the entire audit on every push and goes red unless the published numbers reproduce byte-for-byte. That gate caught three real problems before this post — including a provenance hole in my own board that would have inverted 12 of 18 rows.

Board: https://egnaro9.github.io/reference-fleet/
Repo: https://github.com/egnaro9/reference-fleet — python audit/run_audit.py reproduces every number.

If you maintain an eval suite: which of the six would yours catch? That's an empirical question now.