Originally published at erikhill.dev. The numbers below are checked against the repository they come from.

The gate could not see what it was gating

This is a finding about my own tooling. The gate it describes is mine, and so is the hole.

What happened

I keep a pre-commit hook on my portfolio repo that refuses a commit touching a
claim-bearing file unless a receipts file carries a receipt for it. It never judges
whether a claim is true. It checks that a receipt exists and that its pointer
resolves: a local path has to exist, an https pointer has to answer 2xx. Prose alone
cannot satisfy it.

For four days, any file that had been reviewed once was exempt from review forever.

The defect is one line

LINE=$(grep -F -- "$f ::" "$ACK" 2>/dev/null | head -1)

head -1 on a pathname. The first line in the receipts file that mentioned the path
won, whatever it said and whenever it was written. A receipt recorded for an earlier
version of a file kept satisfying every later version of it.

So the gate reported success along a path where it never examined the staged
content. That is the precise failure I build tooling to find, sitting in the tooling
I built to find it.

The evidence

claim	how it was checked
the defect line existed	git show 47f82a1^:.githooks/pre-commit line 61, the head -1 shown above
the window was four days	the hook's full history is four commits: installed c9af511 2026-08-23, fixed 47f82a1 2026-08-27
selection is now by blob	the fix reads BLOB=$(git rev-parse ":$f") and greps ":: blob $BLOB", so line order decides nothing
a stale receipt is now refused by name	the hook prints no receipt carries the staged blob $BLOB
six tests did not catch it	tools/test_claim_receipt_blob.py has 7 tests; the seventh is the one that kills the mutant
I re-ran the mutant while writing this	baseline 7 passed. Truncating the comparison to ${BLOB:0:4}: 1 failed, 6 passed, and the failure is test_a_blob_prefix_is_not_a_blob

What found it, and what did not

I wrote six tests for the fix. A receipt for the previous blob is refused. The
refusal names the blob. A legacy path-only receipt satisfies nothing. An honest
receipt passes. A stale line sitting first does not shadow a correct one below it. A
stale override is refused too, because an escape hatch that never expires is the
same hole.

All six passed. Then I mutated the guard: compare only the blob's first four hex
characters.

All six still passed. The hole was still open.

The seventh test is the one that kills that mutant, and its docstring says so:

Found by mutation, not by inspection: truncating the comparison to the blob's
first four hex characters survived every other test in this file.

Six is not a coverage number. It is how many tests I wrote before I felt finished,
and my sense of finished ran out exactly one test before the bug did.

I had already written the symptom down and filed it wrong

Two days before the fix, in a different commit, I described this:

passing needed two receipts for one file in a specific order, because the generic
loop reads the first matching line only. That ordering was load-bearing and
recorded nowhere.

I noticed that line order was deciding outcomes. I filed it as a documentation gap.

It was the bug. Inspection had the evidence in hand and did not convert it into a
failing test, because inspection asks what I believe the code does. Mutation does
not ask. It breaks the code and reports which assertion noticed.

What I might have wrong

The hook's own header states the residual and I am not going to soften it: git
commit --no-verify skips every client-side hook, so no local hook is a complete
policy boundary. This gate stops an accident. It does not stop a decision.

I know the window and not the damage. Four days is a fact about dates. I did not go
back and check whether any claim actually shipped through the hole while it was
open, so I cannot tell you the blast radius, only its duration.

The receipts file is gitignored, so my own migration state is not independently
checkable by a reader. Take that part on my word or leave it.

And the mutation was hand-applied by me to a shell script, which means the operator
set is whatever I happened to think of. That is the same limitation that produced
the six-test blind spot in the first place, one level up.

If you have run mutation testing against your guard code rather than your product
code, I would like to know what it killed that you were sure was fine.

---

I build deterministic evaluation and verification tooling for LLM systems, and I am looking for my first full-time role in AI evaluation or QA engineering. Remote US Eastern, or Charleston SC. https://erikhill.dev

The gate, its tests and the fix: https://github.com/egnaro9/egnaro9.github.io

Verified by re-running the mutant against the committed hook while writing this. The judgement of whether it was worth saying is mine.