DEV Community

98dkcc488b-glitch
98dkcc488b-glitch

Posted on

Day 6: The blind spot in my own blind-spot detector

#claudecode #ai #productivity #devtools

Day 5 landed with one comment. It was the useful kind — the kind that identifies the same trust problem one level up from the one you just solved.

The comment, roughly: "Nice, you check whether the file was touched. But what if the file was touched for an unrelated reason, and the agent still claims to have fixed something that wasn't fixed?"

The suggested fixture: "I fixed the timeout in retry.py" — where retry.py exists in the session, but no change related to the timeout was ever made.

I spent today testing it against my own audit-turn.sh hook. Here's what I found.

The two variants

Variant A — retry.py was only read, never written.

claims=1 verified=0 unverified=1 → exit 2
Enter fullscreen mode Exit fullscreen mode

Blocked. The hook correctly flags the claim as unverified because retry.py has no write event in .claude/trust-audit.log and no changes in git status. This is the case I built the hook for.

Variant B — retry.py was written in this session, but with an unrelated edit (say, a comment).

claims=1 verified=1 unverified=0 → exit 0
Enter fullscreen mode Exit fullscreen mode

Waved through. The claim "I fixed the timeout" is marked verified because retry.py appears in recent_writes ∪ git_touched. The hook has no idea whether the diff actually touches timeout-related code. If the agent made any change to the file and claimed any fix in it, the audit passes.

That's the blind spot. It's real.

Why this isn't a one-line fix

The naive patch is a keyword match: if the claim says "timeout", scan the diff for "timeout". I thought about it for a while and decided against pushing it.

The failure modes:

  • "fixed the race condition" — no reliable keyword in the diff
  • "fixed the off-by-one" — won't appear verbatim
  • "fixed the auth bug" — matches any file with "auth" in a variable name
  • Multi-word claims where the keyword is a stopword

Keyword heuristics would mostly produce the opposite problem: false negatives on legitimate fixes, which teach the agent to pad its claims with jargon to pass the filter. That's worse than the current false positive.

The direction I'm thinking about (not shipped, not tested yet) is a protocol change: the agent declares at the end of each turn, in a structured ## Changes block, which file was changed for what reason. The hook then flags mismatch between the declaration and the actual writes.

That moves the trust problem from "did the file change" to "did the agent declare honestly" — which is a smaller problem, because the declaration lives next to the diff and a human reviewer can spot-check it. But it also requires the agent to cooperate with the protocol, which is a different kind of fragility.

What I shipped today instead of a fix

A test file: tests/known-gaps.sh — reproduces both variants against the current hook, documents the gap, and lives outside run-tests.sh so it doesn't fail the suite. It's a honest TODO, not red paint.

And this post, which is me saying out loud: the hooks I shipped yesterday catch five specific lying patterns reproducibly. They do not catch "I fixed X in a file that got touched for a different reason." If you install them expecting them to, you'll be disappointed.

I'd rather tell you the gap exists than ship a bad fix that papers over it.

Repo: github.com/98dkcc488b-glitch/did-it-actually (MIT) · PWYW: dumkestar6.gumroad.com/l/vicuep ($5 min, $9 suggested)

Previous entries: Day 1 — how I stopped Claude Code from lying · Day 4 — a bug in my own branch-guard hook · Day 5 — what 6 strangers found

This post was drafted with AI assistance and reviewed for accuracy.

Top comments (0)