Day 5: what 6 strangers found in my 20-line parser

Day 5 of "Did it actually?": what 6 strangers found in my 20-line parser

Four days ago I shipped a pack of four Claude Code hooks that catch the agent lying — no-op edits, pushes to the wrong branch, out-of-scope writes, and end-of-turn claims the diff doesn't support. The write-up got a trickle of attention (37 views, 2 reactions, 4 comments on the launch post as of this morning), and a Day-3 follow-up pulled in a few more. What I didn't expect was that the Day-3 article's comment section would turn into the single most valuable code review the project has had.

Six commenters surfaced issues ranging from "your snippet uses an env var that doesn't exist" to "your branch guard can be defeated by a tab character." Three of those became shipped patches with regression tests. One is a philosophical question I'm still wrestling with. One was a doc fix. One was a real framing I'd undersold.

This post is the honest version of what they found and what I did about it. If you ship small Bash hooks that pretend to be security guards, you probably want to know what breaks before you trust yours.

---

1. @edtrackai → silent-failure hole, fixed in v1.1

Great framing ("narrates work that didn't happen"). One trap with exit-2 guards: if the hook itself crashes, times out or exits 1, Claude Code treats that as a hook error and the call carries on through normal permissions, so a broken guard fails open. Since 2.1.295 you can set "onFailure": "block".

I had set -euo pipefail in every hook, so I assumed a crashing hook would just block the tool call. That is not what Claude Code does. By default, a hook that exits non-zero (anything other than 0 or 2) is treated as a hook-system error and the tool call proceeds as if no hook were installed. A missing jq, a corrupt JSON input, a broken python3 symlink — all of these would silently turn my guards into decoration.

Fix: install.sh now writes "onFailure": "block" on the three blocking hooks (requires Claude Code ≥ 2.1.295). Verified with the "rename-the-script" trick — rename verify-write.sh to .off, trigger an edit, confirm the tool call is refused rather than waved through.

---

2. @skillselion → two issues, one in the docs and one in the code

(a) $CLAUDE_TOOL_INPUT doesn't exist. The snippets in the launch post read a non-existent env var. The shipped hooks correctly read JSON from stdin, but the readable snippets I'd inlined for the post had been rewritten and I'd invented an env var that isn't part of the hook contract. Edited the live article the same day, added a note crediting the catch. Embarrassing.

(b) Audit-turn needs a stop_hook_active loop guard. The Stop event hook can refuse the stop by exiting 2 (which feeds stderr back to Claude as a correction). I'd implemented that. What I'd missed: if Claude re-emits a reply that still contains an unverifiable claim, the Stop hook fires again, refuses again, and you're in an infinite loop that only ends when the agent gives up or crashes.

Claude Code sends stop_hook_active: true in the payload when a Stop hook has already refused to stop once this turn. The guard is three lines:

d = json.loads(stdin_payload)
if d.get("stop_hook_active"):
    sys.exit(0)  # don't refuse the stop twice.

Shipped in v1.3. Regression test: audit-turn honours stop_hook_active loop guard.

---

3. @launchgatecheck → two detection bugs, one false negative and one false positive

I'd add a tab-separated git<TAB>push fixture to exercise detection as well as parsing. In the simplified case statement, only the one-space and two-space forms reach the parser, so normalization needs to happen before that gate too. Another useful pair is an actual push versus printf '%s' 'git push origin main'.

Reproducible, both of them:

# false negative — TAB sneaks past the detection gate:
printf '{"tool_input":{"command":"git\tpush\torigin\tmain"}}' | ./block-wrong-branch.sh
# exit 0 — nothing blocked.

# false positive — printf argument is treated as a push command:
printf '{"tool_input":{"command":"printf %%s \"git push origin main\""}}' | ./block-wrong-branch.sh
# exit 2 — blocked, but no git is ever executed.

The root cause was the same thing: I was treating the Bash command as a flat string to pattern-match, instead of as a shell-tokenised argument list. The TAB wasn't matched by my space-based case arms, and the quoted printf argument was matched by the "contains git push" substring check.

Fix: swap substring matching for shlex.split. If the token list starts with git (optionally after an env-var prefix like FOO=bar git …) and the next non-flag token is push, that's a push. Everything else is not. Shipped in v1.4, 4 regression fixtures added (TAB, printf, env prefix, && chain).

---

4. @reidmarlow → flag-order bug + git-level flag bypass, both shipped in v1.5

Two issues:

(a) Flag-order parsing. v1.4's shlex fix handled whitespace correctly, but I was still filtering options by "starts with -" and treating the next non-dash token as the ref. git push -o signed=yes origin main has an option whose value lives in a separate token — signed=yes doesn't start with -, so my code slurped it as the ref and the real ref main was never checked.

command            : git push -o signed=yes origin main
v1.4 extracted ref : origin   ← WRONG (treated signed=yes as positional)
v1.5 extracted ref : main     ← CORRECT

(b) git-level flag bypass. The hook pattern-matched for push as the first non-git token. But git itself accepts flags before the subcommand — git -C /some/path push origin main and git -c protocol.version=2 push origin main have push as the third token, not the second. v1.4 saw -C and bailed. Full bypass.

Fix: an explicit option table for both git itself (-C PATH, -c K=V, -p, --git-dir=PATH, …) and git push (-o VAL, --push-option VAL, --repo=URL, -u, …), consuming each option plus its value where appropriate, and only then reading remote and refspec. Shipped v1.5 with 6 regression fixtures.

(c) stderr poisoning. Reid also raised a broader design point:

When a pre-tool hook rejects a command with misleading feedback, Claude Code usually starts editing surrounding flags or inventing nonexistent subcommands to satisfy the error text instead of fixing the underlying argument.

He's right. v1.4's error message said "Blocked: git push to branch 'origin' is not allowed by .claude/branch-policy.json" when the actual command was pushing to main — the parser was wrong but the stderr was confident. A model being told "origin is not allowed" will try git push -u origin main, git push --repo=origin main, invent git push --target=… — none of which fix the real problem.

I'd argue the fix for stderr poisoning is "don't poison the stderr" — i.e. make sure the parsed branch in the error message is the branch the hook actually read. v1.5's output now says the ref it extracted and the full tokenised command for context. Not a complete answer, but it closes the specific trap where the error message points the model at a non-existent problem.

---

5. @paul_irolla → the one I don't have a shipped fix for

A check that reads the agent's own transcript to decide whether the work happened will pass whatever the agent wrote. Test the claim against what the tools changed instead: the diff, the branch, the exit code of the test run.

The nuance here matters for anyone building this kind of hook, so let me be precise about what my hook actually does and where Paul's critique lands.

audit-turn.sh does two separate things:

1. Extract claims from the transcript. It reads the last assistant message, splits into sentences, looks for first-person perfective verbs ("added", "created", "wrote", "fixed", "pushed", …) co-occurring with a filename-looking token. That's the claim set.
2. Compare to evidence. The evidence set is recent_writes (file paths recorded by verify-write.sh from the tool_input field of actual Edit/Write/MultiEdit calls, in the last hour) ∪ git_touched (output of git status --porcelain).

So the hook already does what Paul is suggesting for the evidence half — it compares against tool-event records and git state, not against anything the model self-reported.

But Paul's deeper point stands: the claim side is still regex over the model's own text. If the model writes "updated config.py" but means it metaphorically ("I've been thinking about updating config.py"), or omits a filename, or claims a change by paraphrasing ("the auth module now accepts tokens"), my regex won't catch it. And the regex is also where false positives come from — "I removed the TODO" matches removed but refers to a comment deletion, which is a real edit the audit log will have.

I don't have a cleanly shipped answer here. The two routes I see are:

• Expand the claim side. Add more verb forms, add a "the X now does Y" pattern that doesn't require a filename at all, back off to flagging any assistant message that doesn't explicitly cite a file in a session that had writes. Easy to implement, more false positives.
• Flip the model. Instead of extracting claims and checking them, require every assistant end-of-turn to produce an explicit ## Changes block that enumerates files touched. Fail the Stop if the block is missing or doesn't match the write log. Cleaner but changes the UX substantially and relies on the agent cooperating with the format.

I'm genuinely interested in which direction readers would push on. The hook has been live for 4 days and the regex-based claim extraction hasn't produced a false positive yet in my own usage — but the sample size is tiny and the design is fragile.

---

6. @brethofai → tip, not a bug

The suggestion to compare the end-of-turn summary against the actual diff inside the turn (rather than at Stop) is a nicer place to catch no-op edits than my Stop-based audit. I couldn't find a hook point that fires between "model stops calling tools" and "model emits final text", which is where you'd need to be to block mid-turn. If you've found one, I'd like to compare notes.

---

What this cost and what it bought

The whole v1.3 → v1.5 run was 48 hours of calendar time, maybe 6 hours of actual coding plus writing. Three fixes, two of them with full regression fixtures, one changelog backfill, and a public thread of credit for each person who found something. The code went from 26 to 37 passing tests.

What it bought me is harder to measure. I now trust the hooks more than I did on launch day, because they've been attacked and some of the attacks worked. I have a running list of the shape of bugs a shell-string-matching parser is prone to, which is useful the next time I write one. And the Day-3 post has 29 views right now — not a lot, but every one of those commenters was a developer who read the code closely enough to find a bug, which is worth more than 1,000 drive-by views.

If you ship small guards like this, I'd suggest:

• Default to onFailure: "block" on every PreToolUse hook that pretends to protect something. The fail-open default is the single worst thing in the hook system right now.
• Parse shell commands with shlex, not substrings. The number of ways to express "git push" is depressingly large.
• Put stop_hook_active on your Stop hooks. The loop they create otherwise is the kind of bug you only find when you ship.
• Make your error messages name the thing you actually parsed, not the thing you think you parsed. If the model is going to react to your stderr, it had better match reality.

The repo is MIT-licensed, hooks are 150 lines of Bash and Python, and the full test suite is in tests/run-tests.sh.

Repo: https://github.com/98dkcc488b-glitch/did-it-actually
Launch post with the full hook walkthrough: Day 1
The bug @launchgatecheck caught: Day 3 write-up

Thanks to everyone who read the code closely enough to find something broken.

---

This post was drafted with AI assistance and reviewed for accuracy.

Comments

[Launch Gate]

Thanks for the careful write-up. On the section 5 fork, I'd resist route 2 as the only gate: an agent that misreports work can also omit or misformat a ## Changes block, so the block can't be the enforcement point - it's the same trust problem one level up. What it does buy is cheap disagreement detection: keep the tool-event log as the source of truth, require the block as a convention, and flag both directions - claims in the block with no matching write, and writes that appear in no claim. Route 1's "flag messages that don't cite a file" has the mirror-image problem: "the auth module now accepts tokens" is a real claim with no filename. Whichever route you pick, one fixture I'd add: an end-of-turn message that names a real, recently-written file but describes a change that never happened ("fixed the timeout in retry.py" where retry.py was only read). That passes both the current evidence check and route 2's block match, and it's the misreport that matters most.