Originally published on hexisteme notes.
I audited a 24-day session on a short-form video pipeline I run with a small fleet of coding agents. On paper the headline numbers were unremarkable for a project that size: 52,188 main-thread turns, roughly 1,065 turns per episode, and 1,873 calls to pytest sitting right there in the Bash tool-call log. Then the person actually using the pipeline flagged something that didn't match any of those numbers at all: the worker, they said, kept trying tests it didn't need to. That complaint didn't trace back to Bash. It traced back to a hook I'd wired in during an earlier session and half-forgotten.
Not another measurement-artifact story
It's worth being precise about what kind of failure this is, because it sits right next to a few others I've written up that look similar from a distance and aren't. A measurement proxy inserted to observe a system can quietly change what that system does, inflating the very number you added it to see. Pooling behavioral metrics across two different roles in a fleet — a long interactive session and a short one-shot worker — without separating them first can turn two nearly-identical within-role ratios into a misleading pooled headline. Two different models can bill different token counts for nearly identical input, because the token meter itself is scoped to whichever model is doing the counting. All three of those are stories about a number that comes back wrong, or a number that's right but not comparable to the number sitting next to it.
This one has no number to begin with, wrong or otherwise. The cost wasn't measured incorrectly — it was never inside the measurement's field of view. A tool-call audit counts what runs through the tools it's watching, and a hook doesn't run through Bash; it runs through a layer that audit was never pointed at. Worse, when the hook succeeds it produces no output at all, so there's nothing for even a hook-aware audit to add up except the failures. This isn't an instrument distorting a reading. It's a cost source sitting on a wire nobody had an instrument on.
The mechanism
The hook itself was simple, which is exactly why it had been running unexamined. Every time an agent edited src/<mod>.py, a PostToolUse hook ran the entire corresponding suite, tests/test_<mod>.py — a run costing anywhere from two to three and a half minutes — and only spoke up, waking the model with a failure message, if something in that suite came back red. A clean run produced nothing: no log line, no transcript entry, no evidence that anything had happened.
Why a silent, per-edit hook is worse than it sounds
Three things stacked on top of each other here, and any one alone would have been tolerable.
First, it's invisible to the audit method you'd normally reach for. An audit that counts tool calls counts Bash invocations, API calls, background processes it already knows to watch — not hooks, and especially not a hook whose success path is silent. All you can ever recover after the fact is a lower bound, built out of whichever failures were loud enough to leave a trace.
Second, and worse than the blind spot, the hook was scoring the wrong moment. A change that touches multiple files is, by construction, broken partway through: you edit the first file, and until you've also edited the second, the tests covering the first file are correctly red. That redness is exactly what wakes the hook into failure mode, and the pressure it creates is "fix this now." Faced with that pressure, the model reverts or routes around the file it just touched — the one already edited — rather than moving on to the second file it hasn't gotten to yet, the one that would have actually made the suite pass.
Third, it was a cost nobody had asked for. Neither I nor the model triggered these runs on purpose — the hook fired on its own, off an edit event — which means nobody was ever in a position to ask "why is this running," the question that would normally catch a wasteful process before it repeats hundreds of times.
The numbers
Counting up what the transcripts actually showed over the 24-day window: 971 failed hook runs, totaling 5.0 hours. Edits in the main session alone were enough to trigger the hook 533 times — before adding in whatever the sub-agent workers triggered on their own edits, which ran through separate transcripts not folded into that count. The true number of times the hook fired, successes and failures combined, isn't something I can reconstruct after the fact; silence doesn't leave a receipt.
The per-file cost behind those numbers: test_cli took 216 seconds, test_map_scenes took 160, test_compliance_gate took 134 — every one of those running in full, after every single edit to the file it covered.
After the fix, a smoke test told the other side of the story: a queue of 3 pending files drained down to pytest running against 2 files, in 4 seconds.
The fix: move the trigger from the editor to the author
The fix keeps the same suite and the same enforcement intent — verification still has to happen — and only changes when it fires.
The PostToolUse hook now does one thing: it appends the path of the file that was just edited to a queue file. That's a write of a few milliseconds, not a multi-minute test run.
A Stop/SubagentStop hook — the one that fires when a turn actually ends — reads that queue, de-duplicates it, and runs pytest exactly once, covering every file touched during the turn. It returns a summary to the model only if something in that single combined run fails. A turn with no edits costs nothing, because the queue is empty and there's nothing to drain. A lock keeps two drains from overlapping, and an empty queue is its own stopping condition, so there's no path to the drain looping on itself.
One part of this was easy to get backwards: hook configuration is snapshotted at session start. Change the hook definition mid-session, and the session you're in keeps running on the old snapshot — the new behavior only takes effect starting the next session. That's worth flagging on its own, because it's exactly the kind of fix that looks broken if you test it in the same session where you wrote it.
What generalizes
If you've established a rule like "verification happens once per unit of work," the first thing to go looking for isn't more verification — it's the automation that's already quietly breaking that rule. A written rule changes what a person or a model does next. It does nothing to a hook, because a hook doesn't read rules; it reads trigger events, and it keeps firing on the event it was configured for regardless of what the team has since agreed the right cadence should be.
The practical corollary is about how you audit time and cost in an agent fleet at all: tool calls are not the whole cost surface. Hook output and background processes are part of it too, and a hook whose success path is silent means any total computed from logs is a floor, not a figure — bounded below by whichever failures happened to be loud enough to record themselves, and unbounded above by however many quiet successes never got the chance.
And the timing question generalizes past testing specifically. The right moment to verify a change is almost never "immediately after this one edit." It's "after this one unit of change is finished" — and knowing where that boundary sits isn't something an editor or a file-save event can know. Only whoever is actually authoring the change, and deciding when a turn is done, is in a position to say so.
Where this would turn out to be wrong
The falsifier here is specific enough to check for directly: in the next batch of work, if the end-of-turn drain still comes back red at roughly the same rate the old per-edit hook did — because of the same mid-edit, still-incomplete state — then the fix didn't actually solve the problem it looks like it solved. That result would mean the defect was never about when the check runs. It would mean the test files themselves are too coarse a unit, bundling too much unrelated behavior into one suite for any single trigger point to time correctly. At that point the next move isn't to move the hook again — it's to split the suite.
Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.
More notes at hexisteme.github.io/notes.
Top comments (0)