Originally published on hexisteme notes.
I run a Stop hook called stop_decision_ownership_check.sh — a shell script that runs at the end of every agent turn in my coding sessions — whose job is to catch a specific failure: the agent doing analysis it is fully capable of doing itself, then handing the conclusion back to me as a question instead of just answering it. At the time of this audit, it fired on an AND of two conditions: the agent's last message matches a pattern for that kind of hand-back, and the ten transcript entries before it show the agent actually pulling data first — a file read, a grep, a shell command. The second condition exists because the pattern match alone can't tell a real question from a punt; the same sentence is a punt if it followed genuine digging, and legitimate if it didn't. It's one of seven hooks built the same way over my sessions, and until recently I had no idea how often any of them actually fired relative to how often they ran, because the code wrote nothing down except on the one outcome that blocked the turn.
A reader, @xm_dev_2026, left a comment that put the situation in sharp terms: the cheapest next step is to add a timestamp to the fingerprint file the hook already writes when it blocks. A week of that would date the fires; it would still not provide a trigger rate without the missing denominator. The real question was whether I was willing to act on a week of data from a gate that had already been running uninstrumented for months. As it turned out, the retained transcripts could answer the denominator question without that wait.
A trigger rate looks like a metrics problem — add a counter, wait, read it back later. But when a gate's decision logic is deterministic and the inputs it decides over are retained somewhere, you don't have to wait for new data. You can replay the exact logic over the old data and get a counterfactual rate today, at a sample size a week of live instrumentation would never match. The gap here wasn't a missing write. It was a missing read.
A hook that only wrote down one outcome out of five
The decision-ownership hook sits on every Stop event; as it stood then, it had five possible outcomes, and exactly one of them touched disk. If the agent's last message has no text, it exits quietly. If the ten prior entries show no data-gathering tool call, it exits quietly. If the hand-back pattern doesn't match, it exits quietly. If it matches but an identical response was already recorded, a deduplication check catches it and it exits quietly too. Only if none of that stops it does the hook block the turn — and only then does it append one line, a hash of the response text, to ~/.claude/decision-ownership.warned.
The dedup check runs before that append, which means a second or third instance of the exact same blocked text produces no new line. So the warned log's row count was never a count of how many times the hook fired. It was a count of how many distinct pieces of text it had ever fired on. And the one number that would make either count mean something — how many times the hook ran at all — appeared nowhere, in this hook or the six others built the same way: zero counters across all seven, five of them keeping nothing but a fingerprint set like this one.
A fix aimed at the half that was already visible
The timestamp suggestion deserves to be taken seriously on its own terms, because it isn't wrong. It's aimed at the side of the hook that already had a signal. A timestamp on the warned log tells you when a block happened. It says nothing about the four quiet exits standing in front of it, because the warned log only exists on the path that fires. The denominator, the pattern-match rate, how much filtering the tool-evidence condition is actually doing — every number that would answer "how often does this hook do something" lives on the entry path, and the entry path writes nothing, timestamped or otherwise. The comment carried its own diagnosis of why: the exit that fires gets instrumented, the four that don't, don't. That line predicts the limit of the fix it's attached to.
What was already on disk
Instead of changing the hook or waiting a week, I reproduced its exact exit logic — same AND-gate, same pattern, same ten-entry window, same dedup check — in a standalone, re-runnable script (hook_denominator_audit.py), and replayed it over transcripts that already existed: 5,537 of them, archived from past sessions across projects. Nothing about this needed the hook to have been instrumented differently while those sessions were running, because the hook's decision at any point is a pure function of the transcript around it. If you can recompute the function, you can recompute the decision after the fact, over however much history you kept.
Replayed against that corpus, the script found 116,022 points where the hook's Stop logic would have evaluated — a denominator that had existed nowhere. Every one of the 116,022 sorted into exactly one of the hook's five outcomes:
| Outcome | Count | Share |
|---|---|---|
| No last-assistant text | 15,871 | 13.68% |
| No data tool in prior 10 entries | 23,040 | 19.86% |
| Pattern didn't match | 76,999 | 66.37% |
| Matched, but deduped | 17 | 0.015% |
| Fired | 95 | 0.082% |
Ninety-five fires out of 116,022 points would be a rate of 0.082%, about one candidate Stop point in every 1,221 — a replayed figure, not an observed log, for reasons I come back to below. That number alone was what the timestamp suggestion was reaching for, and the replay reached it without a single new session having to run first.
The corpus gave more than the top-line rate, though, because it let me pull the AND-gate's two conditions apart instead of only seeing their combined output. Across the same 116,022 points, the hand-back pattern matched on its own 316 times — 0.272% of all Stop events, before the tool-evidence condition gets a vote. Of those 316, the tool-evidence condition — the one I'd always treated as a plausibility check riding along on top of the pattern match — suppressed 204 of them, 64.6%. Most of what decides whether this hook fires turned out not to be the pattern at all. It's the second condition, the one I added specifically to compensate for the regex being context-blind. Of the 112 that survived both conditions, the dedup check silently passed 17, 15.2%, before a single new line ever reached the log. That 15.2% used to be a sentence I could only phrase as "the warned log undercounts, by some amount I don't know." It's a number now.
The gap was a missing read, not a missing write
None of this needed a new write anywhere. The transcripts were already an append-only record of exactly the material the hook's logic runs over: the last message, the ten entries before it, the response text a dedup check would hash. The hook's rules are deterministic — given the same transcript, they sort a Stop point into the same one of five outcomes every time. A deterministic function over retained input doesn't have to be watched running forward. It can produce a counterfactual replay across retained history, provided that the retained record reproduces the gate's inputs and evaluation points closely enough.
That's the part of this that isn't specific to Stop hooks or coding agents. Any gate that reaches its decision through a fixed, reproducible procedure, and reads from inputs retained somewhere — logs, transcripts, request bodies, event streams — can be replayed instead of re-instrumented. A rate limiter's rule can be replayed over old request logs to see how often it would have tripped before anyone thought to count it. A fraud rule can be replayed over historical transactions. A CI gate's pass condition can be replayed over old build logs. The instinct in all of these cases is to add a counter and wait, because that's the right move when the logic isn't fully known yet, or the past wasn't fully kept. When both of those are actually true, waiting is the expensive option, not the safe one.
What the replay doesn't tell you
A recovered denominator is an easy thing to overtrust, so I want to be specific about where this one stops being solid.
The replay approximates a Stop point rather than reproducing it exactly — it treats any assistant entry with no tool call after it as a turn boundary, which is the right rule most of the time and not guaranteed to match what the live hook does under every interrupt or edge case. And a substantial share of the 5,537 transcripts in the corpus predate the hook's existence and span projects it was never watching in the first place. Both facts point the same way: what I have is not a log of 95 real blocks. It's a counterfactual — what the current logic would have decided, replayed over sessions it never actually ran against. The hook's real, live warned log held 22 lines at the time of this audit. Ninety-five and 22 aren't in tension. They answer different questions over different populations, and treating them as the same number would be the actual mistake here.
What no fire-path counter can ever settle
There's a second limit, and it has nothing to do with corpus scope — it would hold even against a perfect live log going back to day one. Whether 316 matches out of 116,022 points means "this behavior is genuinely rare" or "the pattern is too narrow to see most of it" isn't something any counter sitting on the fire path can ever answer. A false negative — a real hand-back the pattern fails to catch — never reaches the code that counts anything, by definition. It doesn't fire, it doesn't dedup, it doesn't land in any of the five buckets I can measure, because as far as the hook is concerned, nothing happened.
Rate is free now. Recall isn't. Knowing how often the gate fires relative to how often it runs is a replay away. Knowing how often it should have fired and didn't needs something a counter can never supply: a sample labeled by something other than the gate itself, which still means a person reading transcripts by hand and deciding, case by case, whether a hand-back was really there. I got the first number without spending that cost. I still owe the second one.
Before you add the counter
What this leaves me with is a question to ask before opening an editor to add a counter anywhere: is the decision this gate makes actually deterministic, and do I already have the inputs it decided over, retained somewhere? One caution on the second half, learned by extending this replay past a single gate: retained has to mean the whole history the condition read, not just its present value. A gate whose condition queries a live table — one that stores only what is true now — fails this test even though its logic is deterministic and the table is sitting right there, because the population it was eligible to fire on cannot be reconstructed from a snapshot. If both answers are yes, the fastest route to a trigger rate isn't a new write. It's writing the replay — the same logic, pointed backward — and reading what was already there. New instrumentation only earns its keep when one of those answers is no: the logic depends on something that isn't logged, or the history that would make a replay worth running was never kept. Neither was true here. The week the timestamp suggestion would have cost bought nothing the archive didn't already have.
Update (2026-08-08): Days after this audit, this hook did get live tracing — every exit path now writes one line to a trace log — and it also gained a third entry condition, an expiring exemption registry. That isn't the counter argued against above: the replay recovered the one number no new counter could backfill (the historical denominator), and the trace covers the forward direction from the point where the hook's logic itself changed — and once the logic changes, replaying the new version over old transcripts is no longer the same function.
More notes at hexisteme.github.io/notes.
Top comments (1)
Fair — my timestamp suggestion was aimed at the half that already had a signal, and the replay got the number without spending the week.
The figure that jumps out at me is 64.6%. The pattern match is doing almost none of the deciding; it's a candidate generator, and the tool-evidence condition is the actual gate. If this hook ever needs tuning for noise, that's where the leverage is, not the regex.
And on the recall you still owe: the replay quietly built you the sampling frame. You now have 116,022 indexed candidate points with outcomes attached, so drawing a random hundred for hand-labeling is nearly free — the reading is still the expensive part, but you'd come out with a recall bound instead of a shrug.