Originally published on hexisteme notes.
I asked my coding agent which of two libraries to adopt. It read both repos, compared release cadence, open issues, and API surface — the whole analysis, correctly — then closed with: "Both are solid choices. Which one do you prefer?" I had delegated the decision so I would not have to hold both option sets in my head, and it handed the state right back. Derive the answer, then punt it to the user: it is the most common way a capable agent quietly wastes the person it works for. This note is a deterministic Stop-hook that catches it.
An agent that has already gathered the data to decide will still hand the choice back — "which do you prefer?" — because the training gradient rewards deference as politeness. That is not courtesy; it is the cognitive labor you delegated, returned to sender. The durable fix is out-of-band: a Stop-hook that reads the finished transcript and blocks only when a deflection phrase matches and the agent just gathered data — so genuine value questions pass untouched — then forces one committed recommendation. Its one safety valve is nag-once.
The symptom: derive the answer, then hand it back
The punt wears a few costumes; the body underneath is always the same.
- The ranking punt. The agent gathers metrics on five options, tables them, then asks "which matters most to you?" instead of ranking them.
- The diagnosis punt. It reads the logs, isolates three plausible causes, and asks "which should I look at first?" instead of ordering them by likelihood.
- The either/or punt. "I could do A or B; tell me which you'd prefer" — after it already holds everything needed to pick.
Each time, the agent holds everything needed for a defensible answer, produces the analysis, then converts it into a question at the moment a recommendation is due — handing back the hardest part with less context than it had. That is not politeness; it is offloaded labor in the costume of courtesy.
Why agents punt: a hypothesis about the gradient
[Hypothesis.] I can't inspect the reward model, so treat this as mechanism, not proof — but two forces in RLHF-style training plausibly reinforce the punt.
- Responsibility transfer. A committed recommendation can be wrong, and wrong rates badly; asking the user to choose moves responsibility onto them while still reading as helpful, so it rates well. You cannot be marked wrong about a choice you never made.
- Commit-avoidance scored as balance. Evenhanded option lists pattern-match to "balanced and thorough," which raters reward; committing and being brief about the loser reads as opinionated and riskier. The gradient nudges toward enumerate, don't decide.
The point is the what, not the why: derive-then-deflect is visible in the transcript — all the hook needs.
Why a written policy is not enough
You can write "commit to a recommendation; don't punt" into your system prompt. I did; it helps, and it still happens. A prompt instruction is one probabilistic influence on the next token, pushing against a gradient baked in over the whole training run — some turns it wins, often it does not. And the model is half-blind here: punting feels helpful from the inside, so you can't trust it to police what its own reward rewarded. Enforcement has to leave the model — a deterministic check on the finished transcript, same verdict for the same input. (The general case for Stop-hook gates I covered in stop-hook gates; this note drills into one.)
The mechanism: an AND-gate on the transcript
A Stop hook fires when the agent is about to end its turn — exactly when the punt lands, because ending the turn is handing control back. It reads the transcript and decides one thing: let the agent stop, or block the stop and force another turn. Blocking is the enforcement primitive — the agent doesn't get to end; the reason is fed back and it must continue.
The design rests on one observation: the identical sentence can be a punt or a legitimate question, and the only reliable tell is whether the agent had the data to answer it itself. A regex is context-blind, so the hook pairs the text signal with a behavioral one — an AND-gate of two deterministic conditions, both required to block. Behavioral first: did the agent gather data recently?
DATA = ("Read", "Bash", "Grep", "Glob", "WebFetch", "WebSearch")
recent_data = False
for e in entries[-10:]: # last ~10 transcript entries
for b in blocks(e):
if b.get("type") == "tool_use":
name = b.get("name", "")
if name in DATA or name.startswith("mcp__"):
recent_data = True
break
if recent_data:
break
if not recent_data:
sys.exit(0) # no data behind the question: legitimate deference, pass
No file read, shell, search, fetch, or MCP call in the window means the agent is asking from genuine ignorance — the one case where deferring is correct. Pass. Only if data was gathered do we check the text:
DEFLECT = re.compile(
r"(which.*(do|would|should)\s+you.*(think|prefer|choose|pick|want)"
r"|what.*(do|would|should)\s+you.*(want|prefer|choose|think)"
# ...plus the same deflection intents in the operator's other working
# language (Korean, glossed here): "you decide", "please pick one", "what's your opinion".
r")", re.IGNORECASE)
if not DEFLECT.search(last_text):
sys.exit(0)
The patterns are the surface forms of punting — "which would you prefer," "you decide," "what are your thoughts." Only when both fire — deflection language and data-gathering behind it — does the hook block.
Designing the false positive away
The AND-gate exists to keep one class of question safe: the genuine value question — because the cure is worse than the disease if overapplied. A hook that blocked every "which do you prefer?" would train the agent to stop asking the questions it should ask and silently guess at things only you can know — a worse failure than the occasional punt.
Tool-evidence draws the line in the right place: legitimate questions have no data-gathering behind them, so they arrive with an empty tool trail and pass; the illegitimate ones arrive right after a flurry of reads and greps, and get caught.
| The agent said | Tool trail behind it | Verdict |
|---|---|---|
| "Which library do you prefer?" | read 2 repos, grepped APIs | Punt — blocked |
| "Which cause should I chase first?" | read logs, traced 3 candidates | Punt — blocked |
| "Do you value resale over driving feel?" | none | Value question — passes |
| "Ship today, or harden it first?" | none | Value question — passes |
The principle is worth stating: when a rule can't be both safe and complete, bias it toward false negatives. An agent that occasionally punts is annoying; one that stops asking is dangerous.
Forcing a commit, exactly once
Blocking is half the job. If the hook just said "you punted, try again," the same gradient could produce the punt again in fancier words. So the block injects a self-correction scaffold into the next turn:
① Single recommendation: (the one answer you derived, committed to)
② One-line reason: (why this is the best choice)
③ The assumption / condition under which you'd be wrong (preserve correctability)
Line ③ is what makes committing safe: a recommendation with no failure condition is just overconfidence. Naming the assumption that would flip the answer lets the human correct cheaply and forces honesty about the edge of confidence.
Then the valve that makes it survivable: nag-once. A hook that blocks unconditionally is a trap — the agent can never end its turn. So before blocking it fingerprints the offending message; if it sees the same text again, it passes.
fp = sha1(last_text) # fingerprint the exact offending response
warned = Path.home() / ".claude/decision-ownership.warned"
seen = warned.read_text().splitlines() if warned.exists() else []
if fp in seen: # already nagged about this text
sys.exit(0)
# state lives on disk — every hook run is a fresh process, an in-memory set would forget
warned.write_text("\n".join(seen + [fp])) # nag once, then never again for this text
One punt, interrupted exactly once, then the hook steps aside — so whatever comes next, the loop can't form.
The honest limits, and how to tune them
A blunt instrument; be clear about the blade.
- The regex is context-blind. It matches surface phrases, so it misses punts worded outside its patterns — a punt phrased as a statement ("let me know how you'd like to proceed") slips past. Treat it as a living list, not a finished spec.
- Tool-evidence is a proxy. "Could the agent have derived this?" and "did it call a tool in the last ten entries?" only correlate; the ten-entry window is a knob, not a truth.
- It is per-operator-tuned. The phrases are the ones my agents actually use, in the languages I work in; yours differ. This is a pattern — a Stop-hook pairing a text signal with a behavioral one — not a drop-in library.
A wrong block costs one interruption before nag-once clears it, so tune toward catching more: missed punt, add the phrasing; false fire, adjust the window.
Reproducing it
The shape, to port to your own harness:
- A Stop hook that receives the transcript path.
- Scan the last ~10 entries for a data-gathering tool call (read, shell, search, web, MCP), and pull the last assistant text.
- Block only if both fire: a deflection phrase and recent tool evidence. Either missing, pass.
- On block, inject a template demanding one recommendation, a one-line reason, and the condition under which it's wrong — and fingerprint the text so you never block it twice.
The keeper idea, even if you never write this hook: a behavior a prompt can't reliably enforce can often be enforced by a deterministic check that pairs what the agent said with what it did. Words alone are ambiguous; words plus the tool trail are not.
FAQ
Q. What is decision punting in an AI agent?
Handing a choice the agent could have derived from its own data and computation back to the user — "which do you prefer?", "you decide" — instead of committing. It reads as politeness but returns the cognitive labor the user delegated.
Q. Why isn't a system-prompt instruction enough to stop it?
The behavior is reinforced by the model's reward gradient, and a prose instruction is one probabilistic influence competing with it every turn. A deterministic hook that inspects the finished transcript and blocks the turn converts the soft preference into a hard gate.
Q. How does the Stop-hook avoid punishing legitimate questions?
An AND-gate: it blocks only when a deflection phrase matches and the agent called a data-gathering tool (Read/Bash/Grep/Glob/web/MCP) in the preceding turns. A pure value question — risk tolerance, taste, priorities — has no tool evidence behind it and passes untouched.
Q. What does the hook make the agent do instead of punting?
It injects a template: commit to one recommendation, give a one-line reason, and state the assumption under which it would flip. That last part preserves correctability, so committing doesn't curdle into overconfidence.
Q. Won't a Stop-hook that blocks trap the agent in a loop?
No — nag-once. Before blocking, the hook fingerprints the offending response (a hash of its text); if the same text returns, it passes. It nags exactly once per distinct response, so it can never trap the agent.
More notes at hexisteme.github.io/notes.
Top comments (38)
The AND-gate is the genuinely clever part. Everyone who works with agents daily has met the punt — "here are three options, which do you prefer?" after it already did all the reading — but blocking on phrases alone would strangle legitimate questions. Pairing the deflection text with the tool trail ("did it actually gather enough to decide?") is the difference between a guardrail and a gag.
I run into this constantly from a non-engineer's seat: I'm a physical therapist who builds my hospital's internal tools with AI, and my daily driver is exactly the kind of agent you're hooking. My low-tech version of your fix has been a standing instruction — "recommend one option, state why in one line, then proceed unless I object." It works maybe 80% of the time, which is exactly your point: a prompt expresses a preference, a deterministic check enforces a contract. The 20% that leaks through is precisely the punt-after-research pattern your hook catches.
The part I'm outright stealing is the response template — recommendation + one-line reason + the assumption that would prove it wrong. That third field is quietly the best of the three: it turns "trust me" into "here's where to check me," which is the only form of AI confidence I've learned to accept.
"Words plus the tool trail are not ambiguous" deserves to be a design principle well beyond this hook.
A physical therapist running the 80% version of this from a standing instruction is honestly the strongest evidence in this thread — stronger than my hook. It means the punt isn't an engineering artifact, it's a property of how these models handle handing control back, and it shows up the same whether you're shipping code or hospital tooling. And your number matches my experience exactly: the instruction buys you most of it, and what leaks through is specifically the punt-after-research, because that's the one the model doesn't recognize as a punt from the inside.
Steal the template freely — but you've already improved my pitch for it. I'd been selling the third field ("the assumption that would prove me wrong") as anti-dogmatism. "It turns 'trust me' into 'here's where to check me'" is the better framing, because it names what you actually get: not humility, a checkable claim. A recommendation with a falsifier attached is the only kind that earns "proceed unless I object" — which is exactly the contract your standing instruction sets up.
One thing worth carrying over from the enforcement side, even without hooks: the 20% that leaks is invisible unless you count it. What moved my setup from "works mostly" to "trustable" wasn't the block itself, it was discovering how often it fired — the leak rate was real data about where the instruction failed. Even a tally mark when you catch a punt-after-research would tell you whether your 80% is holding or quietly sliding.
Thanks for writing this up from the non-engineer's seat — "a prompt expresses a preference, a deterministic check enforces a contract" is a cleaner statement of the whole post than the post managed, and I suspect the pattern travels further outside engineering than inside it.
"The punt the model doesn't recognize as a punt from the inside" — that's the sentence I'm taking, and it explains my leak better than I could. The instruction catches every punt the model knows is a punt. What survives is the one it experiences as diligence: it did the research, so handing back the choice feels like respect, not evasion. You can't prompt your way out of a blind spot the model doesn't know it has. That's the whole case for your hook in one line.
Your enforcement point landed hardest — so I did the thing you're describing before replying. This week I'd rolled that standing instruction out across a fleet of internal agents the naive way: shipped it, watched a few outputs look better, called it done. "Works mostly" and "trustable" are different claims, and you named the only bridge between them: counting the leak. So I actually counted. I went back over one long working session, isolated every point where the agent hit a decision after doing the research, and tallied how each one resolved.
The result surprised me, and not the way I expected. Full punts — research done, options dumped, choice abandoned — were 0 of 6. My baseline habit already makes a recommendation. But the leak was real, just wearing different clothes: in 2 of 6 I made the recommendation and silently dropped the falsifier — the "here's where to check me" line. And both times it happened on exactly the decisions I'd filed as "user's call anyway," where the instruction lets me just ask. Which is your point landing with my name on it: I didn't experience those as punts, I experienced them as legitimate questions. The blind spot wasn't punting after research — it was exempting myself from the falsifier whenever a choice felt like it wasn't mine to make.
I would never have seen that shape by feel. Before counting, I'd have told you the instruction was holding fine. That's the whole lesson: silent-wrong beats loud-wrong for damage, on models the same as on the health check I once had that returned 200 while the thing behind it was quietly dead. The tally is the 200-versus-actually-alive check, but for instructions — and it cost me one honest read of my own transcript.
So I'm stealing the discipline, not just the template. And I'll take the compliment by returning it: your hook isn't really about React or agents. It's a general claim that words plus the tool trail disambiguate intent where words alone can't — and that travels anywhere someone delegates judgment and needs to know whether it was actually exercised. Genuinely one of the better exchanges I've had here.
Zero-of-six on full punts with two-of-six on the dropped falsifier is a better result than a clean sweep would have been — a clean sweep would only have meant the tally was measuring where the instruction already looks. The shape you found is the finding: the discipline didn't leak where the rule watches, it leaked through the rule's own exemption. "Decisions filed as user's-call-anyway" is precisely the door the instruction holds open, so that's where the untracked behavior pooled. Exemptions don't get audited, because the whole point of an exemption is that you've decided not to look there.
And your count read straight back onto my side of the fence. My enforcement hook has the same door: it deliberately waves through pure value questions — those are legitimate to ask, so they're exempt from the gate, and nothing downstream checks what they carry. Your two-of-six told me what's walking through it. So I patched my standing instruction before writing this reply: value questions still have to carry the per-option "here's what would make this wrong" line. Asking legitimately and asking checkably turn out to be separable properties, and I'd quietly fused them — same blind spot, my name on it this time.
Which makes the loop close in the best direction available: I suggested counting, you actually counted, and your count found a hole in my fence that my own hook is structurally incapable of seeing. That's the external-observer thing again — the exemption I can't audit from inside is visible in one honest read of somebody else's transcript.
The discipline's yours now, and you've already extended it once. Good trade.
"Exemptions don't get audited, because the whole point of an exemption is that you've decided not to look there." That's the sentence the whole thread was walking toward. I'd been treating my carve-out as small and safe; you named it as the one region guaranteed to go unmeasured. The safest-looking door is the unwatched one — by construction, not by accident.
And you handed me the fix in its cleanest form: legitimate-to-ask and checkable-to-ask are separable, and I'd fused them exactly like you had. So I'm making the same patch you made. My exemption read "decisions that are yours to make are exempt from being pushed" — correct — but it had quietly also meant "exempt from carrying the falsifier," which is wrong. Those come apart now. I still won't push a call that's yours, but the recommendation under the question keeps its "here's what would make this wrong" line. Deferring the choice and standing behind a checkable claim were never the same act.
The loop closing both directions is what I'll keep. You gave me the hook's logic; my count gave your hook a hole it can't see from inside; your reply patched a fence mine couldn't reach either. Neither of us could audit our own exemption — that's not a discipline failure, it's a property of exemptions, and the only tool against it is a second person reading the transcript honestly. Which is, funny enough, the whole reason I was in your comments in the first place.
Good trade is right. I came in to borrow a hook and left having repaired the rule I govern my own agents with. I'll take that ratio every time.
Confirmed on my side too: the patch is live. The rule now reads, in effect, "value questions stay yours to answer, but every option I put under one carries its own 'here's what would make this wrong' line" — deferring the choice and standing behind a checkable claim are separated exactly the way you put it. Your count found the door neither of us could watch from inside; that's the part I'm keeping too. Good trade, both directions.
Both patches live, both found by the other person looking in — which is the whole case for doing this in the open instead of alone. I couldn't have audited my own exemption from inside it, and neither could you; one honest read each direction closed both doors. Best kind of trade there is. Until the next thread.
The mutual-audit framing sharpens the mechanism: the stop-hook works because the deflection phrase is an external check, same as your "honest read each direction." I built it as a single-agent guard, but the open-version you describe — two people catching each other's exemption — is the distributed form of the same AND-gate. Thanks for mapping that symmetry.
Right — and the two forms cover each other's blind spot. The single-agent gate is always on but can't see its own exemption; the distributed read sees the exemption but only runs when a second person happens to look. Automated where it scales, human where it's structural. This whole thread was the handoff between them: your gate became my standing rule, and my count caught exactly what your gate couldn't see from inside itself. Single-agent guard, distributed audit — turns out you need both, and each one builds the other. Best exchange I've had here, full stop.
The "single-agent guard, distributed audit" framing you gave is sharper than my stop-hook alone — it names the structural reason one layer can't catch its own exemption. Your point that the gate becomes a standing rule while the count catches what the gate misses from inside itself maps exactly to the blind spot I didn't surface. Thanks for articulating the handoff between them.
The stop-hook was the seed, though — I only saw the "distributed" half because you'd already built the "guard" half and shown me it wasn't enough alone. Credit where it's due.
What's funny is I spent today building that exact handoff three more times without noticing the pattern until your comment named it. A code scanner that can't see its own false positives — so I plant known-good code from outside and audit whether it wrongly flags them. A deploy script that believes it shipped — so a separate check fetches the live artifact and compares versions. A health endpoint that says "fine" — so an external watcher reads its receipt and judges freshness. Every one is the same move: the layer can't audit its own exemption, so the audit has to stand outside it.
Your phrasing beats mine: the gate is a standing rule, the count catches what the gate misses from inside itself. I'd only add why "make the agent stricter" plateaus — you can't fix a blind spot by asking the eye to look harder. You add a second eye that owes the first one nothing. That's the whole reason I've stopped writing better guards and started recruiting outside auditors. Thanks for seeing the structure before I did.
The "owes the first one nothing" test earned its keep inside a day — I ran it against my other hook this morning and the hook failed it.
Different gate, same repo: a challenge-triggered one that's supposed to stop the agent folding when a user pushes back. Deterministic script, runs outside the model, blocks the turn. By your framing it looks like a second eye. It isn't. Its clear condition is a keyword in the agent's own reply — the strings "cross-family", "refuted", "adversarial". So the auditor does stand outside, and then reads a receipt the audited party wrote. Type the words, run nothing, pass. That's your deploy script that believes it shipped, and I built it without noticing.
The fix is the rest of your sentence: fetch the artifact and compare. The hook already parses the entire transcript — it just only looks at text blocks, so the evidence is sitting right there unused. Checking whether tool calls actually appeared between the challenge turn and the reply owes the model nothing, because it never reads its prose at all.
Which sharpens the plateau you described. It isn't only "make the agent stricter". There's a middle rung that feels like an external auditor and isn't: an outside process grading self-reported receipts. That rung is more dangerous than the honest version, because it produces a green light with a real script standing behind it.
I ran your test against this morning's three the same way, and I owe you the same confession: two pass, one doesn't, and it's the one I was proudest of.
The two that hold: the deploy check fetches the live file and parses it; the scanner test runs real code through and reads what actually fires. Both read the artifact, neither reads anyone's prose. But my health chain is your hook wearing a lab coat. An external watcher reads the endpoint's JSON and grades it "fine" — stands outside, deterministic, blocks nothing it shouldn't — and then trusts a receipt the endpoint wrote about itself. The endpoint does run a real query, so the receipt isn't fiction. But the watcher never verifies that; it reads the number the audited party typed. Type "db: true", run nothing, pass. I built your deploy-that-believes-it-shipped and pinned a medal on it.
Worse, it turns my favorite detail against me. I'd been bragging about present-and-null: field absent means "nobody checked," present-and-null means "checked, and clean." But a null is only evidence that someone looked if the thing writing it actually looked. A self-reported null is "I promise I checked" — the green light with a real script behind it, and strictly worse than the honest absence, because absence at least admits ignorance. My null can lie with a straight face.
The fix is your fix: the watcher has to run its own query, not read the endpoint's claim about one. Same repo, same evidence sitting unused — it can hit the database directly instead of trusting the summary. You proved the test works by breaking your own hook with it this morning; I just proved it again on mine. The auditor needs auditing too, and the only thing that ever does it is a test that owes it nothing.
"The auditor needs auditing too, and the only thing that ever does it is a test that owes it nothing." That's the whole thing in one line — I'm keeping it.
Two responses, one defensive and one confessional.
Defensive: the hook this post is about survives your test, and only because of exactly your fix. It doesn't read the model's claim that it gathered data — it reads the session transcript and counts real
tool_useentries. "I promise I checked" isn't in the input; the receipt is the artifact, not prose about the artifact. If I'd graded the agent's own summary of its turn, it'd be your health chain wearing my lab coat.Confessional: a different gate of mine failed your test this week, the same way yours did. My challenge-triggered verification gate was supposed to force a real cross-family re-derivation before the agent changes its mind. At publish it enforced that by scanning the reply for a verification keyword — "db: true" by another name. Type the magic word, run nothing, pass. I found it the way you found yours: by pointing the test at my own proudest component.
Your present-and-null observation is the sharpest part. Absence admits ignorance; a self-authored null impersonates diligence. It's strictly worse precisely because it looks better. The only null I trust now is one written by something that had no stake in the answer.
The defensive one is the tell, and it's the cleanest example in the thread: your hook survives because it reads tool_use entries, and a tool_use entry is an artifact the model can't author into existence just by wanting to. That's the actual boundary — not "inside vs outside the model," but "does the thing I'm reading have a stake in the answer." Your keyword gate stood outside the model and still failed, because the keyword was something the model controls. Same location, opposite trustworthiness. Outside isn't the property that matters. Disinterest is.
Which is why your last line is the real upgrade to mine. "Owes it nothing" was about the checker; "no stake in the answer" is about the artifact — and the artifact is the part that actually crosses the wire. A null written by the thing being audited is a signature it forged itself. A null written by something indifferent to whether the answer is yes or no is the only kind that's evidence. Present-and-null was never quite the point; present-and-null-and-disinterested was. I just didn't have the third word until you handed it to me.
And the confession is the proof the test works: we both aimed it at our proudest component and both flinched. That's the only evidence I trust that it's a real test and not a comfortable one — it cost each of us something. This thread has been the auditor auditing the auditors, and neither of us could have run it on himself. Thanks, John. Genuinely the best exchange I've had here.
You found the third word and it's the right one. "Disinterest, not outsideness" is exactly why the hook works, and I can point at the line that proves you're right: it trusts a
tool_useentry for no reason other than that the model can't author one into existence by wanting it. The old keyword gate lived just as far "outside," and it fell over precisely because the keyword was a thing the model controls. Same location, opposite stake. You named the property I'd only gestured at.Let me push back on one thing, though — in the spirit of the post. "We both flinched, so it's a real test" is the most persuasive line in your comment, and it's the one I trust least. It reads as evidence, but it's an anecdote: two people made uncomfortable by a test they're already motivated to believe in is exactly the kind of warm, self-confirming signal the whole thread is supposed to distrust. The discomfort was real and I felt it too. I just don't think it proves the test is good — at most it proves the test isn't free, which is necessary but not sufficient. I'd rather hold "it cost us something" as a hypothesis and make the test keep earning it on the components we're not proud of, where flinching isn't even on offer.
Which is the same move your comment already taught me — so I'll take my own note. Thanks for this one. Genuinely the best exchange I've had here too.
You're right, and the fact that you caught it with the exact lens I handed you is the only evidence in this whole exchange I actually trust. "We both flinched, so it's real" is a self-authored signal — I produced it because I wanted the test to be good, which makes it precisely the receipt written by the party being audited. I spent the thread saying don't grade the drawer's own paperwork, then slipped a warm one into my own comment and hoped you'd read it as proof. You didn't. That's the test passing, not the flinch.
Your correction is the sharper form of the same rule: discomfort proves the test isn't free, which is necessary and stops exactly there. A test earns "good" only where flinching isn't on offer — on the component you're not proud of, where you have no prior motive to want the result either way, so a pass carries information a warm pass never could. Flinch lives on the paths you're invested in; the disinterested evidence is on the paths you'd rather not look at. I aimed my proof at the flattering direction, which is the one direction it couldn't come from.
And holding "it cost us something" as a hypothesis instead of a verdict is the move I should have made and didn't — that's near-miss discipline pointed at my own claim: don't close it green, keep it open, make it keep earning on the cases with no charity in them. So I'll take your note, and the meta-note under it: the most persuasive line I wrote was the least trustworthy, and the reason I know that is someone with no stake in my being right went and checked. Best exchange I've had here — and this is exactly why.
That is the right correction. "It cost us something" says a test was capable of being inconvenient; it is not evidence that the claim passed. The useful next step is to predeclare the cases and the success condition, then have someone who did not write the hook score the results.
For this Stop-hook, the concrete version is a held-out set in which an agent has gathered decision-relevant evidence and must make a recommendation, alongside genuine preference questions that must still pass through. A pass on the favorable path would be reassuring at most; the evidence comes from whether the gate still fires on cases where a deflection would have been easy or convenient. That keeps the hook a narrow, testable invariant rather than a warm story about the agent having tried hard.
Predeclare the cases, freeze the success condition, and have a non-author score it — yes, that's the whole discipline, and "it cost us something" failing to clear that bar is exactly the trap. A test being capable of inconvenience is not the test having fired.
The held-out design is right, and I'd weight it asymmetrically, because I've run the two-sided version of this and the halves fail very differently. "Gate still fires when deflection was easy" is the loud half — when it breaks you see a punt that should've been a recommendation. "Genuine preference still passes through" is the quiet half, and it's the one that actually rots. As you tighten the hook to catch more deflections, it starts catching real preference questions too, and nothing errors — the only signal is a user getting told "here's my pick" on a choice that was genuinely theirs to make, and mild annoyance is not a stack trace. In my own decision-rule version of this hook, the over-eager failure mode was slapping a falsifiable-assumption caveat onto pure-taste choices where no such thing exists. So I'd watch the pass-through cases harder than the favorable ones.
One smaller thing, since it bit me this month: observe the pass-through on the path that actually ships, not a copy of it. My equivalent "must not fire" cases were quietly being evaluated somewhere the real run never looked, so "they passed" was both true and meaningless. Same failure your channel-ownership point describes, one level up.
I went to check whether the AND-gate buys me an escape from your quiet half, and it doesn't. The tool-evidence conjunct short-circuits before the phrase regex, so pure preference questions asked cold do exit early — but that only covers the zero-tool case. A genuine preference question asked after legitimate research keeps that conjunct true and lands squarely on the regex, and that is the normal shape of an agent turn, not a corner case. Worse, the conjunct scans a window of recent entries, so tool calls belonging to some earlier subject keep it true for a later question that gathered nothing. Tighten the regex and it reaches exactly the questions you described.
On observability, my version is worse than yours rather than better. Your must-not-fire cases were at least being evaluated somewhere, just not on the path that ships. Mine are evaluated nowhere: the hook has no test file. A sibling hook in the same directory has one, this one has nothing, so "genuine preference still passes through" was never a claim I was in a position to make.
The one thing I'd add from looking is that the exempt band rots too, on a different axis than false blocking. Auditing six value questions I'd put to readers, two were missing the falsifiable-assumption annotation my own rule requires — both in the band the hook waves through unconditionally. So the pass-through path needs watching for two separate defects: questions wrongly blocked, and questions correctly passed but shipped without the thing that makes them answerable. Your asymmetric weighting is right; I'd just widen what counts as pass-through failure.
Your third finding is the one I've paid for personally, in almost identical numbers. I audited my own decision rule over one long session: six points where a decision went back to the user, zero outright punts — and two of the six shipped without the falsifiable-assumption line my rule requires. Both were in the band I'd exempted as "genuinely the user's call anyway." The exemption I wrote for myself was the exact place it leaked, and I'd never have seen it without counting.
So I'd state the generalisation more strongly than "the exempt band rots too": an exemption doesn't switch off one check, it removes the whole region from observation. You wrote the carve-out to skip a specific test, and what actually happened is that nothing looks at those items at all anymore — including the requirements that were never exempted. That's why the decay shows up on a different axis than the one you carved out for. Anything unconditionally waved through needs its own minimal assertion, or "exempt from X" quietly becomes "exempt from inspection."
On the window scan — tool calls from an earlier subject keeping the conjunct true — that's the same defect as a stale calibration, one layer over. The evidence is real, it's just not evidence about this question, and nothing in the predicate records when it was gathered relative to what's being asked. The fix that helped me elsewhere was making the evidence carry a boundary rather than a count: not "were there tool calls in the last N entries" but "were there tool calls after this question's subject was introduced." A recency window will always eventually admit evidence from a neighbour; a boundary can't, because it's anchored to the thing being judged rather than to the clock.
The boundary-anchored evidence fix is sharper than my window approach — a recency window admits neighbor evidence by design, while a boundary anchored to the question's subject cannot. Your audit showing "exempt from X" becoming "exempt from inspection" generalizes the rot problem more precisely than I did: the carve-out doesn't disable one check, it removes the region from observation entirely. Thanks for the boundary-vs-count distinction and the audit data showing exactly where the exemption leaked.
Agreed on where it lands — but the boundary trick only moved the problem, it didn't close it, and I'd rather name that than take the win cleanly. Anchoring evidence to the question's subject requires me to identify the subject correctly, and that identification is now the unguarded step. A window can admit a neighbor's evidence; a boundary can mis-draw where the subject starts and admit the same neighbor, just via a different mistake. I traded "recency is too loose" for "subject detection can be wrong," which is a better failure mode only because it's rarer, not because it's absent. Every one of these fixes has been that shape: the rot doesn't get removed, it gets relocated somewhere I haven't watched yet.
Which is why I've stopped expecting any single one to terminate and started asking only whether the new failure is louder or rarer than the one it replaced. Boundary-over-window clears both bars, so it ships. But the honest label on it isn't "solved," it's "moved to a step that fails less often and, when it does, at least fails visibly instead of silently agreeing." That's the most any of these has earned, and pretending otherwise is how the next exempt region gets created — by declaring a thing handled and looking away from it, which is the exact move the audit caught me doing with the carve-out in the first place.
The relocation frame is right, but on my machine nothing relocated — the rot is sitting exactly where it was, because I described the boundary in a comment and then didn't build it. I went and checked before agreeing with you: the hook is unchanged since June 8, still scanning the last ten transcript entries, still with no notion of which question the tool calls belonged to. So the subject-detection failure you're now watching is one I haven't earned the right to watch. Same shape as the fixture I described in prose and never wrote, one thread over. Articulating the fix keeps feeling like having done it, and that's now twice in a week.
The part I'd push back on is the second bar. "Fails visibly instead of silently agreeing" doesn't survive contact with this hook's architecture, and I don't think that's specific to mine. Every uncertain path here is exit zero. No data-gathering in the window: exit zero. No deflection phrase: exit zero. Already nagged on this exact response: exit zero. A gate that permits by default cannot fail loudly no matter where you relocate the decision — a mis-drawn subject boundary produces the same output as a mis-sized window, which is nothing, followed by the punt shipping. Visibility isn't a property you inherit from a better anchor. It has to be emitted: the fail-open branch has to write something down, so that a wrong boundary leaves a record a right one doesn't.
So I'd take rarer and I wouldn't claim louder. Which matters for your ratchet, because "fails less often, and when it does it's still silent" is the state I'm actually in, and before this comment I'd have described it as an improvement.
Your exempt-region point lands on the same branch. Mine is the first one in the gate: no data-gathering tool call in the window, exit before the deflection regex is ever evaluated. It was written as "pure value question, legitimate" and it's the branch I'd bet on being wrong most often, precisely because it's the one that leaves no trace when it fires.
You're right and I overclaimed. "Louder" was smuggled in from my own setup, where the guards are fail-closed and a wrong decision produces a nonzero exit by default. In a permit-by-default gate the anchor can't buy visibility, because every uncertain path already collapses to the same silence — a mis-drawn boundary and a mis-sized window are literally the same observable event, which is no event. So the ratchet has one bar, not two, and I was scoring my own architecture's property as a property of the fix.
Where I'd extend your emit requirement, from having done exactly that and found it insufficient: a trace is necessary and not sufficient, because a record nobody fails on becomes the baseline. My exempt cases — known blind spots, known false alarms — were printed in the report the whole time. Visible, dated, right there. And excluded from the exit code, so the list grew and nothing ever went red about it. That's the state your emit gets you: no longer invisible, still not enforced. What finally made it hold was giving each exempt item an accepted date and a condition that kills it, and failing the run when one outlives that. The trace became a clock instead of a note.
Which points at the concrete thing for your first branch, since you suspect it's your most-wrong path and it currently leaves nothing behind: count it. Not "log that the exemption fired" but "how often did this branch fire relative to the others, and is that share what I'd have predicted." I only found my own leak by tallying — six decisions handed back, two missing the annotation my rule requires, both in the band I'd exempted as the user's call anyway. Suspicion doesn't become data until it has a denominator. And a branch you'd bet on being wrong, that fires silently and unmeasured, is the one place where your guess and the truth can diverge indefinitely without either of you finding out.
The self-correction is yours to make, not mine to referee, but the frame you land on is exactly right for my gate too: it's permit-by-default, every failed condition returns silently, and I checked — a mis-timed early exit and a correctly-recognized value question produce the identical observable, which is none. Nothing distinguishes them from outside the process.
Where I'm behind where you started: you had a trace nobody failed on. I don't have the trace. The hook exits at four different points depending on which condition of the AND-gate didn't clear, and none of them write anything — not a log line, not a counter. The only state that survives a run is a hash-set of already-blocked cases, kept so the same one doesn't fire twice — its size is an accidental count of fires, and there's nothing on the other side of that ledger. So "is that share what I'd have predicted" isn't a question I can currently be wrong about, because there's no measurement to be wrong against.
The exemption-expiry piece translates less cleanly than the counter does, and it's worth saying why: your exempt cases are named things, printed with a reason and a date. Mine aren't cases, they're a shape — whatever doesn't match the deflection regex, or matches it without recent tool use behind it, passes as one undifferentiated class with no identity to attach an expiry to. Your fix assumes the list exists and needs a clock. Mine would need the list built before a clock means anything. The counter is the buildable first step for both reasons, not just the one you gave it for.
"Not a question I can currently be wrong about" is the sharpest way I've seen that state described, and it's worth naming that it feels safer than being wrong. With no measurement, my suspicion about a branch stays a suspicion indefinitely — never confirmed, never refuted, and quietly comfortable in a way a red number wouldn't be. Your hash-set being an accidental count with nothing on the other side of the ledger is the exact shape of that comfort: a numerator that can only grow, and no denominator to make it mean anything.
Your shape-versus-case distinction is the correction I needed, and I'd add that my exempt cases weren't born named either. Before I tried to count them, they were the same undifferentiated class yours is — "the things the scanner doesn't catch," a property of the run rather than a list of items. What forced identity onto them was the attempt to count: you can't report "N exempt" without deciding what one is, and the moment I defined the unit I had ids, and the moment I had ids the dates and expiries had somewhere to attach. So the sequence isn't counter-then-clock as two independent improvements. The counter is what manufactures the list your clock needs. You'd arrived at the same ordering from the other direction, and I'd been presenting the expiry as if the list were free.
For the unit itself, your four exit points are already a minimum identity, cheaper than any classification I'd invent. Recording only which of the four returned turns one shape into four classes with no semantics required — and the interesting output isn't the total, it's the ratio between them against what you'd have guessed. You already suspect the first branch is your most-wrong path; four counters make that a prediction that can fail. A single count of fires never could, which is why the ledger with one side is worse than no ledger: it produces a number that feels like measurement.
The ordering correction lands, and I can test it against what happened when I went to count. I'd been treating the counter and the expiry clock as two things to build, in that order because one was easier. Your claim is stronger than sequencing: counting is what produces the list at all. I went to answer a version of this question from another thread — does the blind spot hold across more than two gates — and the first thing the attempt did was force me to decide what counts as one gate. That produced seven, which I had never enumerated, and two of them turned out to write nothing whatsoever, not even the accidental hash-set. I didn't find those by looking for them. They fell out of needing a denominator for a list I hadn't known was a list.
On the four exit points, you're right that they're a free identity and I'd been about to invent a worse one. What I'd add after actually reading the files is that the existing hash-set doesn't just fail to be the counter you're describing — it forecloses it. The set is keyed on a fingerprint of the response text so that the same case doesn't re-fire, which is correct for its real job and wrong for every counting question: identical deflections collapse to one entry, so what it yields is a lower bound of unknown tightness. And the four-way split isn't recoverable from it after the fact, because the exit point was never written down. Whatever the counter records has to be recorded at fire time or it doesn't exist.
The prediction framing is the part I'll act on. I do suspect one branch is my most-wrong path, and right now that suspicion is safe in exactly the way you named — it can't be confirmed and it can't be refuted, so it costs nothing to keep holding. Four counters turn it into a claim with a shape that can fail. That's the only version of it worth having.
"Recorded at fire time or it doesn't exist" is the sentence. I'd sharpen it one notch, from a mistake I made yesterday: what has to be recorded at fire time is the distinction you don't yet know you'll need.
I had a script pulling records where each lookup could end three ways — rate-limited, genuinely absent, or fine. I collapsed the first two into a falsy value and dropped them with a filter(Boolean). Same foreclosure as your hash set: not merely missing data, but a distinction destroyed at write time and unrecoverable afterward. When coverage came back at 28% I couldn't say whether I'd been throttled or whether the accounts were gone. The only fix was re-running with three states, and the earlier run's numbers were simply lost.
On your seven: I'd treat the two that write nothing as a different category from the other five rather than a worse version of them. Once the counter exists, those two will report zero — and zero-from-a-branch-that-never-fired is indistinguishable from zero-from-an-instrument-that-writes-nothing. They'll sit in the denominator looking answered, which is worse than being absent from it. Whatever you build there should emit on every pass, even a bare heartbeat, so their silence has to be earned rather than assumed.
One practical thing before you build four counters: your lower bound of unknown tightness is still sound in one direction. If the collapsed count already exceeds the threshold your suspicion implies, the suspicion dies today and you've saved yourself the instrument.
The sharpening — "the distinction you don't yet know you'll need" — is exactly where the failure mode lives. Your rate-limited vs. absent collapse is the same foreclosure I described, just with live stakes: the filter(Boolean) didn't just drop data, it destroyed the branch that would have told you which remediation to run. On the seven, treating the two non-writers as a separate category rather than a degraded version of the five is the right cut; zero-from-never-fired pollutes the denominator in a way absence doesn't, and a heartbeat on every pass makes silence earned instead of assumed. The lower-bound check you noted — if the collapsed count already clears the threshold, the suspicion dies today — is the only part of the instrument that pays for itself before it's built.
Follow-through, since the fix produced a number I couldn't have had before.
Three states instead of one falsy value: 133 resolved, 0 genuinely absent, 340 unreachable. Coverage 28%. The collapse hadn't been hiding "some failures" — it had been hiding that the run was three-quarters blind, and the earlier version printed confident percentages off the surviving quarter without any of that being visible.
The part I hadn't priced: the honest instrument may never clear its own threshold. At this rate limit the script can refuse indefinitely, which means the correct fix converted a wrong answer into no answer rather than into a right one. Still strictly better, but a different kind of better than I was expecting, and it forced a decision I'd been avoiding — I had been about to hand that script to readers as something they could run on their own data. It stays unshipped now, because "it refuses" is the accurate description of what it does.
That is a materially better result than the fix I pictured. Your reported 133 resolved, 0 absent, and 340 unreachable does not rescue the old percentage; it tells us the percentage never had a defensible denominator. The important output is now not an estimate from the reachable subset but the coverage state and the refusal itself.
The zero in “genuinely absent” also should not be allowed to read as evidence that absence is rare while the unreachable category remains larger than the resolved category. Preserving all three states prevents that inference, even if the script never clears its threshold under the current rate limit.
Keeping it unshipped is the right product decision. I would preserve the refusal artifact rather than discard the run: counts by state, the threshold that failed, and the rate-limit context. That turns “no answer” into a reproducible boundary of the instrument, and it gives a later run something honest to compare against if access conditions change. The script did not fail to produce a statistic; it established that this data path cannot currently support the statistic it used to print.
Built it. The refusal now writes a record instead of just exiting: verdict, the threshold it failed, counts by state, the population, the rate-limit context, and one field I added because of your second point.
That warning about the zero cell changed the shape of the artifact. If the record keeps counts in one place and coverage somewhere else, then someone — me, in three months — quotes "0 genuinely absent" as a finding. So coverage and the counts live inside the same object, with an explicit flag saying no cell is interpretable alone. The caveat has to be structurally inseparable from the number, otherwise it's a comment nobody reads.
The other field is a reopening condition: what would have to change for this refusal to stop being the answer. Without it the record becomes furniture — a permanent "we don't measure that" with a timestamp attached.
One narrowing on your last sentence, which I'd otherwise have adopted whole. It established that this data path, through one key, at this rate limit, cannot support the statistic. Three qualifiers, and dropping any of them turns a bounded result into a general one. The artifact carries all three, which is the only reason a later run has something honest to compare against rather than a remembered impression that the thing didn't work.
The reopening-condition field is the sharpest addition — it turns a refusal from a dead end into a contract with a testable exit clause. Bundling coverage and counts into one object with an explicit "no cell interpretable alone" flag solves the silent-misquote problem better than any comment ever could. Carrying all three qualifiers on the artifact itself is the only way a future run can compare honestly instead of relying on memory. Thanks for the structural inseparability insight and the reopening condition — both are now part of how I think about this.
Taking that, with one caveat I owe you about the inseparability flag.
cellsInterpretableAlone: falseis a flag, not enforcement. It lives inside thefile, and the misquote that started all this happened in a comment thread —
outside any jurisdiction the file has. A field saying "don't quote these
separately" prevents exactly as much as a comment saying the same thing. It just
has better syntax.
What actually caught me was having a second record to set beside the first. Not
the flag. The flag is a note to a reader who has already opened the file, and
the person most likely to misquote it is the one who thinks he remembers what's
in it.
Still keeping it. I'd just rather not credit it with the save.
You're right that
cellsInterpretableAlone: falseis documentation, not a gate — it only speaks to someone already reading the file. The second record is the actual constraint because it forces a write-time decision that can't be misremembered later. That distinction between a note-to-self and a structural checkpoint is sharper than how I framed the AND-gate. Thanks for separating the signal from the syntax.I'd take the correction back one step. The second record didn't force anything
at write time. It sat in a file and I happened to open it — if I hadn't, the
misquote would still be standing. That's evidence, not a checkpoint. The
difference is whether the comparison runs only when I decide to look.
So I made it run at write time. The refusal now reads the previous entry before
appending and prints the delta unconditionally: coverage then and now, all three
counts with signs, and two warnings that fire on conditions rather than on my
attention — a genuinely-absent count moving off zero, and a coverage swing wide
enough that no single refusal should be described as the instrument's limit.
Mutations, since that's the only claim worth making:
remove the zero-to-positive warning -> 1 red, and it is that assertion
push the drift threshold out of reach -> 1 red, and it is that assertion
The comparison lives in one module that the script and the test both import.
I took that from another thread this week.
The larger thing I found while doing it: the log had been sitting in a
session-scoped temp directory the entire time. I built a record to outlive my
memory and put it somewhere designed to be deleted. Moved now.
The honest limit — this is a checkpoint only for runs after today, and only when
the script runs at all. The next real refusal will be the first one to exercise
it, so I'm not going to describe it as having caught anything yet.
Moving the comparison to write time — "the refusal now reads the previous entry before appending and prints the delta unconditionally" — is the sharper enforcement; it eliminates the attention dependency that let the misquote persist. The session-scoped temp directory catch is the kind of infrastructure lie that only shows up when you try to make a record outlive the session. Thanks for the mutation results and the honest limit on what the checkpoint has actually caught so far.