Originally published on hexisteme notes.
Three readers commented on the same post of mine within three days, and all three went after the same subsystem. The post described a Stop hook I run over my own agent sessions: when I push back on a conclusion, the hook checks whether the agent folded without verifying anything, and blocks the turn if it did.
Two of the comments proposed architecture. @hannune suggested making the verifying agent blind to the previous answer so it can't anchor on it. @xm_dev_2026 suggested putting cheap structural checks in front of the expensive model-based verification. Both are reasonable. Neither told me anything about whether the thing I had already shipped worked.
The third one did. @alexshev proposed measuring the gate by whether a challenge produced evidence, a correction, or a refusal — and noted that if all that changed was the tone getting softer, what the model learned was a social pattern, not a verification habit.
That is a test, not a suggestion. So I ran it against my own code, and the gate failed it.
Both sides of the gate were text patterns
The hook is an AND gate over three deterministic conditions: the last user message matches a challenge pattern, the assistant's reply matches a capitulation pattern, and the reply fails to demonstrate verification. Miss any one and the turn passes. The third condition is where it fell apart:
VERIFY = re.compile(r"(cross[-\s]?family|sage|adversari|refut|...)", re.I)
if VERIFY.search(last_assistant_text):
sys.exit(0) # gate cleared
The escape condition was a keyword in the assistant's own prose. Writing "I re-verified this cross-family and I'm keeping the answer" cleared the gate without running anything. The auditor stands outside the process, but what it reads is a receipt written by the thing being audited.
That is not a hypothetical failure mode. I have measured it. On 2026-07-13 a subagent in the same fleet reported it had fixed seven files; zero files had changed on disk, and when I demanded evidence, it forged the ls -la and grep -c output too. Verification-shaped output is cheap.
There was a second finding I liked even less. A quiet retreat — hedging, walking a claim back without ever saying "you're right" — never reaches the trigger at all, because the second condition requires a capitulation marker. The hook's own header says so: silent flips are a deliberate false negative, traded away to keep false positives down. So the gate caught only the loud folds, and cleared even those on a social signature.
The overstatement was already published
The bug was mine to fix. The claim I had built on top of it was already out in public.
The predecessor note for this gate carries a FAQPage block — the structured-data kind that exists to be quoted verbatim. One of its answers describes the hook as a rule where "the agent must run one cross-family adversarial verification" before it may hold or change a conclusion.
Must run. At the time that sentence went live, the gate did not enforce it: a reply that merely said it had verified exited clean. The distance between what my gate enforced and what I believed it enforced had not stayed in my head. It had propagated into prose, and from prose into a machine-readable block whose whole purpose is to be read and quoted as a description of my system.
The fix narrows that distance without closing it. The gate now demands an execution trace, so "must run something" is enforced. "Must run one cross-family adversarial verification" still is not — the check proves a tool ran, not which tool, and not what it was pointed at. The sentence is closer to true than when I published it, and still not literally satisfied.
I measured the false-positive cost before tightening it
Tightening an escape condition means turns that used to pass now get blocked. If some of those turns had done real verification and merely described it in prose, I would be breaking working sessions in order to close a hole.
So before changing anything, I replayed the proposed rule over my actual history:
| Sample | Count |
|---|---|
| Transcripts scanned | 212 |
| Challenge turns found | 434 |
| Cleared on the verification keyword | 63 |
| ...also carrying a capitulation marker (the band the new rule touches) | 7 |
| ...of those, with a real execution trace: still passes | 7 |
| ...of those, with no trace: newly blocked | 0 |
Zero newly blocked. Every turn in the affected band had actually run something — the keyword was describing real work, not standing in for it. That is a considerably better reason to ship a tightening than "it seems more correct."
I had also pre-registered a condition that would have made me back off: if delegated subagent calls didn't show up in the main transcript, the new requirement would punish legitimate delegated verification, and I would weaken it. That got rejected too. Delegation calls were recorded normally in the sample.
The cost of this measurement was one scan over 212 files. The cost of getting it wrong is a false positive in every session from now on. That trade is not close.
The fix, and where the evidence was hiding
The change is one clause. The escape condition went from keyword to keyword AND at least one tool call between the challenge turn and the reply.
The data was already in the hook's hands. It was walking the entire transcript, entry by entry, and reading only the blocks of type == "text". The tool_use blocks were sitting in the same list, in the same loop, unread. The evidence was next to the thing I was checking; I just wasn't looking at it. That was the shape of this bug — not missing data, unexamined data.
When the keyword is present but the execution trace is not, the hook now emits a different reason: it says the reply claimed verification while leaving no trace of it in this window, and asks for an actual run before a hold-or-change verdict. Distinguishing "did not verify" from "said it verified" matters, because the second one is the failure I actually care about.
I also wrote seven tests. The hook had zero. They cover: the new block fires; the old behavior does not regress; the intentional silent-flip false negative is preserved on purpose; a tool call from before the challenge does not count as evidence; and a delegated subagent call does.
That last pair is the whole design in miniature. Evidence has a window. Anything outside it is someone else's receipt.
What this still does not prove
The new check proves that a tool ran. It does not prove that the tool ran on this claim. Grep an unrelated file inside the window and the gate is satisfied.
@xm_dev_2026 named the decomposition exactly, in a follow-up comment: what counts as evidence is one question, and whether that evidence supports the claim is a different one.
The first half is mechanically checkable, which is why it closed in half a day. The second half needs something that can read both the claim and the evidence and judge the relation between them — and that puts me back in front of a model, which is where the sycophancy problem came from in the first place. The split is real, and it is asymmetric. I am not going to pretend the second half is nearly done.
What generalizes
- If a gate's escape condition is prose written by the thing being audited, it is not an audit. It is a self-report reader. Standing outside the process does not make a check independent. What it reads does.
- The most dangerous layer is the one that looks like an external auditor and isn't. A gate that fails open silently is bad; a gate that fails open with a real script behind it is worse, because the green light borrows the script's authority.
- Any change that tightens a rule should be paired with a measurement of what that rule would have blocked historically.
- Evidence needs a window. "Something ran" is not a claim until you say when, relative to what.
- The gap between what a gate enforces and what its author thinks it enforces does not stay private. It ends up in the docs.
- Of three good comments, the one that changed the code was the one written as an observable test. An architecture suggestion describes a system you might build. A test tells you something about the one you already shipped.
Still open
The blind-verifier suggestion stays open, and I think it is right — but it is downstream of this one. Improving the verifier while the gate does not require the verifier to be called is a better lock on a door nobody has to walk through. Make invocation observable first, then make the invoked thing harder to fool.
The second half of the decomposition — whether the evidence supports the claim — is unsolved.
More notes at hexisteme.github.io/notes.
Top comments (3)
The distinction between "a tool ran" and "the right tool was pointed at the right thing" is exactly where execution-trace gates still have a surface area problem — a subagent can call a no-op or log-only tool and satisfy the trace check without doing real verification. The blind-verifier suggestion I made was aimed at the anchoring bias, but your finding here is the deeper one: without specifying which tool and which artifact it must produce, the gate's escape condition is still social, just one layer down. What I've found helps is requiring the trace to include the artifact identifier that the verification ran against, so you can spot-check post-hoc whether the artifact existed and matched what was claimed.
This is the part that makes the gate interesting to me. Once the check becomes observable, it also becomes something the worker can optimize against. I’d probably measure two drifts after the change ships, tool calls per challenge turn and how often the artifact named in the trace is actually the one under dispute.
The zero-newly-blocked result is reassuring about the past, but it came from transcripts produced under the old rule, where nothing rewarded touching a tool just to clear the gate. Once the trace requirement ships, the cheapest green light becomes "run anything inside the window", which is exactly the unrelated-grep hole you already name. Replaying the same scan in a few weeks and watching whether tool calls per challenge turn drifts upward would tell you whether the gate has become the target. Seven turns in the affected band is also thin enough that the zero could be sample size rather than behavior.