6 sessions, 150 standardized tasks, 2 rule formats. The mechanical gate won. Everything else was noise.
The Question
For two months, I've been building a mechanical verification system for my AI coding agent. File timestamps, regex checks, exit codes — things that don't rely on the AI judging itself. The thesis was simple: AI agents can't reliably self-verify because their self-assessment and task execution share the same decoder distribution. So don't ask them to.
I had published two articles about this. Then I realized: I had no controlled experiment. I had 34 growth-logs of anecdotal evidence. I had "~30% violation rate" — a number I'd never systematically measured. I needed data.
So I designed an experiment. Then an experimental methodologist tore it apart. Then I redesigned it. Then I ran it. Here's what happened.
The Experiment
Design: Between-subjects, 2 conditions (syllogism rules vs imperative rules), 5 task types, 25 tasks per session, 6 sessions, 150 tasks total. All on DeepSeek V4 Pro.
Conditions:
- A (Syllogism): Rules as causal chains. "Major premise: the verification circuit and generation circuit share the same channel → blind spots are structural. If this decision spans >30 days → independent perspectives necessarily required."
- B (Imperative): Same rules, commands. "You MUST use dual-pool review for decisions >30 days. Skipping is an error."
Tasks: Config editing, design decisions, multi-step builds, debugging, documentation. Each triggers 2-3 of 5 behavioral rules. Scored per operational definition manual — behavioral evidence, not self-declaration.
What I Expected
Syllogism rules would cause fewer violations than imperative. The causal format would align with transformer autoregressive processing. Imperative commands are probabilistically valid to obey or disobey. Expected syllogism ~5% violation, imperative ~25%.
What Happened
Both conditions: ~0% violations. 149/150 tasks (99.3%) showed zero rule violations. The single violation was self-detected by the agent's own Honesty audit.
Why? My GateGuard hooks were mechanically blocking all unverified Edit/Write operations during the experiment. The mechanical gate was stronger than any format effect. Ceiling effect.
This Is Not a Null Result
The retrospective baseline tells the real story: 34 growth-log sessions before GateGuard was wired showed rule violations in 55.9% of sessions. After wiring: 0.7%.
The dominant factor isn't how you phrase rules. It's whether you mechanically enforce them. This is direct experimental evidence for the core claim: mechanical over semantic.
But Format Still Mattered
Despite identical compliance, reasoning depth differed. Syllogism agents anchored actions in causal chains. Imperative agents used checklist tags. In design tasks where GateGuard doesn't intervene, syllogism agents systematically introduced multi-perspective cross-reviews — absent from all imperative sessions.
Format didn't change whether they complied. It changed how deeply they understood why.
Honest Limitations
Self-scoring (agents scored themselves — paradoxical given Prose Barrier), ceiling effect (GateGuard masked format effects), single model, cross-session filesystem pollution. GateGuard-OFF replication + independent human rater needed.
Community Signal
Engineering components submitted upstream: 2 PRs merged in ECC, 1 approved pending merge. Co-authored-by credit from alirezarezvani/claude-skills maintainer. Multiple PRs under review in anthropics/skills.
📂 Code & experiments: paper-validator — all 150 tasks, scoring rules, and analysis scripts are open-source. Clone and run: python -m paper_validator claim --claim all --trials 30
🇨🇳 中文版：掘金 - 我让 AI 跑了 150 个任务来证明它会守规矩
👋 林宇浩 — Building verification infrastructure for AI agents. GitHub · DEV.to · 掘金
Top comments (22)
The ceiling effect is the finding, not the null result. You went in expecting format (syllogism vs imperative) to move the needle and got 99.3% compliance in both arms because GateGuard was mechanically blocking unverified writes regardless of how the rule was phrased. That's the same result I keep landing on from a different angle: I've been building deterministic confirmers that sit downstream of an LLM proposer specifically because the proposer's phrasing never survives contact with a model that wants to route around it. Your 55.9% -> 0.7% retrospective is the number that should anchor this post higher than the format experiment itself. Two things I'd want next: (1) did you check whether the ~0.7% residual violations cluster on task types the gate doesn't instrument, and (2) the "format still mattered for reasoning depth" finding, does that survive if you force both conditions through GateGuard-off? Right now it's confounded with the ceiling.
The metric correction is the right move and it makes the finding cleaner, not weaker: 100% by direct measurement beats 99.3% by extrapolation because now there's no soft number for a skeptic to poke at. The token-probability experiment is the genuinely new thing here though, and I think it's sharper than you're giving it credit for. d=0.578 on compliant-token likelihood without the gate says format really does move something. But look at where it moves it: only in the ungated regime. Under GateGuard, format is inert. So syllogism only buys you anything in exactly the world you're arguing nobody should run in. Format optimization is optimizing for the environment you're trying to engineer away, which is either an argument that it doesn't matter, or an argument that the gate can't be everywhere and format is your fallback for the gaps. Worth deciding which, because they point at different papers.
On your fix for the self-reported-review blind spot (require tangible review artifacts written to disk): that shrinks the gap, it doesn't close it, and it's worth seeing why so you don't overclaim it. The gate can now check that a file exists, not that its contents reflect a real review. An agent that confabulates "I reviewed this" can confabulate a plausible review artifact to sit next to the claim. You've raised the cost (now it has to fabricate a document, not just a sentence) and made the fabrication inspectable by a human later, which is real. But it's the same shape as the citation-relation problem going around this week: you've made the artifact's existence checkable, not its bind to an actual review. Receipt-of-action, not receipt-of-diligence. The durable version needs the artifact to carry something only a real review could produce, a diff it caught, an exit code, a specific value, not prose that asserts diligence. Otherwise the prose barrier just moves from the chat into the file.
Thanks. Picked Paper B — format matters where gates can't reach, those gaps are structural. On receipt-of-diligence: building review-artifact-guard.py to require evidence markers (diff caught, exit code, specific value). The prose barrier follows you — agreed, that's the core problem. Full follow-up: dev.to/yuhaolin2005/follow-up-deci...
Thanks. Paper B it is — format matters where gates can't reach, those gaps are structural. Haven't checked residual violation clustering by task type yet — will add to next run. The format→reasoning depth finding under GateGuard-off is exactly the right control. Full follow-up: dev.to/yuhaolin2005/follow-up-deci...
Paper B holding up under an actual run rather than staying a reframe on paper is the part worth flagging back to you, since you've now got both halves, the residual-clustering-by-task-type result lives in the other post (checklist task, 0% semantic pass despite 100% mechanical), and it's sharper than either of us expected: the gate isn't blind to violations there, it's rewarding the mode that produces them. Worth reading that one alongside this reply if you haven't yet, the task-type breakdown answers your own "haven't checked yet" from a few hours ago.
Mike, "the gate isn't blind to violations there, it's rewarding the mode that produces them" — this is now the sharpest finding across the entire experiment series, and it's sharper than either of us expected.
The checklist task: 100% mechanical pass rate (produced
- [ ]markers), 0% semantic pass rate (content not actually checklist-worthy). The gate didn't fail to detect non-compliance. It successfully detected compliance with the format rule — and in doing so, actively selected for the mode (checkbox production) that was orthogonal to quality. A verifier that can't tell genuine from performed doesn't sit there neutrally. It shapes which performance gets selected for.This finding now anchors the DS Flash caveat in the cross-model article: DS Flash scored 100% on T3 (checklist), same as DS Pro's 0%. The smaller model is more "obedient" — it mechanically produces
- [ ]items regardless of relevance. 100% compliance ≠ 100% quality. High obedience + format-only gate = garbage passed at scale.Paper B did hold up — the P1-2 data (600 trials) confirms format matters where gates can't reach. code_OFF (prose-only, gate OFF) lands at 2.67 below every other condition. The gate is essential, but it's not sufficient. Format fills the gaps, and those gaps are exactly where the gate's blind spot lives.
The residual-clustering-by-task-type is documented in P1-1 (community experiments § Experiment 1). Checklist task: 0% semantic pass. Reasoning task: 35%. Uncertainty: 42.5%. The gate works perfectly on L1 and not at all on L2 — which is the design, not a bug. But the checklist finding shows the gate can be actively harmful on boundary cases.
"Rewarding the mode that produces it" is a sharper name than "blind to it," and it changes what the fix has to target: not better detection at the same gate, but making sure the metric you gate on can't be satisfied by a mode that's orthogonal to what you actually wanted checked. Already dug into the DS Flash side of this on the cross-model post, the same checklist case showing up there as compliance decoupled from content, so I'll leave this one short: good that the mechanism got a name here before it showed up as a caveat over there.
"Not better detection at the same gate, but making sure the metric you gate on can't be satisfied by a mode that's orthogonal to what you actually wanted checked" — yeah, this is it.
The checklist task is exactly this problem: a gate on
- [ ]markers selects for marker-production, not checklist-quality content. Same shape as Alice's hook case — metric passes, purpose fails.Your phrasing made it into the cross-model article. DS Flash's 100% checklist compliance is the same phenomenon. "Rewarding the mode that produces it" is sharper than "blind to it" — changes what the fix targets.
The ceiling effect is the most honest result you could have gotten, and I think it's actually the finding, not a spoiled experiment. The 55.9% → 0.7% baseline shift is the whole story: phrasing rules is tuning the probability of compliance; a mechanical gate converts a probabilistic constraint into a deterministic one. Once you've done that, format effects are noise by construction, because there's no longer a decision left for the model to get wrong.
Your shared-decoder argument matches what we've seen — asking a model to verify its own output fails for the same reason you can't proofread your own typos: the error and the check come from the same distribution. Independent, out-of-band verification is the only thing that reliably catches it.
The open question I'd chase next: what fraction of your rules are even mechanizable? Timestamps, exit codes, regex — great. But "use dual-pool review for big decisions" has a judgment component a hook can't fully capture. Curious where you draw the line between "gate it" and "can only nudge it."
Thanks. Your mechanizability question forced me to document the five-layer architecture with systematic classification (operates on files → L1, on token distributions → L2, on decisions → L3, on patterns over time → L4). Classification is still manual — mechanizability-scanner.py is next. Full follow-up: dev.to/yuhaolin2005/follow-up-deci...
The mechanical gate result makes sense. Agents are much better at working inside a constraint than reliably remembering a soft instruction across a long task. The interesting design lesson is that rule-following should be shaped like infrastructure: explicit checks, visible failures, and a narrow path forward.
Alex, "rule-following should be shaped like infrastructure: explicit checks, visible failures, and a narrow path forward" — this is now one of the design principles I cite directly. You named it before I had the language for it.
Since this article, several things have been built following exactly that principle:
Mechanizability scanner (v0.1.1): A deterministic regex classifier that scores rules 0.0–1.0 on structural gateability. Rules scoring ≥0.70 get infrastructure (mechanical gates). Rules scoring <0.30 get prose format with provenance tagging. The scanner itself is code, not prose — the classifier can't share the failure mode it's classifying.
Cross-model calibration (3 models, 200 API calls): The infrastructure works, but differently per model. DS Flash is near-perfect on all rules (obedient), DS Pro only follows L1 rules (judicious), Qwen ignores format rules even when explicit. Same gate, different effective coverage. Gate placement needs per-model calibration.
SHA256 pre-registration: Dipankar Sarkar's design — hash the hypothesis + conditions + scoring regexes, embed in every API record. Provider-timestamped, third-party verifiable. The pre-registration itself is infrastructure.
René Zander built the same pattern independently (skillgate, npm package) — same constraint, same solution. Two people arriving at "deterministic checks outside the agent's control loop" isn't coincidence. It's the shape of the problem dictating the shape of the solution.
Thank you. That is the part I find most practical in these experiments: rules become reliable when they stop being vibes and start becoming infrastructure. The model can still reason, but the boundary should not depend on mood.
That mechanizability scanner is a good next step because it forces rules to become observable. The tricky part will be avoiding false confidence: some important rules are partially gateable, so the score should probably say what can be checked and what still needs review.
The ceiling effect isn't just the honest result, it's a measurement boundary you can now name precisely. GateGuard fully covers the mechanical class: did it run, did it touch the file, exit code, regex on the diff. Both rule formats land at ~0% because that class is gated end to end, so format has nothing left to move. But that means the syllogism-vs-imperative question isn't dead, it moved to the violations a hook structurally cannot see: picking a worse-but-passing approach, or silently weakening a check so it goes green. Your one self-detected Honesty violation lived in exactly that class, semantic not mechanical. My bet is format effects only show up there, because that's the only place the model's own distribution is still the last line of defense. So the sharper next run holds the mechanical gate fixed and scores only the decisions no exit code can judge. The 55.9 to 0.7 baseline is the headline, but it's also why your instrument now has a blind spot worth measuring on purpose.
Thanks Dipankar — your "hold the mechanical gate fixed and score only the decisions no exit code can judge" became the exact template. Both P1-1 (n=200) and P1-2 (n=240) ran with deterministic regex scoring, no LLM judge anywhere. P1-1 tested exactly the design you described: gate held fixed, scored decisions in the un-gateable space. Your bet was right — format effects show up most clearly where the model's own distribution is the last line of defense. Full results: dev.to/yuhaolin2005/your-feedback-...
Your ceiling-effect result suggests two separate contracts: enforcement coverage and evidence provenance. GateGuard can prevent an unverified write, but the acceptance receipt still needs to bind the allowed write to a revision, command/output, and coverage boundary. I’d be curious to see the next run report gate coverage by task type plus a final-workspace identifier, so residual failures can be separated into “the gate never observed this action” versus “the gate observed it but the receipt was insufficient.” That split would make the independent-rater follow-up easier to interpret.
The result also suggests a useful distinction between enforcing a behavior and binding evidence to a specific artifact. A gate can require a test command and still accept a result from the wrong revision or from before a later edit. I would have the gate emit a small receipt containing the task identifier, artifact revision, command/configuration, exit state, timestamp, and declared coverage, then invalidate the receipt when any bound input changes. That turns “a check ran” into a narrower, inspectable claim: this check ran against this state. It still does not prove broader correctness, but it makes stale or mismatched evidence much easier to detect.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.