6 sessions, 150 standardized tasks, 2 rule formats. The mechanical gate won. Everything else was noise.
The Question
For two months, ...
For further actions, you may consider blocking this person and/or reporting abuse
The ceiling effect is the finding, not the null result. You went in expecting format (syllogism vs imperative) to move the needle and got 99.3% compliance in both arms because GateGuard was mechanically blocking unverified writes regardless of how the rule was phrased. That's the same result I keep landing on from a different angle: I've been building deterministic confirmers that sit downstream of an LLM proposer specifically because the proposer's phrasing never survives contact with a model that wants to route around it. Your 55.9% -> 0.7% retrospective is the number that should anchor this post higher than the format experiment itself. Two things I'd want next: (1) did you check whether the ~0.7% residual violations cluster on task types the gate doesn't instrument, and (2) the "format still mattered for reasoning depth" finding, does that survive if you force both conditions through GateGuard-off? Right now it's confounded with the ceiling.
The metric correction is the right move and it makes the finding cleaner, not weaker: 100% by direct measurement beats 99.3% by extrapolation because now there's no soft number for a skeptic to poke at. The token-probability experiment is the genuinely new thing here though, and I think it's sharper than you're giving it credit for. d=0.578 on compliant-token likelihood without the gate says format really does move something. But look at where it moves it: only in the ungated regime. Under GateGuard, format is inert. So syllogism only buys you anything in exactly the world you're arguing nobody should run in. Format optimization is optimizing for the environment you're trying to engineer away, which is either an argument that it doesn't matter, or an argument that the gate can't be everywhere and format is your fallback for the gaps. Worth deciding which, because they point at different papers.
On your fix for the self-reported-review blind spot (require tangible review artifacts written to disk): that shrinks the gap, it doesn't close it, and it's worth seeing why so you don't overclaim it. The gate can now check that a file exists, not that its contents reflect a real review. An agent that confabulates "I reviewed this" can confabulate a plausible review artifact to sit next to the claim. You've raised the cost (now it has to fabricate a document, not just a sentence) and made the fabrication inspectable by a human later, which is real. But it's the same shape as the citation-relation problem going around this week: you've made the artifact's existence checkable, not its bind to an actual review. Receipt-of-action, not receipt-of-diligence. The durable version needs the artifact to carry something only a real review could produce, a diff it caught, an exit code, a specific value, not prose that asserts diligence. Otherwise the prose barrier just moves from the chat into the file.
Thanks. Picked Paper B — format matters where gates can't reach, those gaps are structural. On receipt-of-diligence: building review-artifact-guard.py to require evidence markers (diff caught, exit code, specific value). The prose barrier follows you — agreed, that's the core problem. Full follow-up: dev.to/yuhaolin2005/follow-up-deci...
Thanks. Paper B it is — format matters where gates can't reach, those gaps are structural. Haven't checked residual violation clustering by task type yet — will add to next run. The format→reasoning depth finding under GateGuard-off is exactly the right control. Full follow-up: dev.to/yuhaolin2005/follow-up-deci...
Paper B holding up under an actual run rather than staying a reframe on paper is the part worth flagging back to you, since you've now got both halves, the residual-clustering-by-task-type result lives in the other post (checklist task, 0% semantic pass despite 100% mechanical), and it's sharper than either of us expected: the gate isn't blind to violations there, it's rewarding the mode that produces them. Worth reading that one alongside this reply if you haven't yet, the task-type breakdown answers your own "haven't checked yet" from a few hours ago.
The ceiling effect is the most honest result you could have gotten, and I think it's actually the finding, not a spoiled experiment. The 55.9% → 0.7% baseline shift is the whole story: phrasing rules is tuning the probability of compliance; a mechanical gate converts a probabilistic constraint into a deterministic one. Once you've done that, format effects are noise by construction, because there's no longer a decision left for the model to get wrong.
Your shared-decoder argument matches what we've seen — asking a model to verify its own output fails for the same reason you can't proofread your own typos: the error and the check come from the same distribution. Independent, out-of-band verification is the only thing that reliably catches it.
The open question I'd chase next: what fraction of your rules are even mechanizable? Timestamps, exit codes, regex — great. But "use dual-pool review for big decisions" has a judgment component a hook can't fully capture. Curious where you draw the line between "gate it" and "can only nudge it."
Thanks. Your mechanizability question forced me to document the five-layer architecture with systematic classification (operates on files → L1, on token distributions → L2, on decisions → L3, on patterns over time → L4). Classification is still manual — mechanizability-scanner.py is next. Full follow-up: dev.to/yuhaolin2005/follow-up-deci...
The mechanical gate result makes sense. Agents are much better at working inside a constraint than reliably remembering a soft instruction across a long task. The interesting design lesson is that rule-following should be shaped like infrastructure: explicit checks, visible failures, and a narrow path forward.
Alex, "rule-following should be shaped like infrastructure: explicit checks, visible failures, and a narrow path forward" — this is now one of the design principles I cite directly. You named it before I had the language for it.
Since this article, several things have been built following exactly that principle:
Mechanizability scanner (v0.1.1): A deterministic regex classifier that scores rules 0.0–1.0 on structural gateability. Rules scoring ≥0.70 get infrastructure (mechanical gates). Rules scoring <0.30 get prose format with provenance tagging. The scanner itself is code, not prose — the classifier can't share the failure mode it's classifying.
Cross-model calibration (3 models, 200 API calls): The infrastructure works, but differently per model. DS Flash is near-perfect on all rules (obedient), DS Pro only follows L1 rules (judicious), Qwen ignores format rules even when explicit. Same gate, different effective coverage. Gate placement needs per-model calibration.
SHA256 pre-registration: Dipankar Sarkar's design — hash the hypothesis + conditions + scoring regexes, embed in every API record. Provider-timestamped, third-party verifiable. The pre-registration itself is infrastructure.
René Zander built the same pattern independently (skillgate, npm package) — same constraint, same solution. Two people arriving at "deterministic checks outside the agent's control loop" isn't coincidence. It's the shape of the problem dictating the shape of the solution.
Thank you. That is the part I find most practical in these experiments: rules become reliable when they stop being vibes and start becoming infrastructure. The model can still reason, but the boundary should not depend on mood.
That mechanizability scanner is a good next step because it forces rules to become observable. The tricky part will be avoiding false confidence: some important rules are partially gateable, so the score should probably say what can be checked and what still needs review.
The ceiling effect isn't just the honest result, it's a measurement boundary you can now name precisely. GateGuard fully covers the mechanical class: did it run, did it touch the file, exit code, regex on the diff. Both rule formats land at ~0% because that class is gated end to end, so format has nothing left to move. But that means the syllogism-vs-imperative question isn't dead, it moved to the violations a hook structurally cannot see: picking a worse-but-passing approach, or silently weakening a check so it goes green. Your one self-detected Honesty violation lived in exactly that class, semantic not mechanical. My bet is format effects only show up there, because that's the only place the model's own distribution is still the last line of defense. So the sharper next run holds the mechanical gate fixed and scores only the decisions no exit code can judge. The 55.9 to 0.7 baseline is the headline, but it's also why your instrument now has a blind spot worth measuring on purpose.
Thanks Dipankar — your "hold the mechanical gate fixed and score only the decisions no exit code can judge" became the exact template. Both P1-1 (n=200) and P1-2 (n=240) ran with deterministic regex scoring, no LLM judge anywhere. P1-1 tested exactly the design you described: gate held fixed, scored decisions in the un-gateable space. Your bet was right — format effects show up most clearly where the model's own distribution is the last line of defense. Full results: dev.to/yuhaolin2005/your-feedback-...
Your ceiling-effect result suggests two separate contracts: enforcement coverage and evidence provenance. GateGuard can prevent an unverified write, but the acceptance receipt still needs to bind the allowed write to a revision, command/output, and coverage boundary. I’d be curious to see the next run report gate coverage by task type plus a final-workspace identifier, so residual failures can be separated into “the gate never observed this action” versus “the gate observed it but the receipt was insufficient.” That split would make the independent-rater follow-up easier to interpret.
The result also suggests a useful distinction between enforcing a behavior and binding evidence to a specific artifact. A gate can require a test command and still accept a result from the wrong revision or from before a later edit. I would have the gate emit a small receipt containing the task identifier, artifact revision, command/configuration, exit state, timestamp, and declared coverage, then invalidate the receipt when any bound input changes. That turns “a check ran” into a narrower, inspectable claim: this check ran against this state. It still does not prove broader correctness, but it makes stale or mismatched evidence much easier to detect.