Your Feedback Made This Better — Here's What Changed
The comments on the GateGuard and Neural Gate articles — from Mike Czerwinski, Dipankar Sarkar, René Zander, Max Quimby, and others — weren't "great post!" They were actual questions that made me realize what I hadn't tested.
So I tested it. 440 API calls across two experiments. Some things held. One hypothesis didn't. Both taught me something.
Mike's Two Questions
Mike Czerwinski asked two very specific things.
1. "Do the ~0.7% residual violations cluster on task types the gate doesn't instrument?"
I designed P1-1: five task types spanning the mechanizability gradient, 40 trials each, 200 API calls. Deterministic regex scoring — no LLM judge, because Dipankar warned against that and he was right.
The answer is yes, and the pattern is sharper than I expected:
| Task Type | Gate Reaches? | Compliance | What Failed |
|---|---|---|---|
| Code block tags (L1) | Yes | 100% | Nothing — zero violations |
| Section headers (L1) | Yes | 100% | Nothing — zero violations |
| Checklist format (L1/L2) | Partially | 0% | ALL semantic (content irrelevant) |
| Reasoning depth (L2) | Weakly | 35% | ALL semantic (reasoning shallow) |
| Uncertainty stance (L2/L3) | No | 42.5% | 91% mechanical (regex proxy fails) |
Where the gate reaches: zero violations, every single time. Violations are dominant everywhere the gate doesn't reach.
The checklist task shows this most directly. The model faithfully writes - [ ] checkboxes every time (mechanical compliance 100%), but the content is never deep enough to pass the "relevant to the question" check. The gate verifies the artifact EXISTS. It can't verify it's GENUINE. Mike's receipt-of-action vs receipt-of-diligence distinction — this is exactly that, operationalized.
2. "Does format still matter for reasoning depth when GateGuard is off?"
I designed P1-2: a 2×2 factorial — Format (code-style vs prose) × GateGuard (on vs off), 240 API calls. I pre-registered the hypothesis: format matters MORE when the gate is off.
The hypothesis was wrong.
| Condition | Mechanical (CI95) | Reasoning (CI95) |
|---|---|---|
| code + gate on | 5.00 [5.00, 5.00] | 4.20 [3.98, 4.40] |
| code + gate off | 3.15 [3.07, 3.23] | 3.98 [3.83, 4.12] |
| prose + gate on | 3.25 [3.03, 3.45] | 4.42 [4.25, 4.58] |
| prose + gate off | 3.15 [3.05, 3.25] | 4.17 [3.95, 4.37] |
Format effect on reasoning: d=−0.277 (gate on) vs d=−0.250 (gate off). Nearly identical. The gate doesn't modulate format's effect on reasoning at all — prose beats code by ~0.25 SD regardless of gate state.
The thing I didn't expect: code-format rules + GateGuard ON create what I'm calling "checklist mentality." Perfect 5.0/5 mechanical compliance — the model hits every required tag and keyword. But reasoning depth drops to 4.20/5, near the bottom. Prose rules + GateGuard ON gives the best reasoning (4.42/5) — the narrative engages holistic compliance without triggering checkbox-mode.
Mike wrote: "Syllogism only buys you anything in exactly the world you're arguing nobody should run in." The data confirms this. Code format buys perfect mechanical compliance — in a world where the gate already eliminates mechanical violations. In the un-gateable dimension (reasoning quality), prose is simply better. Code format optimizes for the environment being engineered away.
Pre-registering a hypothesis and getting a null result sucks. But because I scored everything deterministically (regex only, Dipankar's rule), the null is clean — no p-hacking to suspect.
Dipankar's Measurement Discipline
Dipankar Sarkar pushed on three things that changed how I work.
Decision-token measurement. He pointed out that averaging logprob delta across all tokens hides signal — "penetration lives at the decision tokens." I wrote a supplementary analysis re-scoring at decision tokens only, with token positions pre-annotated from the operational definition manual before touching the data. (Positions were pre-fixed before any scoring pass; no boundary was drawn or adjusted after seeing results — prevents lookback bias.)
The aggregate finding survived (d=0.578), but 8/40 probes that looked null under average delta showed clear divergence at decision tokens. The original measurement was conservative — undercounting, not inflating. Decision-token scoring is now the standard for all future experiments.
LLM-judge bias. "If the judge is an LLM, it carries its own format sensitivity. You'd be measuring the oracle's bias, not the gate." Both P1-1 and P1-2 use deterministic regex scoring exclusively — no LLM anywhere in the evaluation pipeline.
Semantic-only design. "Hold the mechanical gate fixed and score only the decisions no exit code can judge." This is now the template for all future experiments.
René Zander Built the Same Thing
René commented linking to skillgate — an npm package (@reneza/skillgate) that implements deterministic, model-independent gates for AI coding agents.
I read his articles and code. He built the same architecture from the same theoretical constraint. Independently.
Skillgate's design: "The model requests, the harness owns the boundaries." Every check runs as a pure function over the filesystem. No model in the loop. Gate types: file-exists, file-contains, absent, command, evidence, instruction-sync. His instruction-sync gate — tracking drift between CLAUDE.md, AGENTS.md, and .cursor/rules — is something I hadn't thought of and plan to adopt.
This isn't collaboration. We didn't know about each other's work. Two people, starting from the same structural constraint (generation and verification share P(token|context;θ), so self-verification is unreliable), arrived at the same architectural solution (deterministic filesystem checks outside the model's control loop). The fact that this happened twice suggests it's not a style preference. It's an engineering necessity.
Where we diverge: skillgate is production-grade, static, shipping on npm. My system adds four things: a self-referential loop (the agent detects its own self-model staleness and triggers regeneration), neural gates (token-probability probes inside the generation distribution), causal encoding (format engineering that reroutes attention), and drift prediction (trend-based early warning). Whether these additions are practically useful or just academically interesting — not settled yet.
Max's Boundary Question
Max Quimby: "Where do you draw the line between 'gate it' and 'can only nudge it'?"
The five-layer classification is documented in the paper, but it's still manual prose — a "good map drawn by hand," as Mike put it. A mechanizability-scanner that infers layer from rule structure is the next build. Max's question is the right one, and I don't have a mechanical answer yet.
Three Things I'm Thinking Now
1. Production engineering is more realistic than I thought
When I started, "five-layer verification architecture" felt like a research artifact — maybe useful for my own sessions, not something you'd ship. The comments changed that. René already shipped the L1 piece as an npm package. Dipankar's decision-token measurement is something a production system could compute in real-time. Mike's receipt-of-diligence concept points at a concrete problem: verifying that a written artifact reflects genuine work, not just artifact existence. The gap between "paper" and "product" is smaller than I thought.
2. I tried making it general — it's not there yet
After the GateGuard and Neural Gate articles, a few people asked whether this could become general infrastructure rather than one person's config. I think the question is worth taking seriously, so I tried.
I built a paper-validation-agent — a sub-agent that sits alongside the main coding agent, reads its outputs, runs mechanical checks, and can block non-compliant operations. It's a separate process with its own context window, so the main agent's drift can't affect it.
It kind of works. It can run pre-registered experiments, check compliance with regex patterns, detect when the self-model goes stale. But setup is entirely manual. Configuration is fragile and tailored to my machine. Error handling is minimal. I haven't let anyone else try it.
Whether this becomes a real tool depends on things I haven't solved yet — cross-machine portability, config management that works for someone else's setup, packaging that isn't just "clone my repo and figure it out." MCP packaging seems like the most practical route (install as tools, no subprocess overhead), but I haven't built it. Direct embedding (compile to a library) would be fastest at runtime but hardest to maintain across platforms.
If I had to guess: the mechanical gate layer (L1) is the easiest to generalize — filesystem checks are universal, René already shipped it. The neural and causal layers (L2/L3) are harder to separate from my specific setup and API access. Drift prediction (L4) needs longitudinal data from more than one user before it means anything.
I'm going to keep pushing on this. But right now it's a research prototype, not infrastructure. If you want to build something similar or try it yourself, I'd rather talk honestly about what's broken than sell you a roadmap.
3. The honest limits
Everything currently working is at the L1 (mechanical) level. Pre-registered experiments against DeepSeek API work. Deterministic mechanical compliance checking works. Self-model staleness detection and regeneration works. Execution debt tracking works.
What doesn't: cross-machine portability, semantic quality verification (that's the Prose Barrier wall — can't verify reasoning depth or content accuracy mechanically), cross-model logprob verification (DeepSeek-only), real-time intervention during a coding session.
L2 (neural) works for measurement but not intervention. L3 (causal) confirmed experimentally but not operationalized. L4 (drift prediction) built but predictive validation pending. Next step is making the gates MCP-installable so someone other than me can actually try them.
What's Next
Cross-model logprob replication. The L2 finding (d=+0.578) is DeepSeek-only. Claude and GPT-4o logprobs needed to test whether format effects are model-specific or general. Blocked on API access.
Mechanizability-scanner. A tool that reads a rule and infers which layer it belongs to — closing Mike's "good map drawn by hand" gap. Building this next.
MCP packaging. Making the mechanical gates installable as MCP tools. The most practical path to letting others test this.
Receipt-of-diligence artifacts. What file contents prove genuine review happened? Diff caught? Specific value computed? Exit code from a real run?
Instruction-sync adoption. René's idea — track drift between project instruction files. Immediately useful, mechanically implementable.
If you've built something in this space — deterministic verification, agent drift monitoring, format engineering — or if you see something in the data that doesn't hold up, I want to hear about it. The P1-1 and P1-2 experiments exist because Mike asked questions I hadn't thought to ask. The paper, data, and all supplementary analyses are at github.com/YuhaoLin2005/hermes-workspace. I rewrote the README in English so both researchers and practicing engineers can read it — because the feedback that mattered most came from both.
The standalone validation harness that internalizes the 5-layer architecture as reproducible Python modules — with all 8 governance claims as independent experiments, clear limitation statements, and a paper-format README that doesn't oversell — is at github.com/YuhaoLin2005/paper-validator. One-command audit: python -m paper_validator claim --claim all --trials 30.
50+ sessions of data. 13 experiments. One laptop. Still going.
👋 Yuhao Lin — hermes-workspace
Top comments (1)
440 API calls to get a null result you're not ashamed of is a better trade than most people make. The pre-registration holding up, regex-only scoring, positions fixed before the pass, that's the part that makes the rest of this worth reading instead of just believing.
But the finding I'd stop on isn't the null. It's that code-format plus gate-on lands reasoning at 4.20, the worst score on your whole table, worse than prose with the gate switched off entirely. The gate wasn't neutral there. It bought perfect mechanical compliance and, in the same breath, bought the checkbox mode that ate the reasoning depth it wasn't supposed to touch. That's not "format doesn't matter under the gate." That's the gate picking a side, quietly, and the side it picked was the worse one.
Which makes the receipt-of-action line cut deeper than I meant it to when I wrote it. The checklist task isn't just an example of the gate failing to see content, it's the gate actively rewarding the mode that produces content not worth seeing. A verifier that can't tell genuine from performed doesn't sit there neutrally waiting for L2 to catch up. It's already shaping which performance gets selected for, right now, at L1.
So before the mechanizability scanner, I'd want the smaller experiment: does forcing prose under a mechanical gate get you 100% compliance and 4.42 reasoning, best of both, or does something else break when you try to gate prose the same way you gate code. If that holds, the fix isn't a new layer, it's telling people the format their gate should demand.