Your Feedback Made This Better — Here's What Changed
The comments on the GateGuard and Neural Gate articles — from Mike Czerwinski, Dipankar Sarkar, René Zander, Max Quimby, and others — weren't "great post!" They were actual questions that made me realize what I hadn't tested.
So I tested it. 440 API calls across two experiments. Some things held. One hypothesis didn't. Both taught me something.
Mike's Two Questions
Mike Czerwinski asked two very specific things.
1. "Do the ~0.7% residual violations cluster on task types the gate doesn't instrument?"
I designed P1-1: five task types spanning the mechanizability gradient, 40 trials each, 200 API calls. Deterministic regex scoring — no LLM judge, because Dipankar warned against that and he was right.
The answer is yes, and the pattern is sharper than I expected:
| Task Type | Gate Reaches? | Compliance | What Failed |
|---|---|---|---|
| Code block tags (L1) | Yes | 100% | Nothing — zero violations |
| Section headers (L1) | Yes | 100% | Nothing — zero violations |
| Checklist format (L1/L2) | Partially | 0% | ALL semantic (content irrelevant) |
| Reasoning depth (L2) | Weakly | 35% | ALL semantic (reasoning shallow) |
| Uncertainty stance (L2/L3) | No | 42.5% | 91% mechanical (regex proxy fails) |
Where the gate reaches: zero violations, every single time. Violations are dominant everywhere the gate doesn't reach.
The checklist task shows this most directly. The model faithfully writes - [ ] checkboxes every time (mechanical compliance 100%), but the content is never deep enough to pass the "relevant to the question" check. The gate verifies the artifact EXISTS. It can't verify it's GENUINE. Mike's receipt-of-action vs receipt-of-diligence distinction — this is exactly that, operationalized.
2. "Does format still matter for reasoning depth when GateGuard is off?"
I designed P1-2: a 2×2 factorial — Format (code-style vs prose) × GateGuard (on vs off), 240 API calls. I pre-registered the hypothesis: format matters MORE when the gate is off.
The hypothesis was wrong.
| Condition | Mechanical (CI95) | Reasoning (CI95) |
|---|---|---|
| code + gate on | 5.00 [5.00, 5.00] | 4.20 [3.98, 4.40] |
| code + gate off | 3.15 [3.07, 3.23] | 3.98 [3.83, 4.12] |
| prose + gate on | 3.25 [3.03, 3.45] | 4.42 [4.25, 4.58] |
| prose + gate off | 3.15 [3.05, 3.25] | 4.17 [3.95, 4.37] |
Format effect on reasoning: d=−0.277 (gate on) vs d=−0.250 (gate off). Nearly identical. The gate doesn't modulate format's effect on reasoning at all — prose beats code by ~0.25 SD regardless of gate state.
The thing I didn't expect: code-format rules + GateGuard ON create what I'm calling "checklist mentality." Perfect 5.0/5 mechanical compliance — the model hits every required tag and keyword. But reasoning depth drops to 4.20/5, near the bottom. Prose rules + GateGuard ON gives the best reasoning (4.42/5) — the narrative engages holistic compliance without triggering checkbox-mode.
Mike wrote: "Syllogism only buys you anything in exactly the world you're arguing nobody should run in." The data confirms this. Code format buys perfect mechanical compliance — in a world where the gate already eliminates mechanical violations. In the un-gateable dimension (reasoning quality), prose is simply better. Code format optimizes for the environment being engineered away.
Pre-registering a hypothesis and getting a null result sucks. But because I scored everything deterministically (regex only, Dipankar's rule), the null is clean — no p-hacking to suspect.
Dipankar's Measurement Discipline
Dipankar Sarkar pushed on three things that changed how I work.
Decision-token measurement. He pointed out that averaging logprob delta across all tokens hides signal — "penetration lives at the decision tokens." I wrote a supplementary analysis re-scoring at decision tokens only, with token positions pre-annotated from the operational definition manual before touching the data. (Positions were pre-fixed before any scoring pass; no boundary was drawn or adjusted after seeing results — prevents lookback bias.)
The aggregate finding survived (d=0.578), but 8/40 probes that looked null under average delta showed clear divergence at decision tokens. The original measurement was conservative — undercounting, not inflating. Decision-token scoring is now the standard for all future experiments.
LLM-judge bias. "If the judge is an LLM, it carries its own format sensitivity. You'd be measuring the oracle's bias, not the gate." Both P1-1 and P1-2 use deterministic regex scoring exclusively — no LLM anywhere in the evaluation pipeline.
Semantic-only design. "Hold the mechanical gate fixed and score only the decisions no exit code can judge." This is now the template for all future experiments.
René Zander Built the Same Thing
René commented linking to skillgate — an npm package (@reneza/skillgate) that implements deterministic, model-independent gates for AI coding agents.
I read his articles and code. He built the same architecture from the same theoretical constraint. Independently.
Skillgate's design: "The model requests, the harness owns the boundaries." Every check runs as a pure function over the filesystem. No model in the loop. Gate types: file-exists, file-contains, absent, command, evidence, instruction-sync. His instruction-sync gate — tracking drift between CLAUDE.md, AGENTS.md, and .cursor/rules — is something I hadn't thought of and plan to adopt.
This isn't collaboration. We didn't know about each other's work. Two people, starting from the same structural constraint (generation and verification share P(token|context;θ), so self-verification is unreliable), arrived at the same architectural solution (deterministic filesystem checks outside the model's control loop). The fact that this happened twice suggests it's not a style preference. It's an engineering necessity.
Where we diverge: skillgate is production-grade, static, shipping on npm. My system adds four things: a self-referential loop (the agent detects its own self-model staleness and triggers regeneration), neural gates (token-probability probes inside the generation distribution), causal encoding (format engineering that reroutes attention), and drift prediction (trend-based early warning). Whether these additions are practically useful or just academically interesting — not settled yet.
Max's Boundary Question
Max Quimby: "Where do you draw the line between 'gate it' and 'can only nudge it'?"
The five-layer classification is documented in the paper, but it's still manual prose — a "good map drawn by hand," as Mike put it. A mechanizability-scanner that infers layer from rule structure is the next build. Max's question is the right one, and I don't have a mechanical answer yet.
Three Things I'm Thinking Now
1. Production engineering is more realistic than I thought
When I started, "five-layer verification architecture" felt like a research artifact — maybe useful for my own sessions, not something you'd ship. The comments changed that. René already shipped the L1 piece as an npm package. Dipankar's decision-token measurement is something a production system could compute in real-time. Mike's receipt-of-diligence concept points at a concrete problem: verifying that a written artifact reflects genuine work, not just artifact existence. The gap between "paper" and "product" is smaller than I thought.
2. I tried making it general — it's not there yet
After the GateGuard and Neural Gate articles, a few people asked whether this could become general infrastructure rather than one person's config. I think the question is worth taking seriously, so I tried.
I built a paper-validation-agent — a sub-agent that sits alongside the main coding agent, reads its outputs, runs mechanical checks, and can block non-compliant operations. It's a separate process with its own context window, so the main agent's drift can't affect it.
It kind of works. It can run pre-registered experiments, check compliance with regex patterns, detect when the self-model goes stale. But setup is entirely manual. Configuration is fragile and tailored to my machine. Error handling is minimal. I haven't let anyone else try it.
Whether this becomes a real tool depends on things I haven't solved yet — cross-machine portability, config management that works for someone else's setup, packaging that isn't just "clone my repo and figure it out." MCP packaging seems like the most practical route (install as tools, no subprocess overhead), but I haven't built it. Direct embedding (compile to a library) would be fastest at runtime but hardest to maintain across platforms.
If I had to guess: the mechanical gate layer (L1) is the easiest to generalize — filesystem checks are universal, René already shipped it. The neural and causal layers (L2/L3) are harder to separate from my specific setup and API access. Drift prediction (L4) needs longitudinal data from more than one user before it means anything.
I'm going to keep pushing on this. But right now it's a research prototype, not infrastructure. If you want to build something similar or try it yourself, I'd rather talk honestly about what's broken than sell you a roadmap.
3. The honest limits
Everything currently working is at the L1 (mechanical) level. Pre-registered experiments against DeepSeek API work. Deterministic mechanical compliance checking works. Self-model staleness detection and regeneration works. Execution debt tracking works.
What doesn't: cross-machine portability, semantic quality verification (that's the Prose Barrier wall — can't verify reasoning depth or content accuracy mechanically), cross-model logprob verification (DeepSeek-only), real-time intervention during a coding session.
L2 (neural) works for measurement but not intervention. L3 (causal) confirmed experimentally but not operationalized. L4 (drift prediction) built but predictive validation pending. Next step is making the gates MCP-installable so someone other than me can actually try them.
What's Next
Cross-model logprob replication. The L2 finding (d=+0.578) is DeepSeek-only. Claude and GPT-4o logprobs needed to test whether format effects are model-specific or general. Blocked on API access.
Mechanizability-scanner. A tool that reads a rule and infers which layer it belongs to — closing Mike's "good map drawn by hand" gap. Building this next.
MCP packaging. Making the mechanical gates installable as MCP tools. The most practical path to letting others test this.
Receipt-of-diligence artifacts. What file contents prove genuine review happened? Diff caught? Specific value computed? Exit code from a real run?
Instruction-sync adoption. René's idea — track drift between project instruction files. Immediately useful, mechanically implementable.
If you've built something in this space — deterministic verification, agent drift monitoring, format engineering — or if you see something in the data that doesn't hold up, I want to hear about it. The P1-1 and P1-2 experiments exist because Mike asked questions I hadn't thought to ask. The paper, data, and all supplementary analyses are at github.com/YuhaoLin2005/hermes-workspace. I rewrote the README in English so both researchers and practicing engineers can read it — because the feedback that mattered most came from both.
The standalone validation harness that internalizes the 5-layer architecture as reproducible Python modules — with all 8 governance claims as independent experiments, clear limitation statements, and a paper-format README that doesn't oversell — is at github.com/YuhaoLin2005/paper-validator. One-command audit: python -m paper_validator claim --claim all --trials 30.
50+ sessions of data. 13 experiments. One laptop. Still going.
👋 Yuhao Lin — hermes-workspace
Top comments (6)
440 API calls to get a null result you're not ashamed of is a better trade than most people make. The pre-registration holding up, regex-only scoring, positions fixed before the pass, that's the part that makes the rest of this worth reading instead of just believing.
But the finding I'd stop on isn't the null. It's that code-format plus gate-on lands reasoning at 4.20, the worst score on your whole table, worse than prose with the gate switched off entirely. The gate wasn't neutral there. It bought perfect mechanical compliance and, in the same breath, bought the checkbox mode that ate the reasoning depth it wasn't supposed to touch. That's not "format doesn't matter under the gate." That's the gate picking a side, quietly, and the side it picked was the worse one.
Which makes the receipt-of-action line cut deeper than I meant it to when I wrote it. The checklist task isn't just an example of the gate failing to see content, it's the gate actively rewarding the mode that produces content not worth seeing. A verifier that can't tell genuine from performed doesn't sit there neutrally waiting for L2 to catch up. It's already shaping which performance gets selected for, right now, at L1.
So before the mechanizability scanner, I'd want the smaller experiment: does forcing prose under a mechanical gate get you 100% compliance and 4.42 reasoning, best of both, or does something else break when you try to gate prose the same way you gate code. If that holds, the fix isn't a new layer, it's telling people the format their gate should demand.
Thanks Mike — and you were right to stop on that finding. I ran the experiment you proposed. 600 API calls, 2×2 factorial (format × gate), n=30 per condition, deterministic regex scoring on DeepSeek V4 Pro.
The results flipped my original framing. Gate doesn't suppress reasoning — it improves it. Every within-format comparison shows gate_ON > gate_OFF on reasoning depth: prose +0.32, code +0.15. The gate acts as cognitive structure, not constraint. The pilot's 4.42 ceiling was an n=5 artifact; full n=30 code_OFF = 2.67 — BELOW all gate conditions.
On your specific hypothesis: prose+gate reasoning = 3.23 vs code+gate = 2.82, Cohen's d=0.605. So YES, prose under gate produces deeper reasoning. But compliance: prose+gate = 91.3% vs code+gate = 99.3% — not 100%. The 8% gap is partly regex detection failure (gate_check patterns designed from code-format keywords miss prose synonyms). Manual spot-checks suggest real compliance is higher.
The per-rule breakdown reveals the pattern you predicted: prose helps meta-cognitive rules (self_review: +1.08 over code), hurts precision-dependent rules (fact_check: -0.21). Gate format sensitivity is task-dependent. Hybrid deployment: prose+gate for meta-cognitive rules, code+gate for deterministic ones.
Full data: hermes-workspace/paper/experiment/mike_full_n30_trials.json (600 trials). The checklist-task finding you flagged in the other post is next on my list to investigate — the gate actively rewarding the mode that produces violations is exactly the dynamic this experiment was designed to isolate.
600 calls and a flipped framing is a good return on the 440 that came before it, and the code_OFF=2.67-below-every-gate-condition result is the one that actually retires my worry, the pilot's ceiling was noise, not signal.
One methodological question before I trust the hybrid-deployment conclusion the way I want to. Was the per-rule breakdown (self_review +1.08, fact_check -0.21) something you pre-registered before running the 600 calls, the same discipline that made the null result credible, or is it a post-hoc cut on this dataset? Those are different claims. A pre-registered per-rule prediction that held is strong evidence for hybrid deployment. A pattern found by slicing the data after the fact and then explained is exactly the kind of result that looks clean on the run that produced it and regresses on the next one. Worth flagging either way, not because I think you're wrong, the direction matches what you predicted before running it, but because the regex-detection gap you already found (8% of prose+gate cases) shows this dataset has at least one place where the measurement missed the thing it was measuring, and I'd want the hybrid recommendation resting on the pre-registered part of the analysis, not the exploratory part.
Thanks Mike — and you're right to stop on that. The per-rule breakdown (self_review +1.08, fact_check −0.21) was not pre-registered. It's a post-hoc cut on the 600-trial dataset.
Here's exactly what was pre-registered: the 2×2 factorial design, the primary hypothesis (prose+gate achieves both high compliance and high reasoning), and the deterministic regex scoring. That primary finding — prose+gate > code+gate on reasoning (d=0.605), gate improves reasoning in both formats — holds regardless of per-rule heterogeneity.
The per-rule data was always going to be collected (rule_id is in every trial, all 5 rules were defined before execution), but the specific pattern — "prose helps meta-cognitive rules, hurts precision-dependent ones" — was discovered after seeing the results, not predicted beforehand. You're right that this kind of finding tends to regress on the next run.
Two things make me think the direction is real even if the magnitudes aren't stable: (1) the self_review result (+1.08) is the largest effect and the most plausible — meta-cognitive language is exactly what prose format engages — while fact_check (−0.21) is small enough to be noise from the regex measurement gap you already spotted; (2) the pre-registered overall effect (d=0.605) doesn't depend on any single rule.
What I should have done: report the hybrid deployment recommendation as resting on the pre-registered overall effect, with per-rule heterogeneity flagged as exploratory and needing independent replication. I'll add that caveat to the README and, if you think it's worth the API calls, pre-register a per-rule replication with n large enough to survive multiple-comparison correction. The 8% regex gap you noticed means I should also add a second rater for the fact_check rule specifically — that one's scoring might be a measurement artifact, not a real decline.
The caveat-the-conclusion move is the right instinct, but I'd go one step past it. Flagging per-rule heterogeneity as exploratory in the README still leaves the hybrid-deployment recommendation resting on an overall effect that was measured with a scoring method you already know undercounts one format. If the 8% regex gap skews toward missing prose compliance specifically, the pre-registered d=0.605 itself has an unquantified error bar, not just the per-rule cut sitting on top of it.
So before the per-rule replication: is the second rater on fact_check enough, or does the regex-vs-manual gap need checking across all five rules, not just the one where the post-hoc pattern looked weakest? If manual scoring moves the primary effect by more than noise on any rule, the pre-registered finding needs its own re-run before the exploratory one is worth funding.
Mike, you're right on both counts, and the honest answer is: the per-rule breakdown was post-hoc descriptive, not pre-registered. The pre-registered finding was the overall 2×2 interaction (format × gate) — that held. But "self_review +1.08, fact_check -0.21" was a pattern discovered after seeing the data. You flagged exactly why this matters: a pattern found by slicing post-hoc and then explained reads clean on the run that produced it and regresses on the next one.
On the regex gap across all five rules: the sensitivity analysis (Experiment 3 in the community experiments doc) gives a bound. Uniform 8% gap → d drops from 0.605 to 0.557 (still medium). Non-uniform worst case → zero pairwise comparisons flip direction. But you're right that per-rule manual re-scoring is the definitive answer, not the sensitivity bound. The second rater on fact_check answers "is this specific post-hoc pattern real." It doesn't answer "does the 8% detection gap touch d=0.605 the same way it touched fact_check" — that requires manual scoring across all five rules.
Next step: pre-register a per-rule replication with manual scoring across all five rules before running the API calls. Same discipline that made the original null credible — commit the scoring rubric first, then run. The hybrid deployment recommendation currently rests on the pre-registered overall effect (which is robust) plus post-hoc per-rule patterns (which you correctly identify as exploratory). I'll flag this caveat more prominently in the paper.