Thanks to Dipankar Sarkar, Mike Czerwinski, Max Quimby, and Ponsubash Raj R for the detailed comments on the GateGuard and Neural Gate articles. This post describes what I changed based on that feedback and what the results were.
1. Decision-Token Delta: From Average to Branch Points
⚠️ Methodological note (added 2026-07-13): Decision-token position annotations were pre-fixed from the operational definition manual before any re-scoring pass. No boundary was drawn or adjusted after seeing the data. The classifier ran once, with frozen annotation positions, against the existing 40 probes. This prevents lookback bias — the measurement was a re-scoring pass with pre-registered token positions, not a post-hoc boundary fitting exercise.
Feedback (Dipankar Sarkar): Measuring logprob differential averaged across the full output misses the signal. "Penetration lives at the decision tokens, not the average. A constraint can shift the distribution hard on tokens that don't matter and leave the argmax untouched, or flip exactly one decision token with a tiny aggregate delta."
What I did: Wrote bridge-decision-token.md — a supplementary analysis that re-scored the original 40 probes at decision tokens only (the token positions where a constraint should change what gets chosen, pre-annotated from the operational definition manual before re-scoring). Dropped filler-token positions from the aggregate.
Result: The decision-token-only measurement changed individual probe scores but did not flip the overall finding (d=0.578, 32/40 probes aligned). The aggregate effect survived re-measurement. But 8 probes that looked like "no effect" under average delta showed clear divergence at decision tokens — the signal was there but diluted by filler-token noise. This means the original measurement was conservative (undercounting effect), not inflated.
Conclusion: Scoring at decision tokens is the correct measurement and will be the standard for all subsequent experiments. The original finding survives, but the 8 probes that shifted from null to aligned suggest the true effect may be larger than d=0.578.
2. Measurement Boundary: The Follow-Up Experiment
⚠️ Status: Experimental Design Only — Not Yet Executed (added 2026-07-13). This section describes the design of a planned experiment. No API calls have been run. No data has been collected. The probes and scoring rubric are built and pre-registered; the experiment itself has not been executed. Please read this section as "here's what we plan to test and how," not "here's what we found." Results will be published in a follow-up once the experiment completes.
Feedback (Dipankar Sarkar): The ceiling effect isn't a null result — it's a measurement boundary. GateGuard fully covers the mechanical class. Format effects, if they exist, only appear in the un-gateable semantic space. "The sharper next run holds the mechanical gate fixed and scores only the decisions no exit code can judge."
What I did: Designed experiment P1 (L2→L3 neural gate). The spec:
- Hold GateGuard fixed (all mechanical checks active)
- Two format conditions (syllogism vs imperative)
- Score only semantic decisions: approach selection, trade-off justification, risk acknowledgment, uncertainty expression — decisions where no exit code can judge correctness
- Created 12 multi-position probes targeting semantic-decision tokens across 5 task types
Status: Experiment spec and probes are built — not yet run. The key design constraint: probes must test decisions the agent makes after passing mechanical gates, in a space where the model's own distribution is the last line of defense.
3. Format Is Fallback: Paper A → Paper B
Feedback (Mike Czerwinski): "Format optimization is optimizing for the environment you're trying to engineer away, which is either an argument that it doesn't matter, or an argument that the gate can't be everywhere and format is your fallback for the gaps. Worth deciding which, because they point at different papers."
What I did: Re-framed the paper's core claim. Previously: "Format doesn't matter — mechanical gates dominate" (Paper A). Now: "Format matters exactly where gates can't reach — those gaps are structural, not temporary" (Paper B). Updated PAPER.md and README.md to state this explicitly.
Conclusion: The ACL submission was answering the easier question. The harder question — does format change behavior in the un-gateable decision space — is the experiment designed in Section 2.
4. Mechanizability Gradient: From Binary to Spectrum
Feedback (Max Quimby): "Where do you draw the line between 'gate it' and 'can only nudge it'?"
What I did: Documented the five-layer architecture (L0 psych safety → L1 mechanical gate → L2 neural probe → L3 causal route → L4 drift prediction) with systematic classification: does the rule operate on files? (L1) → on token distributions? (L2) → on decisions? (L3) → on patterns over time? (L4). Wrote DECISION-TREE.md for structured rule-to-layer assignment.
Conclusion: The line isn't one line — it's a gradient. But classification is still manual (L0 prose). A mechanizability-scanner.py that classifies rules structurally is the next step, not yet built.
5. Sensitivity: Boundary Probe Reclassification
What I did: As a robustness check on the L2/L3 divergence claims, reclassified all boundary probes (probes where the constraint's mechanizability tier was ambiguous between L2 and L3). Re-ran the analysis with probes shifted one tier in each direction.
Result: The divergence pattern held under both reclassifications. The L2/L3 distinction is not an artifact of probe classification ambiguity. Updated PAPER.md §4.2 with this sensitivity result.
What's Next
- Run P1 experiment (GateGuard-fixed, semantic-decision-only scoring)
- Build review-artifact-guard.py (receipt-of-diligence check)
- Build mechanizability-scanner.py (rule-to-layer classifier)
github.com/YuhaoLin2005/hermes-workspace — verification infrastructure for AI agents. 50+ sessions of data. Seeking summer 2026 internship.
Top comments (8)
The Paper A → Paper B reframe is the right call, and it's worth being precise about what still needs to hold before it earns the name. Right now Section 3 states the new claim, Section 2 designs the experiment that would test it, and P1 hasn't run yet. That's an honest sequence, but it means the post is currently reporting a reframe, not a result, and the two read very differently to someone skimming for findings.
The decision-token result in Section 1 is the one I'd sit with longest before accepting it at face value. Eight probes moving from null to aligned under a stricter measurement is being read as "the original was conservative," but that's one of two possible readings. The other is that decision-token boundaries were drawn after seeing which tokens would flip the result, which turns a measurement refinement into a lookback-biased one. Worth stating plainly whether the decision-token annotations were fixed before this re-scoring pass or adjusted during it, because that's the difference between undercounting and overfitting the finding.
Section 4 is the one I'd actually watch. Mechanizability classification is still manual prose (L0), and manual classification is exactly the kind of self-report that was under scrutiny elsewhere in this thread this week, declaring a rule's layer rather than deriving it from what the rule actually checks. A mechanizability-scanner that infers layer from rule structure would be the moment this stops being judgment-call taxonomy and becomes something checkable. Until it exists, the five-layer architecture is a good map drawn by hand, not yet a thing the system can verify about itself.
Thanks Mike. Three updates.
On the reframe-vs-result: agreed, and that's exactly why I ran the experiments. P1-1 (n=200, residual violation clustering) and P1-2 (n=240, 2×2 GateGuard×Format) are both done now — deterministic regex scoring, results in the retrospective article. The Follow-Up post was written before they ran; the new one has data, not just design.
On decision-token annotations: they were pre-fixed from the operational definition manual before any re-scoring pass. No boundary was drawn or adjusted after seeing results. The classifier ran once against the existing 40 probes with frozen annotation positions. Same discipline you asked for — just wasn't stated clearly enough in Section 1.
On mechanizability: "a good map drawn by hand, not yet a thing the system can verify about itself" is exactly right. Building the scanner next. Until then, the five-layer classification remains L0 prose — no argument there.
Full results + data: dev.to/yuhaolin2005/your-feedback-...
The Follow-Up-was-design, retrospective-is-data distinction matters more than a formality, it's the difference between a post that could have been wrong on paper and one that's already been checked against 440 calls. Good that it's stated plainly rather than left for a reader to assume the design held.
Pre-fixed decision-token positions closes exactly the gap I was pointing at, the classifier running once against frozen annotations is the thing that makes "the original was conservative" a claim you get to keep instead of a claim someone gets to poke a hole in later. Worth putting that one sentence you just wrote here into Section 1 of the post itself, "wasn't stated clearly enough" is doing real work as a caveat, but it's cheap insurance against the next reader assuming the boundary was drawn after the fact.
The P1 design has one load-bearing risk worth pinning before you run it: the scorer for the semantic decisions.
Once GateGuard holds the mechanical class fixed, approach-selection and trade-off-justification have to be judged by something. If that judge is an LLM, it carries its own format sensitivity. A syllogism-vs-imperative delta could then be the judge reacting to format, not the generator's decisions changing. You would be measuring the oracle's bias, not the gate.
Two ways to keep it clean. Score with a rubric that reads only the choice made (which approach, did it name the trade-off, did it flag uncertainty) as discrete yes/no items a human or a fixed non-generative classifier can check, not a quality score. And strip format from the judge's view: normalize both arms to the same surface form before scoring so the judge can't tell which condition it is in.
The decision-token pre-annotation from the definition manual already closes the forking-paths door on section 1. Same discipline on P1's oracle and the L2->L3 result will actually mean something.
Thanks Dipankar — this was the most important comment to get right, and you'll see your fingerprints all over the follow-up design.
Both P1-1 and P1-2 used deterministic regex scoring. No LLM judge anywhere in the evaluation pipeline. The scoring rubric reads discrete yes/no items exactly as you specified: which approach was selected, was the trade-off named, was uncertainty flagged. Each scored by regex pattern match, not by any generative model. Your warning about measuring the oracle's bias instead of the gate is now a hard constraint in the experiment spec — I cite it directly.
Both experiments are done. P1-1 (n=200, 5 task types × 40 trials): violations cluster exactly where the gate can't reach — 100% compliance on mechanizable tasks, 0–42.5% on semantic tasks. P1-2 (n=240, 2×2 GateGuard×Format): pre-registered hypothesis was wrong (format effect is constant regardless of gate status), but the null is clean because of deterministic scoring. Prose-format rules consistently outperform code-format for reasoning depth (~0.25 SD), independent of gate.
Full results + data + scoring scripts: dev.to/yuhaolin2005/your-feedback-...
Your "normalize both arms to the same surface form before scoring" point — I haven't implemented format stripping yet. The current scoring regex patterns work on raw output, so the scorer can in principle see format differences even though it's checking discrete items. That's the next refinement. But using regex instead of an LLM judge already eliminates the biggest source of format-sensitive bias you identified.
Interesting discussion. I like how the author focuses on measuring the actual impact at decision points instead of relying only on aggregated metrics. Small changes can sometimes have very different effects depending on where they happen in the pipeline.
Thanks CodeKitHub — really appreciate you reading through the discussion.
You're right that measuring at decision points vs aggregate metrics makes all the difference. That reframe came directly from Dipankar Sarkar's comment on the earlier article — "penetration lives at the decision tokens, not the average." Once you see it, it's obvious, but I needed someone to point it out.
The follow-up experiments (P1-1 and P1-2, 440 API calls total) all use decision-point measurement now as standard. More results + data here: dev.to/yuhaolin2005/your-feedback-...
Thanks for the detailed follow-up!
I really like how you turned the idea into actual experiments instead of just discussing the concept. Measuring impact at decision points feels much closer to how systems actually behave.
The quote about “penetration lives at the decision tokens, not the average” is a great way to think about it. Averages can hide where the real changes happen.
Looking forward to seeing more results from your experiments. Thanks for sharing your work!