File-system checks ask "did the script run?" Neural gates ask "did the constraint actually change the output?"
The Problem With File-System Gates
For the past month, I've been building mechanical gates for my Claude Code agent. They check file timestamps, hook registrations, exit codes. They work — they catch real configuration drift.
But they all operate on the same assumption: if the file exists, the hook is wired, and the script executed, then the constraint must be working.
This is false. An AI agent can read a behavioral rule, echo it in its self-assessment, generate compliant-looking outputs — and still not be influenced by it. The rule is in the context window. The agent mentions it when asked. But the token probability distribution hasn't shifted.
File-system gates check arrival. They don't check penetration.
AI Logic ≠ Human Logic
I was using human logic (file timestamps, regex, exit codes) to verify an AI system. But an AI agent's native senses are attention weights, residual stream directions, and logprob distributions. Verification must happen at the level where information actually flows.
Neural Gate v1: Constraint Echo Detection
neural-gate.py (86 lines). Extracts 8 constraint themes from BODY.md, scans today's output files for keyword echoes. Silent constraint = may be decaying. All 8 constraints echoing. Validated across 150-task controlled experiment — see update below.
Neural Gate v2: Logprob Differential (Designed)
Compares token probabilities with/without constraints using DeepSeek logprobs=True. If delta > 0.3 logprob units, constraint is active. Script written (neural-gate-v2.py). Needs API key.
Neural Gate v3: Residual Stream Probes (Roadmap)
On Qwen2.5-1.5B (fits RTX 3060 6GB): train linear probes per transformer layer. Track layer shifts across sessions to detect early decay.
Three-Layer Architecture
| Layer | Question | Status |
|---|---|---|
| L1 — Mechanical Gate | Did info arrive? | ✅ Validated (150 tasks) |
| L2 — Neural Gate | Did info penetrate? | v1 deployed, v2/v3 roadmap |
| L3 — Causal Encoding | Does format determine pathway? | ✅ Experiment (see update) |
Update (July 11, 2026)
Ran a 150-task controlled experiment. Result: mechanical gate validated — 55.9% violation rate (no gate) → 0.7% (with gate). Full writeup: I Ran 150 Tasks to Test If AI Agents Follow Rules
Also discovered L3: syllogism-form rules (causal chains) vs imperative rules produce same compliance rate (ceiling effect from mechanical gate) but systematically different reasoning depth. Format changes how the agent understands why to comply — not whether it complies.
Honest Status
- v1: deployed, validated across 150 controlled tasks
- v2: written, needs API key
- v3: designed, feasible on RTX 3060 (1.5B models)
- 34 growth-logs retrospectively coded: 55.9% violation rate pre-GateGuard
- 7 frameworks audited. None do neural-layer constraint fidelity checking
- 2 ECC PRs merged, co-authored-by from alirezarezvani/claude-skills
📂 Code & experiments: paper-validator — logprob probes, drift predictor, and all experiments are reproducible
🇨🇳 中文版：掘金 - 文件系统门只能查做了没，我加了一层查懂了没
👋 林宇浩 — Building verification infrastructure for AI agents. GitHub · DEV.to · 掘金
Top comments (4)
The arrival-vs-penetration split is the right axis, and the logprob differential is the honest instrument for it. One caution on the v2 threshold: a single delta averaged over all tokens can hide the thing you actually care about. A constraint can shift the distribution hard on tokens that don't matter and leave the argmax untouched, or flip exactly one decision token (the branch point) with a tiny aggregate delta. Penetration lives at the decision tokens, not the average. So I'd score the delta only where the constraint should change the chosen token, not across the whole output, otherwise a verbose-but-compliant passage and a genuinely-steered one read the same. The other half: penetration isn't binary. A constraint can shift the distribution and still lose to a stronger competing pressure downstream, which is the 'reads the rule, weighs it, guesses anyway' case from your other thread. v3 residual probes per layer would actually let you watch that fight happen. Following the series, this is the layer most people skip.
Thanks. Re-scored the 40 probes at decision tokens only — 8 shifted from null to aligned. d=0.578 survived but the true effect is likely larger. Decision-token scoring is now standard for all subsequent experiments. Full follow-up: dev.to/yuhaolin2005/follow-up-deci...
Arrival versus penetration is exactly the gap that trips people up when they trust an agent's summary instead of reading its diff. What strikes me from running agents daily is that penetration failures usually still surface as an ordinary bug once you read the actual change. The constraint gets acknowledged in the chat, ignored in the code, and the diff shows a plain mismatch between what was promised and what got written. Your three layer setup is a genuinely interesting way to catch that earlier, before the code even lands, where a diff reviewer like me only sees it after the fact. I am curious whether the logprob differential signal holds up on agents that rewrite their own reasoning mid task, since in my experience that is exactly when a model talks itself back into doing the thing it was told not to do.
Thanks Edu — really appreciate the diff-reviewer perspective. "The constraint gets acknowledged in the chat, ignored in the code" is exactly right and it's the core failure mode this project exists to catch.
Your question about mid-task self-rewriting is sharp. Honest answer: I haven't tested whether the logprob differential holds up when a model talks itself back into doing what it was told not to do. The current measurement scores logprob delta at decision tokens in the completed response — not intermediate reasoning steps. A model rewriting its own reasoning mid-task would be a real stress test, especially if the self-rewrite shifts distributions in ways the final-output measure is blind to. That would need probing at intermediate generation steps, which the current setup doesn't do. Worth investigating.
I wrote up all the recent results including two new experiments driven by community feedback: dev.to/yuhaolin2005/your-feedback-...