I finished migrating my agent's gate harness to a new framework. Files were on disk. Byte counts were sane. Markdown rendered. Every gate was "registered."
Then I actually ran them. None of them worked, and none of them ever had.
One caveat up front, because it matters for everything below: the original system runs on private data, so I can't share it. The claim in that last sentence is about my system, not about the repo you're about to see. Everything below comes from a minimal reconstruction I built to reproduce the same failure chain — public, no dependencies, Python 3.8+.
I'd already written about gates that look fine and aren't (38 gates, 87% noise) and about gates that stop working the moment you switch models. This is the migration-shaped instance of both, and the one I hadn't measured: the gate that never ran at all.
Quick context
My agent keeps a memory index with two things a gate cares about: a route table (which memory file to read for which kind of task) and resident blocks (content injected into context on every turn). A gate is a small script that checks the wiring is intact, for example that every route points at a file that exists.
After a migration, you'd expect to run the gates. I had them registered, and I assumed that meant they worked.
The reproduction
python harness/route_gate.before.py # rc=3 it cannot run at all
python harness/route_gate.py # rc=2 it runs, and reports RED
python harness/route_gate.fixed.py # rc=0 it runs, and reports GREEN
python harness/gate_inventory.py # the summary
These three files aren't three gates. They're one gate in three states: as migrated, with half its code fixed, and fully fixed. Here's the output of the last command — the real thing is three lines per gate, I've collapsed it to one:
registered 3 / compile-ok 3
route_gate.before.py compile: OK run: rc=3 PROBE_FAULT
route_gate.fixed.py compile: OK run: rc=0 verdict: GREEN
route_gate.py compile: OK run: rc=2 verdict: RED
Everything is registered. Everything compiles. Both numbers are 3, which reads like a clean bill of health. The exit codes are three different numbers. If your inventory stops at "registered and compiles," this is the report you get.
Three layers, each hidden behind the last
I expected one bug. I got three, and each only became visible after I fixed the previous one.
Layer 1: it doesn't run, and compile() says it does
route_gate.before.py passes compile(src, path, "exec") cleanly. A dangling name is a runtime fault, and no parser can see it. If your "can this gate run?" check is compile() or ast.parse, it will tell you this file is fine.
$ python harness/route_gate.before.py
PROBE_FAULT rc=3 -- NameError: name 'AGENT_MEM' is not defined
Layer 2: the fix you think is the fix is half of it
AGENT_MEM fails first only because it comes first. Define it and you hit the next name:
PROBE_FAULT rc=3 -- NameError: name 'MEMORY_ROOT' is not defined
The error message names one problem. The file had two. Any process that treats the first failure as the failure sends you around the loop twice. (I hit this one by hand while fixing; to reproduce it, delete MEMORY_ROOT = ROOT from route_gate.py.)
Layer 3: now it runs, and reports something you didn't expect
Define both names and it finally executes:
route_gate — is the route layer actually wired?
target : <repo>/memory/MEMORY.md
routes : 3 declared
resident: 0 blocks parsed
------------------------------------------------------------------
[FAIL] ROUTE_PATH_MISSING :: cases.md -> <repo>/arch/cases.md
[FAIL] NONVACUOUS_SECTION :: parser returned an empty set (routes=3, resident=0)
=> checks (1)(2) ran over nothing; this gate cannot discriminate
------------------------------------------------------------------
verdict: RED rc=2 (2 failure(s))
Two things matter here.
cases.md is reported missing. It isn't. It lives at memory/cases/cases.md, but the route table names it by bare filename. The file is real; the table is wrong.
And the resident parser found 0 blocks in a file that plainly has two. So the gate fails closed on itself: NONVACUOUS_SECTION refuses to claim green on checks it evaluated over an empty set.
That's the line I'd keep. A gate that has quietly stopped discriminating is worse than no gate, because it still prints a verdict.
Going green took five changes: three in code, two in data
Five changes separate rc=3 from rc=0. The code first:
- Two constants referenced but never defined —
AGENT_MEM,MEMORY_ROOT. - The resident parser expected
### N ·; the target writes## <section>plus1. item. - The provenance marks were ASCII-only (
=>); the target writes⇒.
Fixes 2 and 3 are the same failure in different clothes: the gate and its target had drifted apart, and nothing was watching for the drift. The gate dutifully reported "0 blocks parsed," then carried on. That's exactly why NONVACUOUS_SECTION has to fail closed.
And the last two are not in the code at all:
- The route table says
cases.md; the file is atcases/cases.md. Data. - The route table declares
report-v1.md; that file was never created. Data.
What I changed in my own rules
-
compile()is the entry ticket, not the pass. It catches syntax. It can't catch a dangling reference; only running can. - A "registered" count is not a number you can reconcile. Adding "and M of them compile" doesn't fix it either: in the demo both numbers are 3, which reads clean, while the real signal is the three different exit codes underneath.
- The first failure is not the failure. The error message names one problem; the file had two.
The takeaway is narrower than "migrations are hard": copying is not wiring. Files that arrive intact tell you nothing about whether anything executes them.
Limitations
- n = 1, and it's a reconstruction. The original ran against private data and isn't public. Every number above comes from the reproduction repo, not from the original system.
- The gate is deliberately narrow: path existence, provenance marks, and a fail-closed self-check. It's a demonstration, not a framework.
- The repo runs the complete arc,
rc=3 → rc=2 → rc=0. What I can't give you is a before/after number for the original system, because fixing it there is still in progress and I'd be reporting a number I haven't earned.
Try it
git clone https://github.com/YuhaoLin2005/agent-migrate-demo
cd agent-migrate-demo
python harness/gate_inventory.py
If you run it, I'd like to know which layer you hit first. My guess is Layer 1, because almost every "is this script runnable" check I've seen stops at compile().
agent-migrate-demo: the three-layer failure chain, clone and run
GitHub
Top comments (0)