I spend most of my time building an autonomous security agent — it recons a target I'm authorized to test, picks exploits, and tries to prove access, all against my own lab boxes. This week I rebuilt the brain of it, watched every test pass, ran it against a live target, and it popped nothing. Not because the model was dumb. Because my tests were quietly lying to me. AGAIN... so, how is this possible...AGAIN?
Here's the whole story, because the mistake is more useful than the feature.
The thing I built
The old design was a frozen plan. A planner decomposed the goal into an ordered list of subtasks once, up front, and an executor walked that list in order. That's fine until reality disagrees with the plan — a port you didn't expect, an exploit that fails, a service that isn't what the banner claimed. A frozen list can't react. It just marches.
So I replaced it with a closed loop. Every iteration, the policy gets the full current state plus everything that's already failed, and picks one next action. Act, fold the result back into the belief state, re-plan. The planner stopped being a one-shot decomposer and became a per-step policy.
The part I care about most isn't the loop — it's the teeth and the instrumentation, because an autonomous thing that can't stop or can't be measured is a liability, not a tool:
Termination teeth, checked before every action: an iteration budget, a wall-clock deadline, a kill switch, and a stall detector (belief state stopped changing → stop). Plus a repeat guard: if the policy re-emits an action it already tried, don't re-run it — count it as a repeat.
An adaptation score. Every re-plan logs whether the new action was genuinely novel or just a reshuffle of things already tried. Over a run you get novel / total. The whole point: measure whether the model actually adapts, instead of assuming it does. A number that can embarrass me is worth ten that flatter me.
Honest breach confirmation. Success is never the model's say-so. A breach only counts on real evidence, checked by code the model doesn't control. Don't let the system grade its own homework.
Tests: green across the board. Deploy. Run.
Zero shells
First real run against the lab target, the model reconned fine, then started working ports. And on the one port that should be the easiest win in the whole box — a service with a well-known backdoor, the kind of thing that pops every single time through the old path — the log said:
Nothing worked on port 21.
Zero breaches. Over sixteen iterations. And here's the part that saved me from blaming the wrong thing: the model had picked the right target. It correctly identified the vulnerable service and chose to attack it. Its judgment was fine. Something downstream ate the win.
The tests were encoding a fiction
The old, proven path confirms this exact exploit through a challenge–response nonce. The agent mints a single-use token, the exploit has to echo that token back inside a structured evidence line as proof it actually ran our command — not a banner, not a reflected string, an execution-derived fact bound to that one attempt. It's the anti-cheat that stops a tarpit or a reflector from faking a shell.
The new gated attack path… never minted the nonce. So the exploit fired, produced its nonce-bound proof, and the confirmer had nothing to check it against. No token in, no confirmation out. "Nothing worked."
But my tests were green. Why?
Because the tests fed the confirmer a raw uid=0(root) string and asserted "breach confirmed." That was true once — before the nonce hardening. After the hardening, the real exploit stopped emitting a bare uid=0 banner and started emitting the structured, nonce-bound line instead. The tests kept using the old fake evidence. So they exercised a code path that no longer matched reality, and passed with total confidence.
The unit tests weren't testing the system. They were testing a museum exhibit of the system.
The part that actually scared me
When I traced it, the same missing nonce logic sat in another execution path I'd considered working for weeks. It had "popped shells" in a demo run months earlier — before the hardening landed. Nobody re-checked it after. Its tests were green too, for the same fake-evidence reason. So a bug I thought was scoped to my new code had silently disabled breach confirmation across two paths, and my whole test suite waved it through.
One live run against a real target found what a green suite hid for weeks.
What I actually changed
Nothing exotic. The gated path now mints and injects the nonce like the proven path always did, and confirms against it. The downstream re-check couldn't re-consume a single-use token (that's the point of single-use), so it verifies the structured evidence artifact is present instead of demanding a raw banner. And I rewrote every test that had been feeding fake uid=0 strings so they prove a breach the way the real exploit actually does — through the nonce.
Then I reran it. Port 21 landed. Honestly, gated, end to end.
Three things I'm taking with me
A passing test proves your code matches your test, not that it matches reality. Mine matched a fixture that had quietly gone stale. If an assertion hard-codes what "success" looks like, that string is a second source of truth you now have to keep honest. Pin tests to the real artifact your system produces, or they'll rot into fiction while staying green.
A live target is an external judge your unit tests can't be. Everything about my design was aimed at not letting the agent grade its own homework — the nonce, the evidence check, the confirmer the model can't reach. But I'd let the test suite grade its own homework with fake evidence. The real box doesn't care what my fixtures assert. That's exactly why it's valuable.
Instrument for the answer you're afraid of. The adaptation score exists to tell me, out loud, if my local model just reshuffles instead of reasoning. I haven't gotten a clean number yet — the runs kept surfacing bugs first, and the model's slow enough that I'm capping generation length next. But I built the thing that can embarrass me before I built the thing that impresses me, and that ordering is the only reason today ended with a real shell instead of a fake one.
I didn't get the headline number I wanted this week. I got something better: a system that fails honestly, tests that finally match reality, and a very concrete reminder that "all green" is a claim about my tests, not about my code.
Zero shells is a better teacher than three shells you can't trust.
Everything here runs against targets I own and am authorized to test. If you're building agents that act on the world, build the part that can prove — or disprove — the claim before you build the part that makes it.
Top comments (0)