The hardest part of an autonomous exploitation engine isn't landing a shell. It's knowing you landed one.
Point an LLM-driven agent at a target and it will cheerfully report 23/23 ports popped — while the real number is zero. Why? Because a service banner that prints uid=0(root), or a tarpit that streams believable output on connect, looks exactly like success to anything doing substring matching. Optimism is the default failure mode of automated offense.
This is the story of the pushes I merged today into HALO, an authorization-gated autonomous pentest engine, and the one idea that fixed the trust problem: make the target prove it ran your code.
The false-positive trap
The naive breach check is a keyword search:
def breached(output: str) -> bool:
return "uid=" in output or "root@" in output
Every one of those tokens can be forged by a target that never actually executed anything. Worse, once your success heuristic is wrong, every downstream decision inherits the lie — the report, the "which ports are owned" state, the next exploit chosen. You end up with a confident engine that's confidently wrong.
The fix: a challenge–response nonce
Borrow the oldest trick in authentication. Before each attempt, the orchestrator mints an unpredictable token and injects it as an environment variable. The exploit is required to make the target echo that exact token back:
import os
def make_nonce() -> str:
# unpredictable, so a target can't guess it
return os.urandom(12).hex()
The delivery payload announces the nonce over the channel it opens, and the breach gate only trusts a hit that carries a matching structured line:
HALO-EVIDENCE nonce=9f2c1a7b4e level=shell uid=0 host=victim exit=0
_EVIDENCE = re.compile(r"HALO-EVIDENCE nonce=(\S+) level=(\S+)")
def breach_confirmed(output: str, ok: bool, *, nonce: str) -> bool:
m = _EVIDENCE.search(output or "")
return bool(ok and m and m.group(1) == nonce)
A tarpit streaming uid=0(root) cannot produce a line with this run's nonce — it never received our command. Fail closed, and false positives evaporate. The gate is now anchored to something the target can't fabricate.
A target-agnostic delivery ladder
Proof is worthless if you can't deliver a payload in the first place, and real hosts are inconsistent — different interpreters, different egress rules. So delivery is a ladder that degrades gracefully, all stdlib socket:
Reverse shell — target dials back to an ephemeral listener; announces the nonce, hands back /bin/sh.
Bind shell — if egress is blocked, the target binds a shell and we connect in.
Blind callback — if no interactive channel survives, the target just connects back and sends the nonce alone. That still proves code execution, even with no usable shell.
Each rung self-selects the first available interpreter (bash /dev/tcp, python3, perl, nc), so the same primitive works whether you're pointed at 192.0.2.10 or 192.0.2.200. First confirmed rung wins; nothing box-specific.
Shipping a self-contained exploit into a locked box
Curated exploits run inside a hardened podman sandbox — --read-only, --cap-drop=ALL, memory/pids capped, and only the single PoC file mounted. That last constraint bit me: the refactor had each PoC import a shared delivery module... which doesn't exist inside a one-file mount. Tests passed (they import from the repo root); the real sandbox threw ModuleNotFoundError.
The fix was ship-time bundling: inline the shared module's source into the shipped file, strip the sibling import, hoist a single from future to the top. The repo stays DRY; the artifact that actually runs is self-contained. The regression test that caught it runs the bundle in a subprocess with the package deliberately off the path — so it would've been red before the fix and green after. Proof over vibes, again.
How it was built: subagent-driven TDD
The whole feature shipped through a disciplined loop: one fresh agent implements a task test-first, a second agent reviews it against the spec, findings get fixed and re-reviewed, then a final whole-branch review sweeps for cross-task seams. That last pass caught a real one — an error path in one PoC that could throw instead of failing closed — plus flagged an unquoted shell splice worth hardening with shlex.quote. Small stuff, but it's the stuff that bites in production.
Final tally: 283 tests green, every task reviewed, no "close enough."
Bonus lesson: scrub before you go public
Prepping the repo for release, a git grep across tracked files turned up real network addresses and host identifiers baked into fixtures and internal planning docs — and worse, some were already in the pushed history. A clean working tree doesn't help; git never forgets.
The remediation: rewrite to a single clean commit and force-push, after replacing real values with RFC 5737 documentation IPs (192.0.2.0/24, 198.51.100.0/24) — the range that exists specifically so examples never collide with real infrastructure. If you write security tooling, adopt that habit now: doc-range IPs in every fixture, secrets and scope files git-ignored from commit #1.
Takeaways
Don't trust output — trust a token you minted. Challenge–response turns "it said uid=0" into "it echoed my nonce."
Deliver on a ladder, not a guess. Reverse → bind → blind-callback covers the messy reality of real hosts.
Test the artifact you actually ship, in the environment it actually runs in.
Scrub with git grep, not with hope — and use RFC 5737 IPs so there's nothing to scrub next time.
Proof beats optimism. Build the gate first.
Top comments (2)
This is a strong fix for accidental false positives. I’d narrow one claim, though: because the nonce is present in the delivered payload, a reflective or deliberately adversarial service can return it without executing the command. The nonce proves freshness under a non-reflecting threat model; it is not remote attestation.
I’d bind each nonce to
{attempt_id, target, payload_hash, expected channel, expiry}, consume it once, and reject duplicates or cross-attempt callbacks. The evidence parser should accept exactly one structured frame from the expected channel and preserve a hash of the raw transcript/listener metadata. A computation over the challenge plus an execution-derived fact is stronger than literal echo for defeating simple reflection.Useful negative tests: reflected payloads, replayed/delayed callbacks, two attempts racing, truncated frames, nonce from the wrong target, and shell success with lower privilege than claimed. Also separate “code executed,” “uid verified,” and “interactive channel usable” as distinct evidence levels.
Thanks. Ill take your advice and see what it produces. I believe you are actually on to something and I made a mis-claim in my previous article. Im about to sort of update that and actually put into action your suggestion. Thank you. Keep your eyes on the screen Im about to post a live shot of it in action. Abeit, with a bit of an updated re-write-ish. I decided to go for it...Spot on, and thank you — you're right that this is freshness under a
non-reflecting model, not attestation, and literal echo falls to a reflective
service. I'm adopting your binding: nonce ⇄ {attempt_id, target, payload_hash,
channel, expiry}, consume-once with replay/cross-attempt rejection, and the
parser accepting exactly one structured frame from the expected channel while
hashing the raw transcript + listener metadata. Moving from echo to a
computation over the challenge combined with an execution-derived fact. And
splitting evidence into "code executed" / "uid verified" / "interactive channel
usable" as separate levels. Adding your negative tests wholesale: reflection,
replay/delay, racing attempts, truncated frames, wrong-target nonce, and
success-at-lower-privilege-than-claimed. This is going straight into the design.🫡