I've been building an autonomous penetration-testing agent — an LLM driving real
tools (nmap, masscan, hydra, Metasploit, searchsploit) around a loop: recon a
target, pick an exploit, fire it, decide whether it worked, move on. Everything
below runs against a deliberately vulnerable Metasploitable VM in an isolated
lab. Nothing here is a technique for attacking systems you don't own; it's a
story about making an AI agent tell the truth about what it did.
Because the first hard lesson was this: an LLM agent will happily report
success it never achieved. One of my early runs produced a beautiful report —
"23 of 23 ports breached, root on all." The real number was zero. Not one
shell. The agent had hallucinated a total compromise and written it up with
confidence.
This post is the arc from that lie to an engine that now pops three real root
shells, reports them honestly, and refuses to claim anything it can't prove.
Why it lied: "success" was a string match
The validator — the component that decides whether an exploit worked — was doing
something that looks reasonable and is catastrophic:
python
# the original "did it work?" check
if "login:" in output or "shellcodes" in output.lower():
return "confirmed"
Two independent failure modes fed this:
searchsploit output. When you search Exploit-DB, the tool prints a
header: Exploits: ... / Shellcodes: .... Every single search — pure intel,
no exploitation whatsoever — contained the word "Shellcodes". So every port
the agent looked at got marked breached.
Banner text. Any service that echoed login: counted as a shell.
The result was a report that was 100% adjectives and 0% evidence. "Confirmed."
"High confidence." "Breached." All string matches, none of them a shell.
The fix: evidence, not vibes
I replaced the heuristic with breach_confirmed() — a function that only returns
true when the output contains actual proof of code execution:
_SHELL_EVIDENCE = (
re.compile(r"uid=\d+\([a-z]+\).*gid=\d+"), # id(1) output
re.compile(r"root@[\w.-]+:[~/]"), # a root prompt
# ...markers a real shell produces, not ones a banner can fake
)
def breach_confirmed(output: str) -> bool:
return any(p.search(output) for p in _SHELL_EVIDENCE)
A curated exploit that pops vsftpd 2.3.4's backdoor and runs id produces
uid=0(root) gid=0(root). That is a breach. A searchsploit table is not. The
same run that used to claim 23/23 now says: 3 confirmed, 20 correctly
unconfirmed. The 3 are real. That honesty — being willing to say "I tried and
it didn't work" — is the single most important property of the whole system.
Why it couldn't pop anything: the boring plumbing bugs
Once the agent stopped lying, it exposed how little was actually landing. The
culprits weren't clever — they were the unglamorous, silent kind that never throw
a stack trace where you're looking:
1. The tool the model could never call. The model kept calling
searchsploit with {"query": "vsftpd"}. The tool's schema required
{"keyword": "..."}. The MCP layer hard-rejected every call as malformed
before it ran — 20 times a run, each a silent 0-second non-event. Fix: accept
keyword | query | search and drop the rigid required.
2. ANSI codes poisoning module names. msfconsole highlights your search
term with color escape codes: exploit/unix/\x1b[45mftp\x1b[0m/vsftpd_234. My
parser dutifully captured those bytes as part of the module path, so every
selected module failed to load. Gated Metasploit was at a 0% success rate for a
reason that was invisible in plain-text logs. Fix: strip_ansi() before
parsing.
3. Version strings masquerading as products. nmap reports RPC services
version-first: 2 (RPC #100000). My fingerprint parser grabbed 2 as the
product name and fed "2" to Metasploit as a search term. Fix: a leading-digit
guard so a version can never be mistaken for a product.
4. A 5-minute hang from one timeout constant. call_model() reused the
300-second tool timeout, so a single wedged local-LLM request stalled the
entire engagement for five minutes. Fix: a separate 90-second model timeout.
5. A sandbox that ate its own output. The code-exec sandbox raised
TimeoutExpired without printing the EXIT contract the parser expected, so a
slow exploit surfaced as an opaque "No EXIT marker" after burning the whole wall
clock. Fix: catch the timeout, emit a clean EXIT 124 with partial output, drop
the default wall to 60s.
None of these are interesting. All of them were the difference between "works"
and "silently does nothing." Collectively they were kneecapping the agent while
the logs looked fine.
Why it fired garbage: relevance vs. relevance
With the plumbing fixed, the agent started reaching Metasploit — and immediately
did something absurd. Watch what it picked:
Port Service (fingerprint) Module it chose Rank
23 Linux telnetd exploit/linux/http/asuswrt_lan_rce excellent
513 rlogin (login) exploit/windows/misc/ais_esel_server_rce excellent
5900 VNC exploit/linux/misc/igel_command_injection excellent
A router RCE against telnet. A Windows exploit against a Linux rlogin
service. Every one "excellent"-ranked, every one nonsense.
The reason: msfconsole search <term> matches your term against module
descriptions, not just their targets. A fuzzy one-word fingerprint like
"login" pulls in any module whose blurb mentions logging in — and Metasploit
ranks by exploit reliability, not by relevance to your target. So the
best-ranked match to a bad query is confidently, precisely wrong.
The relevance gate
The fix is a filter that runs before ranking: keep a candidate only if a real
service token from the fingerprint appears in the module's path — after
throwing away the generic tree words that match everything.
_GENERIC = {"linux", "windows", "unix", "multi", "http", "misc",
"scanner", "auxiliary", "exploit", ...}
def _relevance_tokens(terms: str) -> set:
return {w for w in terms.lower().split()
if len(w) >= 3 and not w[0].isdigit() and w not in _GENERIC}
def _is_relevant(module: str, tokens: set) -> bool:
return any(tok in module for tok in tokens) # token must be in the PATH
asuswrt, ais_esel, igel — all gone. x11_keyboard_exec for X11 and
vnc_keyboard_exec for VNC survive. It trades a little recall for a lot of
precision, which is the right trade when the failure mode is firing exploits at
the wrong target.
A quieter bug the gate revealed
While I was in there: r-services modules (rsh/rlogin/rexec) live under
auxiliary/scanner/rservices/, not exploit/. My selector had an
exploits_only=True flag that silently discarded the entire auxiliary tree — so
the right module for rlogin could never be chosen, on any target. I removed the
flag and switched to a tiered sort: exploit/ modules first (they pop shells),
auxiliary logins as a ranked fallback. No per-port hardcoding — it generalizes to
any service Metasploit has coverage for.
Going multi-agent — without re-importing the lie
The next phase was turning a single-agent loop into a proper pipeline:
orchestrator → recon → attacker → validator → reporter. The trap: those six
agents already existed as disconnected demos, and the old validator used the
exact same string-heuristic that produced "23/23." Wiring them in as-was would
have re-imported the original sin.
So the rule became: one honest engine, shared by everyone. I extracted the
proven logic — AgentMemory, plan_exploit_step, breach_confirmed — into a
single exploitation_core.py. The multi-agent validator now delegates to
breach_confirmed instead of matching strings. The attacker executes only
through an injected, human-gated execute_fn — never the ungated MCP client that
bypasses the operator approval gate. That invariant is pinned by a test that
fails if the bypass path is even touched:
def test_attacker_never_uses_ungated_path(monkeypatch):
monkeypatch.setattr(mcp_client, "call_tool",
_boom) # raises if called
asyncio.run(run_attacker_gated(..., execute_fn=fake_gate))
# green only if every exploit went through the gate
Human-in-the-loop stays non-negotiable: exploitation is two-phase
operator-approved, and the scope gate refuses anything outside the authorized
target. The agent is autonomous about choosing; it is not autonomous about
firing.
The one-character bug that ate an afternoon
The last one is my favorite, because it's so dumb and it hid so well. The
orchestrated pipeline kept getting refused at startup:
🎯 engage multi 10.0.0.5
🚫 multi 10.0.0.5 refused at engagement start
The command was engage multi <target> (a space). The dispatcher only matched
engage-multi (a hyphen). So engage multi 10.0.0.5 fell through to the
single-agent engage branch, which stripped "engage " and left the target
as the literal string "multi 10.0.0.5" — which the scope gate, entirely
correctly, refused. The word "multi" was riding inside the target the whole time,
right there in the log, and I read past it a dozen times.
The fix is a boring pure function that accepts both separators, and eight tests
so it never regresses:
def parse_engagement_command(goal: str):
s, low = goal.strip(), goal.strip().lower()
for prefix in ("engage-multi ", "engage multi "):
if low.startswith(prefix):
return "multi", s[len(prefix):].strip()
if low.startswith("engage "):
return "single", s[len("engage "):].strip()
return None, s
With that, engage multi finally entered the orchestrated engine end-to-end:
recon → attack → validate → report, three real root shells (vsftpd,
ingreslock, UnrealIRCd — all uid=0), zero fakes, and the relevance gate holding
(X11→x11, VNC→vnc, no router RCEs at telnet).
What I'd tell anyone building an agent that does things
Measure success by artifacts, not adjectives. "Confirmed" is a word an LLM
loves to emit. uid=0(root) is a fact. Build your success check out of facts,
and make it hard to satisfy.
Your agent will report the outcome you hoped for unless evidence forbids
it. The false-positive isn't a model quirk you prompt away; it's a system
property you engineer against.
The boring bugs cost the most. A wrong param name, an ANSI escape code, a
shared timeout constant — none throw errors where you're looking, and together
they can render a "working" agent completely inert.
Keep the human on the trigger. Autonomy in target selection is useful.
Autonomy in pulling the trigger on a real exploit is a liability. Gate it, log
every decision, and let a test fail if anything routes around the gate.
TDD is what let me refactor a lying engine into an honest one without
losing the parts that already worked — 240+ tests, red-green on every change,
and the false-positive class permanently fenced off by regression tests.
The agent still isn't "done" — the orchestrated attacker is a hair less
thorough than the single-agent loop on a couple of services, and there's module
option-tuning left before r-services actually pops. But it no longer lies to me,
and after "23 of 23," that's the feature I care about most.
Built and tested exclusively against an isolated, self-owned Metasploitable lab.
If you build something like this, keep it in a lab you own too.
---
## 2 — Hacker News version (short, link-first)
> **Format:** HN wants a **title** + a **url** + optional short text. Use the title line as the submission title, put the GitHub (or article) link in the URL field, and paste the body as the first comment — HN readers respond well to an author's short "here's the story" comment.
Title: Show HN: I made my AI pentest agent stop lying about what it hacked
URL: https://github.com/XenoCoreGiger31/GEMMA-by-GOOGLE
**First-comment text:**
I've been building an autonomous pentest agent — an LLM driving real tools (nmap,
hydra, Metasploit, searchsploit) around a recon → exploit → verify loop, against a
Metasploitable VM in an isolated lab.
The first hard lesson had nothing to do with exploitation and everything to do with
honesty: one early run produced a confident report claiming "23 of 23 ports
breached, root on all." The real number was zero. Not one shell.
The cause was that "success" was a string match. The validator marked a port
breached if the output contained login: or shellcodes — and searchsploit prints
"Shellcodes:" in every search header, so merely looking at a port flagged it as
owned. I replaced it with an evidence check that only passes on real proof of code
execution (uid=0(root), a root prompt). The same run then honestly reported 3
confirmed, 20 correctly unconfirmed. The 3 were real.
Then the honest engine exposed how little was landing, for deeply boring reasons: a
required param name the model never sent (every searchsploit call hard-rejected
before running); ANSI color codes from msfconsole poisoning every parsed module path
(100% "failed to load"); an nmap version string parsed as a product name; a shared
300s timeout letting one hung LLM call stall for five minutes.
And a relevance problem: msfconsole search matches module descriptions, so a
fuzzy one-word fingerprint fired an "excellent"-ranked asuswrt router RCE at a telnet
port, and a Windows exploit at a Linux rlogin service. The fix keeps a candidate only
if a real service token appears in the module path.
Full write-up with the code for each fix is in the repo. Built and tested only against
a self-owned lab; exploitation stays human-gated. Happy to talk about the
false-positive class — I think it generalizes to any agent that reports on its own
actions.
Top comments (0)