I spent a whole day this week making my autonomous pentesting agent slower. On purpose, sort of. Here's how that happened, and why it turned into the most honest day of building I've had in a while.
Some context: I'm ~6 months into teaching myself to code, and I'm building HALO — a local, self-owned pentesting agent driven by a small model running on my own hardware. No cloud API, no $475/yr scanner subscription. The goal isn't to pop the easy boxes. It's to build something that hunts the bug after the easy three, on targets that aren't gift-wrapped.
The fix that didn't exist
The day started with a "speed fix." My model calls were taking ~90 seconds each, and an engagement was crawling. The obvious move: cap the model's output tokens so it stops rambling. Small change. Ship it.
It made everything worse.
The runs got faster and started failing completely — the agent would find all the open ports and then just... give up on every one. When I finally pulled the actual logs instead of guessing, the truth was ugly and simple:
The model I'm running is a reasoning model. It burns ~1,200 tokens thinking in a hidden channel before it writes a single token of the answer I actually need. My token cap was set below the thinking budget. So the model would think, hit the ceiling mid-thought, and return an empty string. Every "smart" cap I added was decapitating the answer.
The kicker: capping tokens can't speed up a reasoning model at all. The slowness was just... the model. On my hardware, that's the floor. There was no speed fix. I'd spent hours optimizing a lever that doesn't exist.
Lesson one: read the logs before you theorize. I knew this. I did it anyway. The receipts were 30 seconds away the whole time.
The part I'm actually proud of
Once the agent could think again, I watched it work a lab box (a deliberately-vulnerable Metasploitable VM — the standard punching bag). And here's the thing that made the whole day worth it.
The agent has three curated exploits it knows work. It fired them and came back with this:
HALO-EVIDENCE nonce=1a837238e19fc8fb56014af0 level=shell uid=0 user=root host=metasploitable
root@metasploitable:/#
That nonce is the whole point. My orchestrator mints a fresh random string for each attempt, and the only way that string shows up in the output is if a real shell on the target actually ran the payload and echoed it back. The exploit script can't fake it. The model can't claim it. A banner that says "root" doesn't count. The system cannot grade its own homework — it has to produce a receipt minted by something outside itself.
This is the idea you and I keep circling back to in the comments here: don't trust a component's self-report. Make it prove the claim with something it couldn't have forged. My agent popped three genuine root shells, and I believe it because I made it hard for it to lie to me.
And then it fired at everything else — and told me it failed
Here's the honest ending. Beyond the three curated exploits, I'd just wired up a Metasploit pipeline so the agent could reach for the ~2,000 exploit modules Metasploit knows. Watching it run against the other 20 open ports was humbling:
It fired an Apache 2.4.49 exploit at Apache 2.2.8. (The name matched. The version was nonsense.)
It picked heavyweight staged payloads where a dead-simple command shell would've been more reliable.
It reached for a MySQL exploit that needs credentials it didn't have.
Every single one of those came back "Nothing worked on port X." Not a fake success. Not a padded number. The exact same nonce gate that confirmed the real root shells refused to confirm the ones that didn't land.
That's the feature. A tool that lies to you about coverage is worse than useless — it's dangerous. Mine popped three and honestly told me the other twenty didn't work, and why, in the logs. That "why" is my whole to-do list for next week.
Where I actually landed
When I committed the day's work, I made myself write the honest version of the message. Not "working Metasploit path." The real one:
Curated PoCs pop nonce-verified root reliably; the MSF path fires end-to-end but session-landing refinement is still pending.
Both halves of that sentence are true, and I only know they're true because the system won't let me round up.
Two kinds of honesty came out of one day:
The agent's — it can't claim a breach without a receipt it couldn't forge.
Mine — I committed what's real and labeled what isn't, instead of shipping a hype commit.
They're the same principle, pointed in two directions. If I'm going to build something I actually trust to point at a target, both have to hold.
Still a long way from the thing I want. But it's a long way built on receipts instead of vibes — and after a day that started by optimizing an imaginary problem, I'll take it.
What's the ugliest "the system was grading its own homework" bug you've caught in your own builds? I'm collecting them.
Top comments (0)