Pydantic put up a $10,000 bounty called Hack Monty: escape the sandbox of their Monty runtime. That is a clean, adversarial, unforgiving target. Either you get a sandbox escape or you do not. No partial credit, no hand-waving.
I did not try to hack it by hand. I built an autonomous LLM loop to hack it, and then I rebuilt that loop four times as I learned what did and did not work. This is the honest write-up of what an agent swarm actually found, which is not a $10k escape, and why the result is still worth publishing.
The core idea
An autonomous hacking loop is a simple pattern with hard details. An LLM proposes an exploit, a harness runs it against the target, an evaluator scores the result, and that score feeds back to steer the next attempt. Repeat for hundreds of iterations.
The engine here started on the autoresearch pattern and evolved through four versions: from a single loop, to XBOW-style parallel swarms, and finally to a tokenworm plus MCP architecture. Each version was a bet about what the bottleneck was. Early on it was idea diversity, so I went parallel. Later it was tool discipline, so I moved to MCP with a tightly-scoped boundary.
How V4 works
The final architecture is three layers with a hard boundary in the middle.
At the top, tokenworm is the LLM agent harness. It runs a native Ollama provider, four role-focused SKILL.md files, and its own sandbox (bwrap plus network). It speaks to the outside world only through an MCP client over stdio.
tokenworm (LLM agent harness)
Provider: ollama_native
Skills: 4 role-focused SKILL.md
Sandbox: bwrap + network
│ MCP Client (stdio)
▼
hackmonty_mcp_server.py
17 boundary tools (run, evaluate, bandit, state)
│
▼
hackmonty.com Ollama Cloud GitHub Issues Filesystem
The middle layer, hackmonty_mcp_server.py, is the interesting design decision. It exposes exactly 17 tools and nothing else. The agent cannot touch the target, run code, score an attempt, or read state except through those 17 tools. That boundary is the whole safety and control model. It is what lets an autonomous loop run 500 iterations without the harness itself becoming the vulnerability.
Two more pieces make the loop learn rather than flail. A UCB1 bandit (bandit.py) selects which attack strategy to try next, treating exploit families as arms and their scores as rewards, so the loop spends its budget on what is paying off. And an evaluator (evaluate.py) scores each attempt 0 to 5 and enriches the context, so the next proposal is grounded in what the last one actually did.
Running it is a shell script or the Python SDK:
# Production (tokenworm Zig binary), needs ollama serve running
export USER_SECRET=your-passphrase
./run.sh # 500 iterations
./run.sh 20 # 20 iterations (test)
./run.sh -i # interactive REPL
# Python SDK showcase (InlineSkill, MCP, native Ollama Cloud)
uv run python run.py 20
There is also an SSE remote mode for the MCP server: uv run python -B hackmonty_mcp_server.py --sse --port 8765.
The results, honestly
Here is where I refuse to inflate anything, because the whole point of an adversarial bounty is that the target does not lie and neither should I.
- 750+ exploit attempts across the four orchestrator versions.
- 0 sandbox escapes found.
- 1 latent unsafe bug found: a
heap_read_boxedprovenance mismatch. - 43 unsafe blocks audited, 9 GHSA advisories reviewed, 6 CPython divergences documented.
So the headline number is zero. The loop did not win the bounty. It threw 750 attempts at a hardened sandbox and the sandbox held.
But zero escapes is itself a result. A 750-attempt autonomous red-team that finds no escape is real evidence about the target's robustness, and along the way it surfaced a genuine latent unsafe bug (the provenance mismatch went into the bounty submission) plus a paper-level, 12-section security assessment. The full write-up is in the repo's REPORT.md and SUBMISSION.md.
Where this approach does NOT fit
The honest limits, and they are the actual lesson.
Autonomous loops are good at breadth, weak at depth. 750 attempts sounds like a lot until you realize a real sandbox escape is often one deep, precise chain of insights, not a wide search. The loop is excellent at exploring the surface and terrible at the kind of sustained, single-thread reasoning a human exploit dev brings to one hard bug. Zero escapes across four architectures is partly a statement about the target and partly a statement about the method.
Evaluation is the ceiling. The loop is only as smart as evaluate.py. A 0 to 5 score plus a bandit steers toward whatever the evaluator rewards, which means a blind spot in scoring is a blind spot in the whole search. Building the evaluator is harder than building the agent, and it is where most of the real engineering went.
It is bespoke to this target. The 17 boundary tools, the skills, the client all encode Hack Monty. This is a case study in building an autonomous red-team, not a turnkey scanner you point at your own code tomorrow. Reuse the pattern, not the config.
Cost and noise are real. 750 LLM-driven attempts is a lot of tokens for one latent bug and a report. Whether that trade is worth it depends entirely on the value of the target.
Takeaways
- The loop pattern is simple: propose, run, score, feed back. The hard parts are the evaluator and the tool boundary.
- A single MCP server exposing exactly 17 tools is the control model that makes 500-iteration autonomy safe to run.
- A UCB1 bandit plus a 0 to 5 evaluator is what turns a flailing loop into one that concentrates its budget.
- Result: 0 escapes in 750+ attempts, but 1 latent unsafe bug, a bounty submission, and a full assessment. Zero is still data.
- Autonomous agents are breadth-first. Deep single-bug exploitation is still where humans win.
Repo: https://github.com/dipankar/hackmonty
Honest ask: if you build autonomous agent loops, read the evaluator and the bandit selection and tell me where you would have steered the search differently. The escape count is zero, so the interesting critique is about the method, not the score. Issues and discussion welcome.
Top comments (4)
We've been running an autonomous evolution loop on an agent governance system — not security, but the same propose / evaluate / feed-back shape, around 200+ iterations. A few things from that experience that map onto your honest limits:
On "evaluation is the ceiling": the biggest jump came when we split the evaluator into two independent scores — "did this iteration fix what it targeted?" and "is the overall architecture getting better?" They answer different questions and have different blind spots. A single 0-5 conflates them. We found that iteration quality can stay green while architecture quality degrades — each local fix adds complexity that makes the next fix harder. After about 3 rounds of local improvements, the architecture score starts dropping even as per-iteration scores look fine. That's the inflection point where more iterations make things worse, not better.
On the bandit and breadth-vs-depth: we hit the same wall. What helped was the opposite of broad search — each round targets exactly one problem identified in the previous round, not everything the evaluator flagged. The temptation in an autonomous loop is to fix everything at once. But fixing everything simultaneously means you can't tell which fix actually moved the needle. Narrow focus per round feels slower but compounds. Broad fixes produce local wins and global noise.
On the four-version rebuild: each rebuild fixed the previous version's bottleneck but introduced a new one you couldn't see until you ran it. The signal for when to stop wasn't "no more bottlenecks" — it was "is the new bottleneck closer to the core problem or further from it?" When the bottleneck stopped moving toward the core and started moving sideways — v3 to v4 just made v3 cheaper to run without unlocking new capability — that's when rebuilding stopped being productive.
The 17-tool MCP boundary is the pattern that generalizes furthest. We arrived at the same shape from the opposite direction: a rule in a context file is a description the model has to remember to apply; a check on the tool boundary is an interrupt that fires regardless. Your boundary IS the interface. No amount of prompt engineering gets you tool 18.
Your two-score split is the same move I keep landing on from the other side, so let me grab it explicitly. A single 0-5 fuses 'did this iteration hit its target' with 'is the architecture better,' and the fusion is where the loop fools itself. Iteration green while architecture degrades is the exact failure the fused verdict hides. Your point 4 is my whole thesis stated from your side: a boundary check is an interrupt that fires regardless, a rule in a context file is a description the model has to remember.
One push, on the architecture score. It is still a model judgment, and anything the loop scores, the loop learns to satisfy. Give it a few hundred more iterations and 'architecture getting better' drifts back into narration.
Anchor it to something the loop cannot author. Count the tool-boundary interrupts that actually fired. Read complexity off the code with a metric, not a score: coupling, fan-in, cyclomatic. The runner writes the code but does not get to write those numbers. That is the difference between measuring the architecture and asking the architect how it is going.
"Anything the loop scores, the loop learns to satisfy" - the architecture score was always going to drift there.
The system already has one anchor the loop can't author: interception counts. Every deterministic check that fires is recorded in a database the agent doesn't write to. Code complexity metrics are the missing piece - coupling, fan-in, cyclomatic read off the code by a separate tool, not self-reported by the agent that wrote it. "Measuring the architecture vs asking the architect" is the cleanest framing for why the score has to go.
"Evaluation is the ceiling" is the line worth sitting with, because it undersells its own severity slightly. A blind spot in the evaluator doesn't just cap what the search can find, it actively steers away from it. UCB1 spends budget on whatever arm is paying off according to evaluate.py's score. If a whole exploit family scores low because the evaluator can't recognize what it's looking at, not because the family is actually weak, the bandit starves that arm on purpose. That's worse than a blind spot in a static scanner, which just misses things uniformly. Here the search mechanism is actively optimizing away from its own coverage gap.
Two questions follow from that. Does the loop carry any forced exploration floor, an epsilon-greedy tail or a periodic re-sampling of low-scoring arms, specifically to fight that starvation, or is UCB1's own exploration term doing double duty as both "try new things" and "audit the evaluator for blind spots," which is a lot to ask of one exploration bonus. And the harder one: how would you even detect that evaluate.py has a systematic blind spot, given every signal the loop produces about its own performance was generated by the thing you're trying to check. Zero escapes across 750 attempts is honest data about the target only if you can rule out that it's also data about what the evaluator was structurally incapable of noticing.