I threw 750 autonomous LLM exploit attempts at a $10k sandbox bounty. Zero escapes.

Pydantic put up a $10,000 bounty called Hack Monty: escape the sandbox of their Monty runtime. That is a clean, adversarial, unforgiving target. Either you get a sandbox escape or you do not. No partial credit, no hand-waving.

I did not try to hack it by hand. I built an autonomous LLM loop to hack it, and then I rebuilt that loop four times as I learned what did and did not work. This is the honest write-up of what an agent swarm actually found, which is not a $10k escape, and why the result is still worth publishing.

The core idea

An autonomous hacking loop is a simple pattern with hard details. An LLM proposes an exploit, a harness runs it against the target, an evaluator scores the result, and that score feeds back to steer the next attempt. Repeat for hundreds of iterations.

The engine here started on the autoresearch pattern and evolved through four versions: from a single loop, to XBOW-style parallel swarms, and finally to a tokenworm plus MCP architecture. Each version was a bet about what the bottleneck was. Early on it was idea diversity, so I went parallel. Later it was tool discipline, so I moved to MCP with a tightly-scoped boundary.

How V4 works

The final architecture is three layers with a hard boundary in the middle.

At the top, tokenworm is the LLM agent harness. It runs a native Ollama provider, four role-focused SKILL.md files, and its own sandbox (bwrap plus network). It speaks to the outside world only through an MCP client over stdio.

tokenworm (LLM agent harness)
  Provider: ollama_native
  Skills: 4 role-focused SKILL.md
  Sandbox: bwrap + network
        │  MCP Client (stdio)
        ▼
hackmonty_mcp_server.py
  17 boundary tools (run, evaluate, bandit, state)
        │
        ▼
hackmonty.com   Ollama Cloud   GitHub Issues   Filesystem

The middle layer, hackmonty_mcp_server.py, is the interesting design decision. It exposes exactly 17 tools and nothing else. The agent cannot touch the target, run code, score an attempt, or read state except through those 17 tools. That boundary is the whole safety and control model. It is what lets an autonomous loop run 500 iterations without the harness itself becoming the vulnerability.

Two more pieces make the loop learn rather than flail. A UCB1 bandit (bandit.py) selects which attack strategy to try next, treating exploit families as arms and their scores as rewards, so the loop spends its budget on what is paying off. And an evaluator (evaluate.py) scores each attempt 0 to 5 and enriches the context, so the next proposal is grounded in what the last one actually did.

Running it is a shell script or the Python SDK:

# Production (tokenworm Zig binary), needs ollama serve running
export USER_SECRET=your-passphrase
./run.sh            # 500 iterations
./run.sh 20         # 20 iterations (test)
./run.sh -i         # interactive REPL

# Python SDK showcase (InlineSkill, MCP, native Ollama Cloud)
uv run python run.py 20

There is also an SSE remote mode for the MCP server: uv run python -B hackmonty_mcp_server.py --sse --port 8765.

The results, honestly

Here is where I refuse to inflate anything, because the whole point of an adversarial bounty is that the target does not lie and neither should I.

• 750+ exploit attempts across the four orchestrator versions.
• 0 sandbox escapes found.
• 1 latent unsafe bug found: a heap_read_boxed provenance mismatch.
• 43 unsafe blocks audited, 9 GHSA advisories reviewed, 6 CPython divergences documented.

So the headline number is zero. The loop did not win the bounty. It threw 750 attempts at a hardened sandbox and the sandbox held.

But zero escapes is itself a result. A 750-attempt autonomous red-team that finds no escape is real evidence about the target's robustness, and along the way it surfaced a genuine latent unsafe bug (the provenance mismatch went into the bounty submission) plus a paper-level, 12-section security assessment. The full write-up is in the repo's REPORT.md and SUBMISSION.md.

Where this approach does NOT fit

The honest limits, and they are the actual lesson.

Autonomous loops are good at breadth, weak at depth. 750 attempts sounds like a lot until you realize a real sandbox escape is often one deep, precise chain of insights, not a wide search. The loop is excellent at exploring the surface and terrible at the kind of sustained, single-thread reasoning a human exploit dev brings to one hard bug. Zero escapes across four architectures is partly a statement about the target and partly a statement about the method.

Evaluation is the ceiling. The loop is only as smart as evaluate.py. A 0 to 5 score plus a bandit steers toward whatever the evaluator rewards, which means a blind spot in scoring is a blind spot in the whole search. Building the evaluator is harder than building the agent, and it is where most of the real engineering went.

It is bespoke to this target. The 17 boundary tools, the skills, the client all encode Hack Monty. This is a case study in building an autonomous red-team, not a turnkey scanner you point at your own code tomorrow. Reuse the pattern, not the config.

Cost and noise are real. 750 LLM-driven attempts is a lot of tokens for one latent bug and a report. Whether that trade is worth it depends entirely on the value of the target.

Takeaways

• The loop pattern is simple: propose, run, score, feed back. The hard parts are the evaluator and the tool boundary.
• A single MCP server exposing exactly 17 tools is the control model that makes 500-iteration autonomy safe to run.
• A UCB1 bandit plus a 0 to 5 evaluator is what turns a flailing loop into one that concentrates its budget.
• Result: 0 escapes in 750+ attempts, but 1 latent unsafe bug, a bounty submission, and a full assessment. Zero is still data.
• Autonomous agents are breadth-first. Deep single-bug exploitation is still where humans win.

Repo: https://github.com/dipankar/hackmonty

Honest ask: if you build autonomous agent loops, read the evaluator and the bandit selection and tell me where you would have steered the search differently. The escape count is zero, so the interesting critique is about the method, not the score. Issues and discussion welcome.