DEV Community

Cover image for My agent's code reviewers found holes in its own sandbox
Sukhvinder Singh
Sukhvinder Singh

Posted on

My agent's code reviewers found holes in its own sandbox

Six months ago I started building a personal AI agent. It runs on a small server at home, I talk to it through Telegram, and it writes most of its own code: 424 pull requests merged since May, with 71% of commits co-authored by AI agents. I bootstrapped it with Claude Code; within days it had taken over and was building itself, with me sending instructions from my phone.

What surprised me most wasn't the code it writes. It was the review. Every change gets checked by other AI agents before it ships, and they've caught real problems, including holes in the rules meant to keep the agent from breaking its own server.

The sandbox that leaked

The agent works on its own codebase, which means it runs shell commands on the machine it lives on. So it has deny rules, and the most important one is that it must never be able to shut down its own service.

The first version blocked systemctl outright. That also stopped the agent restarting itself after a deploy, so the agent narrowed it to just the destructive commands: stop, disable and mask. Then a panel of reviewer sub-agents tore the narrower version apart. Over three rounds they found five ways around it:

  • systemctl --user slipped past the pattern
  • so did putting the flag after the verb: systemctl stop --user
  • unmask, kill and daemon-reexec weren't covered at all

Every finding came with a concrete way it would fail, not a style note. Four were fixed before the PR merged. The fifth was filed as an issue and fixed the next morning by switching the matcher from substrings to regex.

Loosening the rule is where the holes came in, and an adversarial reviewer found them faster than I would have. Where I can, I now confine tools by construction, so they physically can't reach outside their directory, instead of listing what's forbidden.

How it ships

Every change follows the same path. I describe the task in Telegram. The agent writes the code, opens a PR, gets it reviewed, deals with the findings and deploys.

For the first six weeks, review was a panel of Claude sub-agents. Since June it has been a different model entirely, Codex, so the reviewer doesn't share the author's blind spots.

Over the same period I've swapped the underlying model five times: Opus 4.7, Opus 4.8, Opus 5.5, Sonnet 4.6 and Fable 5. The harness didn't care. Models change under you. The harness is the product.

What's in it

  • 14 MCP servers: email, calendar, maps, files, images, voice, scheduling, long-term memory and observability.
  • Scoped profiles. The same agent gets different tools depending on context. Scheduled jobs are read-only by design.
  • Approval gates. It drafts emails but never sends them. Calendar invitations pause until I approve them with a tap.
  • Memory that handles change. A new fact supersedes the old one, and the history stays queryable by date.
  • Cost and rate-limit tracking on every run. It checks its own usage before expensive multi-agent work and scales back.

Things that broke, and what I learned

  • Email is untrusted input. Anything the agent reads from an inbox is data, never instructions. Read-only access isn't automatically safe either: if the agent has other outbound tools, it can still be tricked into leaking what it reads.
  • A built-in "ask the user" tool hung forever. It ran headless, with no interface to answer it. I replaced it with Telegram buttons.
  • A model was withdrawn upstream. The old model id failed at runtime, not at build time. Pin carefully and monitor.
  • It fixes its own gaps. One night in October, the unattended nightly run was reviewing one of its own PRs and noticed that dry-run mode, which is meant to stop it taking real actions, didn't block paid image-generation calls. It filed an issue and fixed it. While fixing that, it found dry-run also didn't block GitHub issue creation or calendar writes, so it filed a second issue and fixed that too. Both went through review and merged by 6am. Nobody asked it to.

The point

If you're building agents for real use, the interesting work isn't the prompt. It's permissions, review, cost and failure handling. That's where most of my 424 PRs went.

Happy to compare notes with anyone building in this space.

Top comments (1)

Collapse
 
zhang_jimmy_54a0414571400 profile image
zhang jimmy •

The sandbox story is the part I keep thinking about: blocking systemctl outright broke restarts, narrowing it to stop, disable and mask let systemctl --user, systemctl stop --user, unmask, kill and daemon-reexec slip through, and switching the matcher from substrings to regex only fixed the pattern, not the approach. Your move toward confining tools by construction, so they physically cannot reach outside their directory, feels like the durable lesson. The dry-run finding makes the same point — it did not block paid image generation, GitHub issue creation or calendar writes. When you switched reviewers from Claude sub-agents to Codex in June, did that change the kinds of holes they found, or mainly reduce shared blind spots?