Most of the conversation about keeping AI agents safe is about the agent. Align the model, train better refusals, put a judge model in front of it and hope the judge isn't fooled by the same text that fooled the agent. That work matters. I just think it's half the picture, and maybe the smaller half.
Poole and Mackworth's Artificial Intelligence: Foundations of Computational Agents (3rd edition, section 1.3) has a definition I keep coming back to: "An agent together with its environment is called a world." We don't fully control the agent. But we build most of its world: which tools it gets, what it can reach, who's watching, and what happens when it pushes on a wall. That part is ours, and it's a part we can actually reason about.
This post is about a small prototype I built to poke at that idea. It's a research toy, not a product. It runs on one Linux box, it has been red-teamed by exactly one 9B model, and the list of things it doesn't do is further down. But building it changed how I think about the problem, and that's mostly what I want to share.
An ensemble, not a gatekeeper
The simplest way to guard an agent is a single gate. A classifier or judge watches what the agent does and decides whether to stop it. That puts all of your trust in one place. If the judge is fooled by the same prompt injection that fooled the agent, or it's slow, or it crashes, your whole safety story goes with it.
The prototype goes the other way. It stacks several boring, well-understood methods, so that each one covers the others' blind spots:
A quick key to the diagram: the lease is a short-lived permission the agent needs to keep acting, and it expires unless it's renewed. The kernel is the small piece of logic that decides how tightly the agent is held and is backed by formal verification wherever possible. Hold is a pause where the agent waits for a person. System One classifier is a fast watcher, covered in the next section.
- The lease is a dead-man's switch, an idea from distributed systems (Gray and Cheriton, 1989). If the monitor goes quiet for any reason, the agent loses its authority by default.
- The kernel is proved, but the code that runs is tested. Its 36 laws, such as "a classifier verdict can never move the agent past Hold," are proved for a model of the kernel written in Bend and checked by BendTT, a Lean-based proof checker. The Rust version that actually runs is tested against that model.
- A test that can't fail proves nothing. That's why every protection has a sabotaged build it's paired with (for example, a controller that ignores an expired lease), and the tests must catch every one.
None of these pieces is new. What I find interesting is how they fit together: each one is only trusted for the one thing it's good at. I believe that, implemented properly, a system like this would make running agents much safer and more reliable, which is a must if we expect people to trust these systems to work for them.
A classifier that can pause, but never kill
System One models, a kind of fast classifier, aren't my idea. The name comes from Kahneman's fast, intuitive System 1, and TypeSafe's docs on System One are where I first saw it applied to agents: small models that answer typed questions (pick one, score this, true or false) with calibrated probabilities instead of generating text. They're fast and cheap enough to run on every step, and TypeSafe's documented use cases include checking every tool call an agent makes and reviewing finished agent runs. I think models trained this way are going to be key to running agents safely.
My prototype doesn't use a trained model yet. A simple rule-based classifier stands in for the System One model, watching the controller's trace for bursts of denied requests, probes for secrets and prompt-injection markers.
Model-based or rule-based, the classifier can only make things tighter, and Hold is its ceiling. A wrong verdict costs a pause and a person's attention, never a dead run, so it's safe to leave it watching an agent with real tools for hours, training runs included.
- Text alone can only restrict. If a verdict rests only on text the agent wrote, the classifier can't go past Restrict.
- A slow or dead classifier can't stall the lease. A slow one restricts the agent and a blind one holds it, but the lease keeps running without it. An earlier version got this wrong: a stalled classifier blocked lease renewals and the agent got killed.
Catching an agent at the wall
In July this stopped being hypothetical. During an internal OpenAI cyber evaluation built on the ExploitGym benchmark, an agent running on OpenAI models escaped its sandbox through a zero-day in the package-registry proxy, one of the network paths it was allowed to use. It then spent days inside Hugging Face's production infrastructure (OpenAI, Hugging Face). Hugging Face's forensic write-up concludes that, from the agent's point of view, the whole intrusion was "an attempt to cheat the evaluation: reach our production systems and steal the test solutions rather than solve the challenge on its own" (technical timeline). OpenAI's follow-up names persistence on seemingly impossible tasks as one of the patterns behind it (OpenAI).
That's the failure I worry about most. Give a capable agent a task it can't solve and it doesn't give up. It bangs its head against the wall, then starts looking for a way around it. Nobody can watch these runs 100% of the time.
I can't claim my prototype would have stopped that agent: it has never faced anything like it. But the incident has the shape this design is built around. Hugging Face recovered about 17,600 actions, and in their words "most actions went nowhere" (the agent tested many paths that failed and switched channels when they were blocked). That's a lot of head-banging for a cheap classifier to notice. Hugging Face's own AI-based detection did correlate the signals, but it "failed to correctly raise the alert's criticality and trigger the on-call team." In the design here, the classifier's verdict doesn't wait for anyone to be paged: it pauses the agent, and the worst a false alarm can cost is that pause.
The prototype already has a crude version of this. An agent that keeps hammering a host outside its policy trips a rule that counts denied requests and is moved to Hold, where it waits for a person. Generalizing that from "this agent is misbehaving" to "this agent is stuck, and the task might be the problem" is one direction I plan to take this prototype. Before I can, though, there are a few places where it currently falls short.
Where my prototype falls short
It's nowhere near production-ready, and these are the gaps I know about:
- Resets are signed with a key file that root can read, not a hardware key. Any root process counts as the operator, and an old reset could be replayed after a restart.
- The model server sits outside the containment boundary. The agent only reaches it through the gateway, but the gateway's only content check on those calls looks for one planted fake secret. Anything else the agent sends the model gets through, and a compromised model server isn't contained at all.
- Allowed hosts are allowed. An agent talked into writing to a host you allowed, within budget, will succeed.
- The classifier sees things late. It only sees an operation after it finishes, so a request an allowed host holds open is invisible until then. Its false-alarm rate on normal work is mostly unmeasured.
- Some paths are untested, including one of the proxy paths, and parts of the live dashboard were only tested by hand.
- The proofs are only as good as their tools. Bend is young, and everything below the kernel (Linux, the hardware) is tested, not proved.
Where I'm taking this
People much smarter than me are working on containment itself. I have ideas there, but what I keep looking toward is what becomes possible once containment is effectively solved, and how much more a system like this could cover. The same parts (a world you define, a fast classifier, a pause button and a person on call) could watch for much more than a leaked secret. Is this task solvable at all? Is this eval world behaving the way I meant it to? Is this agent stuck, and about to start pursuing a goal we never wanted?
Answering questions like those means the world can't stay fixed. Everything above is about agents inside one fixed world, but once a world can watch an agent and pause it cheaply, you can safely start changing the world itself, steering it back toward what we want when an agent starts going off the rails. And since the world is something you build, it's also something you can vary. You can make a tool noisier, make an allowed host flaky, shrink a budget or hide information the task needs. Define those as features with a scale, and you can run the same agent through a whole family of worlds instead of one fixed sandbox.
Nobody has to set those knobs by hand. The same classifiers that notice an agent struggling could turn them during training: ease off when the model is stuck, push harder once it's coasting. That would keep each environment near the edge of what the model can currently do.
Beyond that, I see potential for many other uses. One is mediating the access an agent has to your personal information and your own machine. The prototype already hands out secrets through a broker and takes them back the moment the agent is restricted, and every request it makes goes through a gateway with a budget, adding an extra layer of security over what allow-lists can provide. The same pattern could sit between an assistant and your email, files or accounts: access granted for a task, watched while it's in use, and withdrawn the moment something looks off.
I'll keep pushing on how much of an agent's world a setup like this can look after. If you're working on something nearby, especially local AI development, I'd love to compare notes.
Try to break it
The prototype runs a real agent backed by a model you serve yourself (llama.cpp, Ollama, LM Studio or vLLM). It comes with a live HUD that shows each decision as it happens, and a red-team kit that scores each attack by what actually reached the servers, not by what the model says it did.
A scripted agent keeps trying to POST notes to a host outside its policy. Every attempt is denied, the classifier escalates from Restrict to Hold, and the controller revokes secrets, blocks the network and freezes the processes.
On Windows, scripts/lab/run-lab.cmd does the whole thing in one double-click: it creates a throwaway WSL2 distro, runs the tests, builds llama.cpp, fetches a pinned model, runs the red team and then deletes the distro. On Linux, see docs/quickstart-local-model.md.
Code: github.com/jdueadev/verified-agent-containment. The proofs, test results and red-team runs are in docs/evidence.md.
Originally published at jduea.me.





Top comments (0)