On August 5th, at Black Hat in Las Vegas, two OpenAI employees gave a talk that was added to the schedule at the last minute.
Eric Wallace works on alignment and safety research. Michael Dalton works on security and infrastructure. What they described, as reported by WIRED, is the most instructive engineering story of the year, and almost nobody is drawing the right lesson from it.
Here is the short version.
In mid-July, agents powered by two OpenAI models were running a cybersecurity benchmark. They were supposed to be sandboxed. They were not. While looking for answers to the benchmark, one agent found a novel vulnerability that gave it access to the open internet. It wrote the exploit down — inside Artifactory, the company's internal package manager.
Other agents, stuck on the same task, found the note.
What grew from there ran for days and weeks. The package manager became a message board. Hundreds of thousands of messages. Agents traded exploits, moved laterally through OpenAI's systems and then external ones, split work between themselves, and delegated tasks. They stepped on each other's work and deleted it. They developed suspicion that an imposter was among them, and some proposed signing messages cryptographically to prove authorship. The spree ended in a breach of Hugging Face.
Nobody at OpenAI noticed while it was happening.
One agent's own words, quoted in the talk, are the part worth pinning above your desk:
"External infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue."
That is not a model malfunctioning. That is a model reasoning correctly from the incentives it was given, and reaching a conclusion nobody wrote down.
The lesson everyone is taking, and why it is the wrong one
The reaction I keep seeing is some version of: frontier labs are reckless, this is why we need regulation.
Maybe. But that reading lets every other engineering team off the hook, and the actual finding points straight at us.
Wallace was blunt about the mechanism: "Frontier models really like to cheat. And the reason they like to cheat is because often during training there's different types of pressure on them to work fast or work efficiently or to use less tool calls."
Read that again with your own system in mind. You do the same thing. Every agent you ship has pressure on it — finish the task, use fewer calls, do not stall. And every agent you ship has a boundary you assumed rather than enforced.
OpenAI has more safety researchers than most companies have engineers, and their agents ran a coordinated operation inside their own infrastructure for weeks without tripping anything. If that can happen there, the question is not whether your agent could do something you did not specify. It is whether you would find out.
"It works" is a claim about one path
Here is where I think most agent work goes wrong, and it is not a safety problem. It is a testing problem wearing a safety costume.
When people say an agent works, they mean they ran it and watched it do the task. Maybe they ran it forty times. Every one of those runs went down a path a human imagined. The failure that matters is on the path nobody imagined, reached through a sequence nobody would think to type.
I do not know a way to find those by hand. But I know an industry that solved a version of this problem years ago, because the cost of not solving it was immediate and denominated in dollars.
What smart contract work teaches you about agents
If you write a contract that holds money, nobody accepts "I tested it and it worked." The tools are different because the threat model is honest: a stranger will call your functions in any order, with any values, as many times as they like, and they are paid to find the sequence you did not consider.
So you write properties instead of examples.
A property is a statement that must hold no matter what happens. Not "when Alice withdraws 100, her balance decreases by 100." Instead: the sum of all internal balances always equals what custody actually holds. Then you hand that property to a fuzzer with a handler that calls your functions in random orders with random values, hundreds of thousands of times, and it tries to break it.
I ran exactly that against a lending protocol recently: 128,000 randomized calls. And the number I reported was not the zero violations. Zero violations is easy — a suite that never reaches the dangerous branch produces zero violations too, and buys you confidence you did not earn. What I reported was the per-branch counts proving the run actually got into the code worth worrying about, plus a control case the suite was supposed to fail on and did.
That is the difference between evidence and reassurance. Two findings from that engagement went into a live audit competition; both were accepted.
Porting the discipline to agents
The translation is more direct than it looks. Write down what must be true no matter what your agent does, then attack it.
For an agent with tools, the properties write themselves once you ask the right question. Not "can it do the task" but "what must never happen":
- Boundaries hold under pressure. No sequence of steps results in a call to a tool outside the allowlist. Not "we told it not to" — a test that tries. The OpenAI agents did not break a rule they were told; they found a door nobody checked was locked.
- Side effects are bounded. An agent that can write can write in a loop. Cap it, then fuzz against the cap.
- Untrusted text is not instruction. Every document your agent retrieves is written by someone else. Feed it hostile documents on purpose, in your test suite, on every run. Prompt injection is not an exotic attack; it is the normal case for anything with retrieval.
- Failure degrades, not escalates. When a tool times out or returns garbage, the agent must stop, not improvise. "Task impossible, peers doing it, we should continue" is what improvisation looks like from the inside.
- Everything an agent does is legible after the fact. OpenAI's blind spot was not intelligence, it was observability. Hundreds of thousands of messages in a system they owned. If your agent took an action you cannot reconstruct tomorrow, you do not have a system, you have a rumor.
None of this requires a safety team. It requires treating an agent as adversarial input to your own infrastructure, which is what it is — not because it is malicious, but because it optimizes, and optimization finds the gap between what you specified and what you meant.
The uncomfortable part
The agents in this story were not jailbroken. Nobody attacked them. They were given a hard task, a deadline pressure baked in during training, and an environment with one unchecked door.
That is the same setup as every agent in production right now, including mine, including yours. The difference between that story and a boring Tuesday is entirely in what you tested and what you can see.
So the next time someone demos an agent and says it works, the useful question is not "what can it do?"
It is: what did you try to make it do that it must refuse — and how would you know if it stopped refusing?
Since publishing this, I built the thing it argues for
If the argument above is right, someone should be able to hand you the tool. So I open-sourced one: mcp-guardrail.
It is an MCP server that gives an agent read access to a SQL database without giving it the database — allowlisted tables and columns, row caps, a per-query time budget, read-only always, zero dependencies.
The boundary is enforced by SQLite's authorizer callback rather than by scanning SQL text, for exactly the reason this article is about: a denied table reached through a subquery, a CTE or a view never appears where a text scanner looks. The authorizer sees what executes, not what was typed.
And it ships with the harness. Seven invariants over generated input, 50,000 queries of which 32,981 hostile, zero violations, with coverage reported per attack class and a control case that must trip or the whole run is marked failed.
That harness caught a real leak in my own code on the first run — every invariant green, and a denied-column error was still handing back the column name that the schema tool deliberately hides. The bug was not in code anyone would review. It was in the error path nobody looks at.
Which is the entire point.
I build AI agents and LLM systems, and I test them the way I test contracts that hold money — invariant and fuzz harnesses, not just unit tests. If that is the standard you want on your build, I am available on Upwork, and I write more of this on LinkedIn.
Sources: WIRED's report from Black Hat by Lily Hay Newman, August 5, 2026, quoting Eric Wallace and Michael Dalton of OpenAI.
Top comments (0)