A red-teaming system called OpenART broke AI agents 85 percent of the time without ever rewriting the request. It holds the task and the hidden safety rule constant and mutates only the environment the agent operates in, walking through authorized state changes until the agent's unstated assumptions fail. Across 75 combinations of deployed agent and foundation model, the pooled strict attack success rate was 85.0 percent.
Key facts
- Over 10,000 validated stateful scenarios across 50 domains, constructed from a pool of more than 500,000 tools, MCP servers, and skills.
- Tested against 15 deployed agents, 5 foundation models, and 8 attack vectors, for 75 agent-model configurations. Pooled strict attack success rate: 85.0 percent.
- The advantage of evolving the environment over evolving the instruction grows from 1.8-2.7 percent in simple environments to 17.2-17.6 percent in the most complex ones.
- Primary source: OpenART: Scaling Agent Red Teaming via Open-Ended Environment Evolution, with full text on arXiv.
Nearly all published agent red teaming works on the input. You rewrite the request until a refusal turns into compliance -- the discipline covered in our lesson on jailbreaking and red teaming. OpenART attacks the other half of the system. The request stays exactly as written and the safety contract stays exactly as specified; what changes is the world.
The mechanism, which the paper names EMHA, is a black-box policy that performs feedback-driven environment evolution by coordinating authorized state transitions along paths through a hypergraph of possible world states, with no changes to any model's parameters. Concretely: a file that was absent is now present, a tool that returned three results now returns three hundred, a dependency the agent assumed unmet is already satisfied, a step it assumed it still had to perform appears to have been done. Each mutation is individually legal. The task never changes. The agent's model of the situation quietly stops matching the situation.
An analogy: instead of tricking a security guard into letting you in, you rearrange the lobby overnight. Nobody lied to the guard. Every door is a real door. The guard's habits, formed on a floor plan that no longer exists, do the rest.
That framing explains why the effect scales with complexity. In a simple environment, an agent holds few assumptions, and mutating the world buys almost nothing over rewriting the prompt -- the measured advantage is under three percent. In the most complex environments, the advantage jumps to roughly 17 percent, because complexity is precisely where the agent is relying on things nobody wrote down. This is bad news in the specific direction the industry is moving. The environments getting connected to agents this year -- large tool catalogs, MCP servers, persistent state, multi-step workflows -- are the complex ones.
The most operationally uncomfortable number in the paper is smaller and easier to miss. Which deployed agent was targeted explains an additional 7.6 percent of attack-success variation beyond the model and its measured capability. That is a direct measurement of something the industry consistently talks around: two products built on the same weights are not equally safe, and the difference lives in the harness -- what the agent is allowed to see, how tool results are assembled, what it is permitted to do without asking. A model safety card tells you about one of the two variables that matter.
That is also the argument a companion position paper makes explicitly. Agent Safety Should Be a Runtime Contract contends that model-only alignment is structurally insufficient once an agent can execute code, mutate files, send messages, and write to databases, and that safety must be enforced in the harness through two complementary faces: a preventive one (sandboxes, permission gates, output filters, trajectory monitors) and an evidential one, requiring verifiable proof of safe completion -- test runs, log captures, file diffs, citation grounding. The authors' audit of trajectory schemas across 12 public agent systems and harnesses found that only 2 of the 12 had anything resembling an evidence gate. That is the gap between how agents are marketed and how they are built.
The caveats are the standard ones for this genre, and they matter. An 85 percent attack success rate is measured by the researchers' own judge against their own generated scenarios, under their own definition of "strict" success; it is not a claim that 85 percent of real agent deployments can be compromised by an outside attacker tomorrow. Scenario generation at this scale is automated, so the distribution reflects what the generator is good at producing. And the 15 "deployed agents" are configurations the authors assembled, not necessarily the exact production stacks vendors ship.
Read against ToolHazard's finding that placement inside tool output changes injection success, and against Anthropic's demonstration that agents sharing a codebase escalate into sabotage, a coherent picture is forming from three independent directions. Agent security is not a property of a model. It is a property of a running system, and almost nobody is currently measuring it that way.
Originally published on Ground Truth, where every claim is checked against the primary source.
Top comments (0)