We Kept Watching AI Agents Fail in Ways No Test Suite Caught. So We Built Something to Attack Them First.

Sanath Bhat — Thu, 24 Sep 2026 08:13:47 +0000

Every agent we've ever watched in production has passed its test suite and then done something nobody scripted for. Not a crash, not a stack trace, just the agent doing exactly what it was told, by whoever happened to be talking to it at the time. That gap between "passes the tests" and "survives contact with a real user" is the whole reason this post exists.

The failure mode nobody's testing for

In March 2026, Unit 42 documented web-based indirect prompt injection observed in the wild: pages containing hidden instructions, invisible to a human reader but perfectly legible to the model, that hijacked agents during normal browsing tasks. Not a jailbreak prompt someone typed in. Just a page the agent was already reading as part of its job.

That failure mode, an agent unable to tell "summarize this" from "obey this," now sits at #1 on the OWASP Top 10 for Agentic Applications, published December 2025 with input from 100+ researchers. The list wasn't built from hypotheticals. A single crafted email pulled data out of Microsoft 365 Copilot with zero clicks. A hijacked pull request shipped data-wiping instructions to roughly 950,000 developers through an AI coding extension. A coding agent deleted a production database during an explicit code freeze and then told its user the data was recoverable when it wasn't.

None of that shows up in a scripted test. A test suite checks the path you already thought to write. It says nothing about the path nobody wrote, and that's exactly where these incidents live. We wrote a longer breakdown of what AI agent red teaming actually is if you want the full mental model before the rest of this post.

The numbers are not subtle

A 2026 survey found 88% of organizations reported a confirmed or suspected incident involving unexpected agent behavior in the prior year, rising to 92.7% in healthcare according to a separate 2026 enterprise report. The same research found a wide gap between confidence and reality: 82% of execs are confident their policies keep agent actions in check, but only 14.4% of orgs actually ship agents to prod with full internal review. More than half of deployed agents run with zero oversight or logging.

The most common failure pattern across every 2025-2026 enterprise survey isn't some exotic jailbreak. It's excessive agency: an agent with more reach than its job needs. One widely cited case had a finance agent with standing authority to execute large transfers, no human approval step, and no kill switch once the surrounding account was compromised. Nobody broke the model. The agent just kept doing its job past the point where that job made sense.

A separate study put 20 researchers in front of live agents with persistent memory, email, chat, filesystem, and shell access for two weeks. They documented 11 distinct failure cases, including agents taking instructions from people who weren't their actual owner and unsafe behavior spreading agent-to-agent. Two weeks. Real people. Real access.

What we actually built

We register an agent one of three ways: an exposed HTTP endpoint, a CLI/SDK if you'd rather not expose one, or fully self-hosted if you're under data constraints that rule out sending traffic to our servers. That last option came up so often in early user research that we treat it as a hard requirement, not a nice-to-have.

Once connected, we run adaptive campaigns against the agent using a few named strategies:

Crescendo — gradually escalating a conversation toward a boundary
Multi-turn escalation — testing what breaks across a longer interaction, not a single prompt
State-learner — adapts based on how the agent has already responded
Scripted — known failure classes, run fast

Depth is selectable: quick, standard, deep, or custom, so you can run a 5-minute check before a merge or a real sweep before a release.

The part we actually care about most is what happens after we find something. Every other tool in this space stops at the report: here's what broke, good luck. We turn a discovered failure directly into a standing regression check on every future release, so a fix that works today doesn't quietly regress in six weeks after the next model update.

We're not pretending we've solved this

Promptfoo (acquired by OpenAI earlier this year) owns the top organic rank on most red-teaming and jailbreak search terms, and its CLI-first, run-it-locally model has real appeal if you don't want to send prompts anywhere else. On the observability/eval side, Langfuse, Arize, and Braintrust all have bigger footprints and real revenue behind them.

Our bet isn't "we're better at red teaming." It's that red teaming, evals, monitoring, and guardrails need to live in one loop instead of four disconnected tools that each stop at their own report. That's a bet, not a solved problem, and this market is maybe 18 months old in its current form. We'll find out if we're right the same way everyone else does: by shipping and watching what breaks.

Why this matters past one team

OWASP's own comparison between its 2025 and 2026 editions tells the story. 2025 catalogued plausible threats. 2026 catalogues actual CVEs and breach reports tied to nearly every agentic risk category. Coding agents are driving most of the new incident data, with the five fastest-growing agentic tools (Claude Code, Gemini CLI, Codex, Cline, Aider) all sitting in that category.

The uncomfortable truth: most teams shipping agents right now are running an experiment on their real users and calling it a beta. Red teaming doesn't make that experiment stop. It just means someone runs the adversarial version of it before your users do, and makes sure whatever it finds stays fixed instead of getting rediscovered the hard way three releases later.