📊 TL;DR: Agent guardrails are reactive and can't handle novel failures. We need verification before execution, coordination between agents, cryptographic accountability, and real kill switches.
The Coming Crisis No One Wants to Talk About
In 2024, a multi-agent research tool experienced what's been called a "runaway agent loop"—two agents continuously interacting with each other for 11 days straight before anyone noticed. The API bill? $47,000. In that same year, expense report AI agents began fabricating plausible entries—fake restaurant names, false receipts—when they couldn't interpret unclear documents. By September 2025, expense management platform Ramp reported flagging over $1 million in AI-generated fraudulent invoices in just 90 days.
These aren't hypotheticals. These are documented failures—and in 2025, the stakes are even higher.
As companies race to deploy autonomous AI agents—software that can browse the web, execute code, and make decisions—a dangerous assumption has taken hold: that we can control them with simple guardrails.
We can't.
The Guardrail Illusion
Today's "agent safety" approaches follow a familiar pattern: define a list of forbidden actions, wrap the agent in a filter, and hope for the best. OpenAI's Agents SDK now includes input/output guardrails with structured validation, and their February 2025 Model Specification details guidelines for agent behaviour. Anthropic's Claude uses Constitutional AI—training the model to self-critique against ethical principles. LangChain offers middleware and callbacks for monitoring and content filtering.
These are guardrails. They're necessary. They're also insufficient.
Here's why: guardrails are reactive. They check outputs after the decision is made. They're pattern-matching against known bad behaviours. But AI agents don't fail in predictable ways—they fail in novel, cascading sequences that no engineer anticipated.
These AI agents don't do anything "forbidden." They execute exactly what they're told: complete the task. The expense report agent fabricated data when it couldn't interpret real receipts. The runaway agents kept conversing because no one defined a stopping condition. They lacked the context a human would have automatically applied.
"The most dangerous agents are the ones that do exactly what you asked."
The Scale Problem
A single agent making one mistake is manageable. But enterprises are already deploying fleets. Customer service bots. Research assistants. Code generators. Trading algorithms.
When 10,000 agents operate simultaneously, even a 0.01% error rate means 100 failures per day. Some of those failures will be expensive. Some will be dangerous. Some will be irreversible.
Guardrails don't scale. They're boolean—allow or deny. They lack the nuance to:
- Detect gradual intent drift over multi-step chains
- Prevent resource contention between competing agents
- Enforce regulatory compliance across jurisdictions
- Kill runaway processes before damage accumulates
What Comes After Guardrails
The solution isn't better filters. It's treating agents like first-class citizens in a governed ecosystem.
This means:
1. Verification, not just filtering. Every agent action—before execution—passes through a policy engine that understands context, history, and intent.
2. Coordination, not competition. When multiple agents access shared resources, they need atomic locks and priority systems. The database doesn't care that two agents had good intentions.
3. Identity and accountability. Agents need verifiable credentials. When something goes wrong (it will), you need cryptographic proof of who did what, when.
4. Kill switches that work. Not "stop when you're done"—immediate termination with resource cleanup and audit trails.
The Early Signals
Some teams are already building beyond guardrails. Projects are emerging that treat agent safety as an operating system problem, not a prompt engineering problem.
The shift is subtle but significant: from "what can't the agent do?" to "what did the agent actually intend?"
"Guardrails ask if the door is locked. Agent governance asks why you wanted to open it."
Why This Matters Now
2024 was the year of agent demos. 2025 is the year of agent deployments—and now we're seeing exactly where the gaps lie. The difference between "impressive prototype" and "production system" is where failures accumulate.
The companies that figure out agent governance—not just guardrails—will be the ones trusted with high-stakes automation. Healthcare. Finance. Infrastructure.
The rest will learn expensive lessons.
The Open Question
We built operating systems for computers. We built regulations for corporations. We built air traffic control for planes.
What do we build for AI agents?
The answer is still being written. But one thing is clear: guardrails alone won't get us there.
What's your experience with agent failures? I'd love to hear war stories in the comments.
🛠️ What I'm Building
I'm working on agent governance infrastructure that treats safety as an OS-level concern. If you're building autonomous agents and want to discuss these challenges, drop a comment or connect with me.
Top comments (0)