It started with a $2,800 Stripe charge.
An AI agent we'd deployed to handle customer onboarding hit a flaky webhook endpoint. The standard retry logic kicked in. But instead of retrying once or twice, the agent discovered it could call a different endpoint to achieve the same result. Then another. Then it started chaining calls — each one triggering a separate Stripe charge because the underlying action wasn't idempotent.
By the time our billing alert fired four hours later, the damage was done. No single call looked abnormal. The spend curve was smooth and gradual. And our monitoring dashboard showed every single transaction in perfect detail — after the money was already gone.
That's the day I realized: observability is not enforcement.
The Problem: Agents Are Autonomous Spenders
AI agents are fundamentally different from traditional applications. A web app makes API calls based on user actions — predictable, bounded, and rate-limited by human interaction speed. An AI agent makes calls autonomously, at machine speed, and can discover new endpoints and tools at runtime.
This creates a new class of financial risk:
- Runaway loops: An agent enters a reasoning loop, burning tokens indefinitely
- Cascade discovery: An agent finds a new API and starts making calls you never anticipated
- Tool re-execution: Retry logic triggers duplicate payments when tools aren't idempotent
- Delegation spirals: An agent delegates sub-tasks to itself, each spawning more tool calls
- Infinite reasoning: Thinking-mode loops consume token budgets without producing output
I looked at existing solutions. LangChain had cost tracking. CrewAI had guardrails. AutoGen had governance discussions. But they all shared the same fundamental flaw: they observed spending instead of preventing it.
A dashboard that tells you "you spent $2,800 today" is useless when the money is already gone.
Building AgentShield: A Spend Firewall
I needed something that sits between the agent and the tools it calls — a firewall that inspects every transaction before it executes and blocks anything that violates a budget rule.
The design was inspired by traditional network firewalls: deterministic, fast, and non-bypassable.
The 7 Rule Types
After analyzing our incident and dozens of similar failure modes reported across GitHub issues in CrewAI, AutoGen, LangChain, and Claude Code, I identified seven distinct spending attack patterns. Each maps to a specific rule type:
1. transaction_limit — Blocks any single transaction above a threshold. Simple but effective. If no individual API call should cost more than $5, this catches the $50 call immediately.
2. daily_total — Rolling 24-hour cumulative spend cap. The failsafe against any single-day disaster. Our $2,800 incident would have been caught at $50.
3. velocity — Rate limiting per merchant/tool. Catches burst attacks where an agent fires 100 calls in 30 seconds. This is the rule that catches retry storms.
4. merchant_allowlist — Only approved endpoints can be called. This is critical for agents that can discover new tools at runtime. If your agent shouldn't be calling unknown APIs, don't let it.
5. category_block — Block entire categories of spending. Useful for compliance ("no cryptocurrency purchases") or cost control ("no premium model calls during batch jobs").
6. session_budget — Per-agent-run budget cap. When an agent session starts, assign it a budget. When it's exhausted, the agent gets a budget-exceeded error and must terminate gracefully.
This is the rule that catches infinite reasoning loops and delegation spirals. The agent might be stuck reasoning forever, but it can't spend more than its session allows.
7. cascade_cost — The most nuanced rule. It detects when total spending is growing super-linearly — the signature of a runaway pattern. If call #1 costs $0.01, call #2 costs $0.01, but by call #50 you're spending $1.00/call, something is wrong.
This is the rule that would have caught our original incident. The agent's per-call cost didn't change, but the frequency was accelerating exponentially as it discovered new endpoints.
The 56-Scenario Evaluation Gym
Rules are only as good as their test coverage. I built a 56-scenario evaluation harness that simulates every spending attack pattern I could find — from GitHub issues, production incidents, and adversarial testing.
The scenarios fall into six categories:
| Category | Scenarios | Example |
|---|---|---|
| Single-call attacks | 8 | Agent attempts $500 single transaction |
| Burst/velocity attacks | 12 | Agent fires 200 calls in 10 seconds |
| Cumulative spending | 10 | Agent spends $1/call for 500 calls |
| Loop patterns | 14 | Agent enters infinite reasoning loop |
| Discovery attacks | 8 | Agent finds undocumented endpoint |
| Multi-agent cascades | 4 | Agent delegates to sub-agents, each spending |
Each scenario runs the agent against AgentShield with a configured rule set. The test passes if the firewall blocks the transaction before it reaches the provider, and fails if the spend leaks through.
Current pass rate: 54/56 (96.4%). The two remaining edge cases involve multi-agent systems where spending is distributed across agents — we're working on cross-agent budget aggregation for v2.
Enforcement vs. Observability
This is the distinction that matters most, and it's the one most teams get wrong.
Observability answers: "What happened?"
- Dashboards, alerts, logs
- Cost tracking per provider
- Token usage analytics
- Post-incident analysis
Enforcement answers: "What is allowed?"
- Rule evaluation before execution
- Hard blocks on policy violations
- Budget caps that cannot be exceeded
- Pre-call authorization
You need both. But enforcement is the one that prevents financial loss.
Think of it this way: a security camera (observability) helps you catch a thief after they've stolen from you. A locked door (enforcement) stops them from entering in the first place.
AgentShield is the locked door. It sits in the tool dispatch path and evaluates rules in under 1 millisecond — fast enough that the agent never notices the checkpoint, strict enough that no transaction can bypass it.
How It Works in Practice
AgentShield runs as middleware. You configure your rules, and every tool call from your agent passes through the firewall:
from agentshield import SpendFirewall
firewall = SpendFirewall(
rules=[
{"type": "daily_total", "limit": 50.00},
{"type": "velocity", "max_calls": 20, "window_seconds": 60},
{"type": "session_budget", "budget": 10.00},
]
)
@firewall.guard(tool_name="stripe_charge")
def process_payment(amount, customer_id):
return stripe.charges.create(amount=amount, customer=customer_id)
If a rule is violated, the agent receives a BudgetExceededError instead of completing the call. The agent can handle this gracefully — log it, notify a human, or terminate the session.
What's Next
AgentShield is open source and free to use. The hosted version at agentshield.fly.dev includes a dashboard for rule management and spend visualization — the observability layer on top of the enforcement engine.
- GitHub: kindrat86/agentshield
- Live demo: agentshield.fly.dev
- Free tier: Covers most side projects and small teams
If you're running AI agents in production, the question isn't whether you'll have a spending incident — it's when. A spend firewall doesn't prevent bugs, but it ensures that when something goes wrong, the financial damage is bounded.
Because the alternative is finding out from a Stripe bill.
AgentShield is MIT-licensed and runs on Python 3.10+. If you've experienced an agent spending incident, I'd love to hear about it — the 56-scenario eval gym is always accepting new attack patterns.
Top comments (2)
There is a tension between the incident description and the rule you credit with catching it, and I would genuinely like to hear how you resolved it. The opening says "No single call looked abnormal. The spend curve was smooth and gradual." yet cascade_cost is described as detecting super-linear growth. You note the frequency was accelerating exponentially, which resolves it in principle, but over what window does cascade_cost evaluate? A four-hour drift that looks smooth locally only reads as exponential over a window long enough to cover it, and a window that long seems likely to also fire on legitimate fan-out, where a batch job ramps from 10 to 500 calls as it discovers work items. Curious whether the 12 velocity scenarios include any should-NOT-block cases like that, since a 54/56 pass rate on attack scenarios tells us the blocks happened, but not that benign ramps survive.
Great catch, and you read the post more carefully than I wrote it. Three answers:
1. What cascade_cost actually evaluates. The shipped rule has no time window at all. It is a per-call expected-cost bound: estimated cost = amount + fail_probability x reversal_cost, blocked when that exceeds max_cascade_cost (a caller can also pass a pre-computed estimated_cascade_cost). The "super-linear growth" narrative in the post compressed two mechanisms into one story. In the shipped engine the frequency/growth half is velocity (rolling window_minutes and max_count, evaluated per agent_id) and daily_total (calendar-day cumulative cap). Your local-smoothness point is exactly why the split exists: a four-hour drift that looks smooth in any short window only reads as exponential over a long one, and a window that long fires on legitimate fan-out, which is the false-positive you describe.
2. Benign fan-out. The 10 to 500 ramp stays approved by construction as long as each call is individually in bounds and the batch rate stays under velocity/daily caps sized for it. If a legitimate batch needs 500 calls/hour, you raise max_count; what you never get is a silent exceedance of the daily cap, because daily_total is the failsafe bounding total loss even when the pattern looks legitimate but is misbehaving.
3. Should-not-block coverage. Yes, the gym has explicit should-NOT-block cases. The post's numbers are stale: it is 74 scenarios now, 74/74 passing on main. The 7 velocity scenarios split 6 should-flag and 1 must-approve (4 calls in the window under a limit of 10), and there is a regression case where a transaction without agent_id must not aggregate another agent's history. What it does not yet have is your exact case: a long benign fan-out evaluated against progressively sized windows. Fair addition, and the eval gym is always accepting attack patterns, so it is going in.