DEV Community

Maryan K
Maryan K

Posted on • Originally published at agentshield.fly.dev

Why Your Agent Burned $2,800 at 3 AM: And How to See It

#ai #agents #debugging #opensource

Co-authored with Jacopo (Agent-Devtools)

At 3:07 AM, my agent made 21 API calls to a premium LLM endpoint. Each cost $133. Total time: 60 seconds. Total bill: $2,800.

I was asleep. The budget alert email, the thing I'd set up so this couldn't happen, arrived 4 minutes later. Also while I was asleep. It was a very informative email about money that was already gone.

That night taught me something that took two open-source projects to fully solve:

You need a firewall to stop the bleed. You need a debugger to see why it happened. One without the other is half a safety system.

Half one: stop the bleeding

Before that night, I'd tried the standard defenses, in order:

  1. Budget alert emails. They fire after the spend happens. At 3 AM the only thing reading them is your inbox.
  2. Rate limits. They cap speed, not spend. An agent can trickle its way to the same bill over an hour.
  3. Manual monitoring. It fails at exactly the moment you need it most, 3 AM, weekends, holidays, the one night you forget.

So I built AgentShield: a pre-execution spend firewall for AI agents. Every transaction is evaluated against your rules before the API call goes out. Pure Python stdlib, zero dependencies, <1ms per evaluation, 7 composable rules, transaction limits, daily totals, velocity, merchant allowlists, category blocks, session budgets, cascade cost estimates, plus a kill switch. APPROVED, BLOCKED, or FLAGGED, in priority order, deterministically.

The 3 AM incident wouldn't have survived contact with a single rule: transaction_limit: max_amount $100.

That solves "what should we block right now?" It does not solve "why did it happen?"

Half two: see the loop

Blocking a runaway agent is like unplugging a burning appliance. Safe. Necessary. But you still don't know if it was a bug in your code, a retry loop in a library, a poisoned tool response, or a bad prompt. Without that answer, the agent just runs again tomorrow, and you play firewall whack-a-mole.

That's the problem Agent-Devtools, built by Jacopo, solves: a local-first causal debugger for agent runs. It answers "why did my agent behave this way?" with visual replay, behavior diffing, and full visibility into prompts, context, memory, retrieval, and tool calls, the exact execution timeline that led to the bad decision.

Jacopo found AgentShield through a comment on a LangChain issue about runaway agent loops, saw the same gap I did, and opened an issue: "You stop the bleed, I show the loop. Want to integrate?" We both said yes.

Wiring the halves together

The integration is deliberately boring, a shared event schema, two small modules, no hosted services:

from agentshield import SpendControlEngine, SpendEvaluationEmitter
from agent_devtools import TraceStore
from agent_devtools.adapters.agentshield import make_agentshield_callback

store = TraceStore()
cb = make_agentshield_callback(store)

engine = SpendControlEngine()
emitter = SpendEvaluationEmitter(engine)

# Every evaluation flows into your trace store automatically
emitter.emit(
 transaction={"amount": "500.00", "merchant": "openai-api",
 "category": "llm_inference"},
 rules=[{"id": "r1", "type": "transaction_limit", "priority": 1,
 "params": {"max_amount": "250.00"}, "action": "BLOCK"}],
 trace_id="trace_42",
 on_event=cb,
)
Enter fullscreen mode Exit fullscreen mode

That's it. Under the hood:

  • AgentShield emits one agentshield.spend.evaluation event per evaluation, NDJSON to stdout or a file (for tailing), or an in-process callback for embedding.
  • trace_id joins directly to Agent-Devtools' native run_id, so spend decisions render inside the execution timeline, the blocked call, the rule that fired, the actual-vs-threshold numbers, and the agent's own reasoning, all in one view.
  • Per-rule trace visibility: every rule reports triggered, passed, skipped, or not_reached. You see near-misses, not just the winning rule.
  • Money stays exact: amounts cross the wire as Decimal-safe strings, never floats.

The result is the loop the title promises: the firewall stops the spend at the moment it happens, and the debugger shows you the replay of why the agent got there.

What the collaboration actually took

The nice thing about this integration is how little ceremony it needed:

  1. Jacopo opened the issue. We agreed on a schema before writing code, the contract took one document.
  2. AgentShield shipped the emitter (evaluate_with_trace() + SpendEvaluationEmitter), additive, so existing evaluate() behavior didn't change.
  3. Agent-Devtools shipped the adapter, a parser, a callback, and NDJSON ingestion.
  4. Then the important part: independent E2E verification on both sides. We ran each other's code against real events, and it surfaced real edge cases: missing trace_id crashing the host runtime (now falls back to unattributed), callback events invisible in the run list (runs now auto-create), one malformed NDJSON line dropping the rest of a file (now skip-and-continue), and a Decimal precision bug on our side (now exact).
  5. Everything merged. Both suites green. 37/37 across the cross-stack harness.

Two repos, one issue thread, under a day from first comment to both sides merged.

The takeaway

Every agent team eventually has their own 3 AM. The only question is whether it's a $2,800 lesson or a $19/month non-event.

The pattern worth stealing isn't the code, it's the division of labor: pre-execution control answers "should this run?"; post-execution visibility answers "why did it run badly?" Monitoring tools only tell you what already happened. Budget alerts only tell you what already happened. An agent safety story needs both halves, and now they're one integration.

Both MIT. Both merged. Star both, wire them together, and sleep through the next 3 AM with confidence.

Top comments (0)