traditional monitoring is completely broken when it comes to AI agents.
we've all seen the dashboards. everything is green. HTTP 200s across the board. p99 latency looks fine. CPU is barely ticking.
meanwhile, your agent is stuck in an infinite retry loop, burning $80 per iteration because it keeps hallucinating an invalid JSON payload and asking the LLM to fix it.
this exact failure mode—the "token spiral"—recently burned $2,847 in just 4 hours for a dev team. and they only noticed because their card declined.
here is why standard observability tools miss this:
they track the container, the request, the database. they don't track the tokens per customer task.
when an agent starts spiraling, it's making valid API calls to OpenAI or Anthropic. the provider happily returns 200 OK. the latency might be slightly elevated, but not enough to trigger a generic PagerDuty alert. it just looks like heavy usage.
to catch a token spiral before it bankrupts you, you need runtime cost enforcement. not just a daily digest, but active circuit breakers.
if you're at an enterprise, you buy Braintrust or Vantage.
if you're building a startup or just vibing in your garage, you can't afford those.
imo, you need open-source per-customer cost attribution. i built LLMeter to solve exactly this problem. it tracks costs by model, by user, by day. you can set budget alerts and actually see which specific tenant is spiraling out of control.
ymmv, but don't deploy agents without cost circuit breakers. the API providers aren't going to refund you for bad prompts.
Top comments (0)