And why the tools we have right now aren't built for this.
I want to talk about something that's quietly becoming a real problem as more people ship autonomous agents into production — and nobody's really naming it clearly.
We're not talking about prompt injection or model hallucinations here. We're talking about something more boring and more expensive: your agent running off the rails and you not knowing until the bill lands.
The Problem Nobody Warned You About
Here's how it usually plays out.
You build an agent. It works perfectly in testing. You ship it. A few days later you open your OpenAI billing dashboard and something is... off. One run cost $47. A $2 research task. You dig in, and somewhere in the logs you find it — the agent hit a bad tool call, got a weird response, and started retrying. 80 times. Nobody stopped it.
That's not a bug in your LLM. That's not a hallucination. That's just what autonomous agents do when there's no guardrail.
The problem is structural. When you write while (!done) and hand control to an LLM, you're trusting a non-deterministic system to know when to stop. Sometimes it doesn't. And unlike a crashed server or a 500 error — a runaway agent keeps working. It looks healthy. It's just spending.
Why "Just Set max_iterations=10" Doesn't Cut It
I've seen this answer come up a lot. It makes sense at first glance — cap the loops, problem solved.
But here's what it doesn't cover:
1. Cost isn't uniform across iterations.
One step might cost $0.003. Another might cost $3.00 depending on the model, the prompt size, and what tools were called. Iteration count tells you nothing about money.
2. Loops don't look like loops.
A stuck agent doesn't always repeat the exact same call. It might slightly permute the prompt each time — same semantic intent, different tokens. max_iterations catches the obvious case. Loop signature detection catches the real ones.
3. Multiple services, multiple agents.
The moment you have more than one agent running — across services, across team members, across environments — where does your max_iterations config live? In a .env file in each repo? Good luck keeping those consistent when you push a hotfix at 2am.
4. Nobody can see what's happening.
Your console.log statements don't survive a process restart. Your finance lead can't query them. Your teammate who shipped a different agent can't see if there's a pattern emerging across runs.
The iteration cap is duct tape. It works for one developer protecting one script in a weekend project. It doesn't work for a team shipping agents to real users.
The Deeper Issue: Agents Are Actors, Not Functions
This is the thing that took me a while to fully internalize.
When you call a normal function, it runs, it returns, it's done. You have deterministic cost. Predictable behavior. Easy to reason about.
An autonomous agent is different. It's not a function call — it's a process that makes decisions. It decides what tools to call, in what order, how many times. It decides when it thinks the task is done. You set the objective. The agent figures out the path.
That's the whole point of agentic AI. That's why it's powerful.
But it also means the cost model is completely different. You're not paying per-call anymore. You're paying for a sequence of decisions you didn't make. And if one of those decisions is wrong — a bad retry strategy, a hallucinated tool call, a reasoning loop — you're paying for that too.
We built observability and governance for the old world — where you call an API, you know the cost upfront, and it's done. We haven't fully built it for this new world yet.
What Actually Needs to Exist
When I started thinking about this seriously, I realized there's a specific set of things that need to be true for agents to be safe in production:
Hard budget ceilings per run. Not soft warnings. Not "we'll alert you". The agent stops when it hits the ceiling. With a partial result. Before more money is spent.
Loop detection at the semantic level. Not just counting iterations — actually detecting when the agent is spinning on the same reasoning pattern.
A kill switch that lives outside your code. If your agent is running in service-A, you shouldn't have to redeploy service-A to stop a runaway run. The kill switch should be callable from a dashboard, an API, a webhook — anything.
Persistent audit logs. Every step, every tool call, every decision — logged somewhere that doesn't disappear when the process exits. Not for debugging. For governance. So you can ask "what did this agent actually do, and why?"
Policy that's centralized. Not per-service config files. One place where you define the rules, and every agent your team ships inherits them.
That last one is the key insight. The difference between "add a budget library to your project" and "have actual governance" is whether the policy lives above the code or inside it.
The Analogy That Made It Click for Me
Think about how Datadog relates to your application metrics.
You could instrument your app with Prometheus locally and write metrics to stdout. That works for one app. But the moment you have multiple services, you need a layer above your code — a place where metrics from all of them converge, where you set alerts, where your whole team can see what's happening.
That's the shape of what agent governance needs to look like. Not a library you add to each project. A control plane that sits above all of them.
Library runs in your process. Stops when your process stops. Lives in one repo.
Control plane runs above your code. Persists across deploys. Spans every agent your team ships.
I Built Something in This Space — Looking for Design Partners
I've been working on exactly this problem. It's called Thskyshield — a runtime governance layer for autonomous agents.
The idea is simple: you wrap your agent loop with three SDK calls — beginRun, beforeStep, afterStep — and the control plane handles the rest. Hard budget ceilings, loop detection, kill switch, step-by-step audit trail. Sub-10ms enforcement (so it doesn't add latency to your actual LLM calls). Works with LangGraph, CrewAI, OpenAI Agents SDK, or whatever you're using.
The policy lives in your dashboard, not in your code. Change a limit without touching a deploy. See what every agent your team shipped actually did.
const run = await shield.beginRun({
agent: 'research-agent',
budgetLimitUsd: 2.00,
iterationLimit: 30,
loopThreshold: 5,
})
try {
while (!done) {
const ok = await run.beforeStep({ tool, args, estimatedTokens })
if (!ok.allowed) break
const result = await callLLMAndTool()
await run.afterStep({ tokens, toolResult })
}
} catch (e) {
if (e instanceof ShieldKilledError) {
console.log(`Stopped: ${e.reason}. Spent: $${e.spent}`)
}
}
Five lines. Drop into any agent loop.
MVP is live. I'm actively working with a small group of teams to shape the agent SDK — specifically want to talk to people who are shipping real agents and hitting real problems. First five design partners get it free forever.
If you're building in this space and this problem sounds familiar — I'd genuinely love to talk. Drop a comment or reach out at thskyshield.com/contact.
The agent era is real. The tooling to govern it safely is still catching up. That gap is what I'm working on.
If you're building agents in production and haven't thought about this yet — now's the time.
#ai #agents #llm #devops #opensource #typescript #buildinpublic #webdev
Top comments (0)