DEV Community: Babar Hayat

Cost Governance for AI Agents: From Detection to Enforcement

Babar Hayat — Tue, 18 Aug 2026 19:23:56 +0000

Your AI agent is burning money. It hasn't crashed, it's running fine, and the only way you find out is when the bill shows up.

Here's the pattern: an agent gets confused and calls the same tool over and over. Or a model upgrade quietly triples the cost per token. Or one bad execution fires a hundred times in parallel. From the SDK's point of view, every one of those is a success. Status 200, output returned, nothing to flag.

We think about cost governance as three layers, and they build on each other.

Detection

You need per-execution visibility, not a monthly total. Token count, cost in dollars, latency, per run. That's how you find out which agent burned $500 yesterday instead of which month burned $10k.

Diagnosis

An agent that costs $5 one day and $500 the next isn't random. Either the cost per call spiked (something's asking for way more tokens than usual) or the call count spiked (it's looping). Knowing which one happened tells you what to actually fix.

Enforcement

This is the part most teams skip. Once you know an agent's normal cost profile, you can pause it automatically if it breaks that pattern hard enough, or if it crosses a budget you set. That check runs before the next API call goes out, so it stops the spend rather than just reporting it after the fact. It fails open too, so a network hiccup on the check itself never breaks your agent's normal operation. And restarting it once you've found the root cause takes seconds.

Detection and diagnosis should always be on. Enforcement is opt-in, because the policy is yours to set. One team wants a hard stop. Another wants an alert and a human in the loop.

None of this works without seeing every execution, though. Sampled metrics will miss the loop that cost you a thousand dollars in five minutes.

If you're running agents in production, the cost story shouldn't be a surprise invoice weeks later. It should be a dial you can turn today.

We Built Monitoring Into Our Own AI Agents. Here's What We Learned.

Babar Hayat — Mon, 17 Aug 2026 17:30:42 +0000

We run marketing workflows on an AI agent. When we tried to monitor it with existing tools, we found ourselves flying blind in ways we didn't expect.

The Problem We Didn't Know We Had

Three months ago, our marketing agent was supposed to draft social posts every morning. One Tuesday, it hadn't. We checked the logs. No errors. The LLM call succeeded. HTTP 200. The response came back. And yet: no draft. Just a blank.

This is the failure mode that most observability tooling doesn't catch. The agent executed. The infrastructure said "success." But the agent produced nothing — zero output tokens, an empty response, a request that returned 200 OK but accomplished zero.

We had to manually check the execution history to find it. By then, someone else had noticed the absence.

That's when we realized: we couldn't see what was actually happening in our own agents in real time. We needed to build visibility from scratch.

Starting from First Principles

We asked: what do we actually need to know about an AI agent execution?

Not just "did it error?" — because errors aren't the only way an agent fails. We needed:

Did it run at all? (HTTP status, latency)
Did it consume what we expected? (tokens, cost)
Did it produce anything? (output length, not just "success" status)
Did it do the right thing? (the harder one — we'll come back to this)

Most APM tools watch infrastructure: request latency, error rates, dependencies. None of those answer "was the output empty?" They watch the wrapper, not what happened inside.

So we wrote an SDK wrapper for our agents. Simple design:

Intercept the LLM client calls (OpenAI, Anthropic, etc.)
Capture the telemetry: tokens in/out, cost, latency, and — crucially — the output itself
Send it somewhere we could query and alert on it

The SDK runs inside our own environment, not between us and the LLM provider. Your API keys stay in your process. We only ever see the telemetry.

That design choice matters: it means we can capture what the model actually returned, but we can't see your proprietary prompt or system instructions. Read-only observability.

What We Built (And Why)

The first version just captured raw metrics. We logged tokens consumed (input/output), cost calculated from per-model pricing, latency, status (success/failure/timeout), model name (so we could auto-calculate cost from a built-in rate table), and a summary of the output.

We pushed it to a dashboard and set up basic alerts: if cost spiked, flag it. If latency crossed a threshold, flag it.

This caught infrastructure problems. It didn't catch silent failures.

So we added silent-failure detection: if status is "success" but output_tokens equals 0, that's an alert. The agent ran. It returned 200. It produced nothing.

We caught three more of them within a week using this rule alone.

Then came the harder question: what if the output looks fine but is actually wrong?

An agent returns a well-formed response. Tokens flow normally. Cost is where we'd expect. But the output is garbage — a recommendation that doesn't match the input, a calculation that's off, a response that's just not what was asked for.

No metric catches that. No baseline can. You need a human (or another model) to grade it.

The Correctness Problem

We built a "correctness check" feature: you write a rubric in plain English describing what a correct output should look like ("The response must include a specific name and a recommendation. The recommendation must be actionable."). Then we run every execution's output past an AI grader against that rubric.

The grader isn't perfect. That's the whole point of making it reviewable. On every alert, we let you give a thumbs-up or thumbs-down: "did the judge get this right?" Over time, that feedback becomes a record of accuracy.

And if the judge is systematically wrong about something, you can flag a disagreement and add it as an example the judge learns from. The correction is always manual — we don't auto-add examples from feedback — so a bad rubric can't teach itself to fail.

This catches the "200 OK but completely wrong" case. It doesn't solve the problem of an imperfect judge. It just makes the imperfection visible and reviewable, rather than hidden.

What We Learned

The infrastructure lies to you. HTTP 200 is not a guarantee of work done. Latency is not a guarantee of usefulness. Error rates don't tell you about silent failures. You need eyes on the actual output.

Secrets stay secrets. You can have observability without sharing your API keys or your prompt logic. An SDK that runs in your own environment sees everything your code sees, but only sends us aggregates and summaries. That's the design that lets us help you without breaking your security model.

The easiest failures to miss are the ones that don't error. An agent that times out, an API that 500s — those are obvious. An agent that returns 200 with no response? That's what sits in your queue unnoticed until a customer tells you.

Correctness is a human problem with AI help. You can't write a metric that "correctness" is. You can write a rubric and ask another model to grade it, and you can let humans correct the grader. The system becomes useful when the correction loop is tight and transparent.

The Gaps We Hit

When we were building this, we looked at what existed:

Application performance monitoring tools watch latency, errors, and infrastructure — not AI-specific problems like silent failures or cost anomalies.
LLM observability tools (the newer ones) watch tokens and cost, which is closer — but most don't distinguish between "agent returned success with no output" and "agent returned success with output." The metrics look the same.
Logging frameworks let you log whatever you want, but they don't automatically flag patterns. You're reading through logs manually or writing custom rules.

We needed something that:

Auto-detected which model you were using (so cost was automatic, no config)
Flagged silent failures specifically (not just errors)
Didn't require you to share your API keys with us
Let you define what "correct" means and alert when it's violated
Gave you real-time visibility without slowing down your agent

That's what we built. And we're eating our own dogfood: this very marketing system runs on an AI agent that we monitor with our own product.

What Comes Next

We're learning as we use it. The silent-failure detection is solid — it catches what the infrastructure can't see. The cost anomaly detection helps catch runaway loops. The correctness check is useful but imperfect by design: we don't claim to know what "correct" means for your use case, only to help you define and enforce it.

The thing we're still figuring out: how to make the correctness feedback loop even tighter. Right now, if the judge disagrees with you, you can promote that disagreement into a training example. But how many examples does the judge actually need to improve? When does a rubric become accurate enough to trust? How do you know when you've taught it enough?

Those are open questions. We're answering them live, with real agents, real failures, and real feedback from the builders using the system.

If you're building AI agents and you're wondering whether your monitoring is good enough, ask yourself: could you spot a silent failure in your own agent right now? Not an error — just an execution that returned success but produced nothing?

If the answer is "not without checking the logs manually," you've found the gap we're trying to close.

Design Patterns for Budget Enforcement in AI Agents

Babar Hayat — Sun, 16 Aug 2026 11:33:28 +0000

When cost control becomes a design decision

You set a monthly budget for your AI agents. Then an agent hits it on day 3. Do you pause the whole system, just that agent, or wait and see?

This isn't a rare edge case. As AI agents scale, especially agentic loops with tool calling, reasoning chains, or retry logic, cost can climb in ways nobody predicted. A single runaway execution can burn weeks of budget in an afternoon. So most teams want a kill switch. The hard part isn't wanting one. It's deciding which kind.

The three enforcement patterns, and what they cost you

Pattern 1: Org-wide budget ceiling

Set a monthly limit for the entire organization. Hit it, and every active agent pauses at once.

The logic: maximum safety. One team member spinning up an expensive agent can't surprise the whole company. The cost stops immediately.

The cost: it's blunt. If one agent loops and burns the budget on day 3, your production agent, the one paying customers actually depend on, stops working too. That's an outage with no warning. You're trading uptime for certainty.

When it makes sense: you have a hard cap you will not cross under any circumstances (compliance, a fixed budget line, a card-on-file limit), and your agents either aren't customer-facing or an outage is genuinely tolerable.

Pattern 2: Per-agent daily or monthly limit

Each agent gets its own budget. If Agent A burns through its allocation, only Agent A pauses. Everything else keeps running.

The logic: blast-radius containment. The expensive agent stops; the reliable one doesn't.

The cost: you have to set per-agent limits correctly. Too tight, and a legitimate spike (a genuinely complex query) triggers a false pause. Too loose, and the agent still runs expensive for weeks before hitting its limit. And there's a delay built in — if an agent's daily limit is $50 and it burns $60 on day one, you don't know until tomorrow's review.

When it makes sense: your agents have wildly different cost profiles, you want failures isolated from each other, and you can tolerate a day or two of overage while you investigate.

Pattern 3: Cost anomaly detection

An agent's cost stays reasonable until it doesn't. Track its 30-day baseline. If three consecutive executions each run 3x that baseline, pause it.

The logic: you're catching deviation, not absolute spend. An agent that normally costs $0.05 a run but suddenly costs $0.15 is telling you something's wrong, a hallucination, a retry loop, unexpected recursion, and you catch it without needing to guess the "right" budget in advance.

The cost: lag. The detector doesn't fire until the third expensive execution. If a run costs $10 against a normal $0.05, you've already spent $30 before the kill triggers. It's also noisy during ramp-up — a new agent with no baseline yet can produce false alerts until it stabilizes.

When it makes sense: you trust your agents' normal behavior and mainly want to catch unexpected shifts, and you'd rather tolerate a few expensive runs than deal with false positives.

The tradeoff every team hits

All three patterns share the same tension: safety versus responsiveness.

A kill switch that fires instantly (org-wide budget) is safest but dumbest, it can't tell a legitimate spike from a real problem. One that waits for a pattern (anomaly detection) is smarter but slower, the first runaway has already cost you money by the time it reacts.

There's no universally correct answer, because the answer depends on what actually breaks your business. If an unexpected $5k charge would get your card declined, you want the tight, fast control. If it would just be absorbed but wasteful, you want the pattern detector. If your agents have genuinely different cost profiles, per-agent budgets let you tune each one independently.

How to design your own

Know your agents' baseline costs. Run them in production, or close to it, for a week. Calculate the median cost per execution and the 95th percentile. If Agent A costs $0.10 and Agent B costs $5, that gap is your signal that they need different limits.

Define your pain threshold. Ask: what's the largest unexpected bill we could absorb without it being a crisis? If the answer is $100, set your org-wide limit there, or slightly under. If the answer is "it depends on the agent," use per-agent limits instead.

Pick your trigger type, and be honest about the lag. An org-wide limit fires immediately but affects everything. A per-agent limit fires once that agent crosses its cap, with maybe an execution or two of delay depending on your check interval. An anomaly detector expects the first few expensive runs to happen before the 3-sigma rule trips — it's built for catching trends, not isolated spikes.

Decide what happens when the kill fires. Does the agent stay paused until someone restarts it, or does it auto-resume after a fixed time? Most teams prefer a manual restart. It forces you to actually look at what happened instead of letting the same bug retry itself.

One more reality check

No kill switch prevents the first expensive execution. Set a $5 per-agent limit, and the agent that costs $20 still runs that one $20 execution before the limit kicks in. The switch is a brake, not a prevention system. It stops the bleeding, not the first wound.

If you need to prevent expense before it happens, that's a different layer entirely: prompt guards, token limits, output truncation. The kill switch works alongside those, not instead of them.

The practical move

Start with per-agent budgets if your agents have different cost profiles. It's granular without being overwhelming. Add anomaly detection a month in if you find yourself constantly resetting false positives.

If you're early and your agents are all similar, an org-wide budget is simpler, at least until one agent's behavior starts to diverge from the rest.

And actually measure what happens. Check whether your kill switches fire for real problems or false alarms, and adjust the thresholds. The design that works isn't the one that looks best on paper. It's the one that catches real runaways and lets legitimate spikes through.

We build this into how we monitor our own AI agents at agents.opsveritas.com — org-wide budgets, per-agent limits, and cost anomaly detection, all with a manual-restart kill switch.

Why Your LLM Agent Monitoring Is Blind to Empty Outputs

Babar Hayat — Sat, 15 Aug 2026 14:26:09 +0000

You wrap your OpenAI call in a try-catch. The API responds with HTTP 200. Your monitoring logs success. Your dashboard shows green. Twelve hours later, a customer emails: why did my request return nothing?

This is a silent failure — the detection gap that almost every monitoring setup misses by default.

The problem: what "success" actually means

When you instrument an LLM agent, most observability tools watch for one thing: did the API call fail?

try:
    response = client.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": query}]
    )
    # Success — log it
    log_to_monitoring(response)
except Exception as e:
    # Failure — log and alert
    log_to_monitoring(error=e)

The catch block fires on network timeouts, authentication errors, rate limits, malformed requests — the visible failure modes. But there's a whole class of failures that return HTTP 200 anyway:

The API call succeeds.
The response object is syntactically valid.
But response.choices[0].message.content is empty or whitespace-only.

The monitoring layer sees the response code and declares victory. The agent proceeds with nothing to work with. Your system continues silently.

Why this happens (and why it's surprisingly common)

Empty outputs aren't always bugs. Sometimes they're hints of a real problem:

The model hit an internal limit. Certain Anthropic models or OpenAI beta versions will return 200 with an empty completion if the output is too long for the context window after token counting, or if the model's internal filtering catches the request. The API succeeded; the inference did not.
Prompt injection or safety filtering. A user prompt contains adversarial input that the model's safety classifier flags. The model refuses to respond, returns 200, and sends back an empty string — by design. Your monitoring sees 200 and thinks it's fine.
Tool-call-only agents. An agent framework (LangChain, CrewAI) is configured to return tool calls only, never direct text. The model returns [{"type": "tool_use", "name": "...", ...}] with no text content in message.content. Strictly speaking, the agent did run — but if your monitoring only looks at content, it sees empty and logs it as a failure when it was actually correct.
Provider parsing failures. A non-standard provider or a custom SDK wrapping an OpenAI-compatible endpoint garbles the response parsing. The outer call returns 200 (the wrapper succeeded in getting a response), but the actual content extraction fails and returns null.

All of these return HTTP 200. All of them are invisible to standard error traps.

What observability layers typically miss

Let's map the layers of a typical LLM observability stack and see where the gap lives:

Layer	Watches	Catches HTTP 200 + empty?
Try-catch / error handler	Exception objects	No — no exception raised
HTTP status codes	4xx, 5xx, timeouts	No — 200 is success
Token counts	Input tokens captured	Depends — output tokens = 0 or missing?
Response parsing	JSON validity	No — empty string is valid JSON
Logging middleware (OpenTelemetry, etc.)	Request/response metadata	Only if you explicitly log content length
LLM-specific SDK (e.g., LangChain callback)	Message count, tool calls	Only if `message.content` is explicitly checked

The gap: none of these layers default to checking whether the actual output is non-empty. They all pass the call through as "success" because, by the narrow definition of "did the API succeed," it did.

How to detect it: the mechanics

To reliably catch empty outputs, you need to add an explicit check in the instrumentation layer — a guard that inspects the content, not just the status.

Here's the pattern:

import openai
from opsveritas import opsveritas

# Initialize monitoring
opsveritas.init(secret="your-secret")

# Wrap the client — this instruments tokens and cost automatically
client = openai.OpenAI(api_key="...")
wrapped_client = opsveritas.wrap(client)

# Make the call
response = wrapped_client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Translate this to French: ..."}]
)

# Now the explicit check — capture the output and test it
output_text = response.choices[0].message.content or ""
output_is_empty = not output_text.strip()

# Log or alert
if output_is_empty:
    print("SILENT FAILURE: HTTP 200 but empty output")
    # This is the signal your monitoring should catch
else:
    print(f"Success: {output_text}")

The SDK here captures tokens and cost automatically. But the empty-output check has to live in your code, right after the response lands. Why? Because only you know what "empty" means for your agent — it might be zero characters, or it might be a response that's just whitespace, or it might be null. The infrastructure can't make that judgment.

For agents running in frameworks like LangChain or CrewAI, the same pattern applies — you wrap the model, then add a post-call assertion:

from langchain.chat_models import ChatOpenAI
from opsveritas import opsveritas

model = ChatOpenAI(model="gpt-4")
model = opsveritas.langchain("my-agent")(model)

result = model.invoke({"query": "..."})

if not result or not str(result).strip():
    # Alert: silent failure detected
    ...

The SDK sends telemetry (tokens, cost, latency). But the empty check is application logic — it has to live where the response is handled.

What happens at the monitoring backend

Once you've added that check, here's what a proper monitoring backend should do:

Receive the telemetry — tokens, cost, status, and a flag for empty output.
Classify the failure — status: success but output_tokens: 0 or content_length: 0? That's a silent failure, not a normal success.
Alert separately — silent failures are different from timeout failures or rate-limit failures. Your runbook for each is different. A silent failure might mean "retry this agent with a different prompt" or "check the safety filter logs." A timeout means "increase concurrency limits."
Track baseline — over time, the monitoring system learns this agent's normal output-token distribution. If output suddenly drops to zero for 3 runs in a row, that's a regression signal.

Why this matters for cost and reliability

Silent failures compound two problems:

Cost: An agent that runs repeatedly, returns nothing, and never triggers an alert keeps calling your LLM API. With millions of requests a day, even one silent-failure loop can burn thousands in cost — because the loop keeps retrying, each retry succeeds (200), and each success is invisibly empty.
Reliability: A customer-facing system that silently returns empty results erodes trust before you even know there's a problem. By the time complaints arrive, you're debugging backwards through hours of logs trying to spot the failure that your monitoring said never happened.

The fix isn't complicated — but it requires two pieces:

Explicit output checks in your agent code (one line per agent).
Monitoring that understands empty output as a distinct failure mode (not lumped in with general "success").

Neither is rocket science. But most setups skip both, which is why HTTP 200 + empty output remains one of the industry's favorite silent killers.

Have you hit this? Silent-failure loops that your monitoring missed because they all returned 200? The earlier you catch them, the less cost and reputation they burn. The gap is real — and closeable.

More on how we built this detection into our own monitoring: agents.opsveritas.com

Two of Ten Items Ran, and the Log Said Success: A Guide to Spotting Batch Silent Failures

Babar Hayat — Thu, 13 Aug 2026 05:30:11 +0000

Your batch agent processes 10 items. The logs show success. But only 2 actually ran. The other 8? They vanished, without error, without trace, without the system ever telling you something went wrong.

Why this happens

Batch processing is where silent failures breed. Here's the mechanism:

Your agent loops over an input list. It calls a model (or a chain of tools) for each item. If the model returns an empty response, HTTP 200, but zero output tokens, the agent has a choice: explicitly handle that empty case, or skip it and move to the next item.

Most codebases skip it. The loop continues. No exception is thrown. The run completes with a "success" status because the loop itself completed without crashing. From the outside, everything looks fine. From the inside, 80% of your work never happened.

The trap: success at the orchestration level does not equal success at the execution level.

How to spot it in your logs (without a dashboard)

You don't need monitoring infrastructure to catch this. You need reasoning plus your existing logs.

Step 1: Count what you expected vs. what ran

Start here:

Input: [item_1, item_2, item_3, item_4, item_5, item_6, item_7, item_8, item_9, item_10]
Expected executions: 10

Now grep your logs for the actual model calls, the real token-generating events. Look for patterns like:

OpenAI API call or Anthropic API call or your model provider's log marker
tokens_sent, output_tokens, or whatever your SDK/client logs
Anything that shows "a model was asked and answered"

Count them. If you see 2 real model calls but 10 input items, you already have your problem.

Step 2: Trace the gap, which items were processed?

Extract the item identifiers from the successful model calls. For example:

Call 1: processed item_2 → 120 output_tokens
Call 2: processed item_7 → 98 output_tokens

Items processed: {item_2, item_7}
Items missing: {item_1, item_3, item_4, item_5, item_6, item_8, item_9, item_10}

Now look at the logs around each missing item. Search for:

Did the loop reach item_1? (Look for log lines like "processing item_1" or similar.)
If it reached the item, did it call the model?
If it called the model, what was the response? Was it empty? Did it include output tokens?

You're looking for the place where the logic broke. Usually it's one of these:

Scenario A: The loop never reached the item. Your batch input was truncated, filtered, or partially failed before iteration even started. Less common, but check your batch construction code.

Scenario B: The loop reached the item, but skipped calling the model. Your code has a conditional: if item.valid(): call_model(), and several items failed validation silently. Or a fallback triggered without logging it visibly.

Scenario C: The loop called the model, but the model returned empty output. This is the silent failure. The model returned HTTP 200, but output_tokens == 0 or the response body was blank/whitespace. Your code saw the empty response and either returned it as-is or skipped it without alerting you.

Step 3: Check token math

For each item that did run, log the input and output tokens:

item_2 input: 450 tokens → output: 120 tokens (good, output exists)
item_7 input: 480 tokens → output: 0 tokens (empty response)

If an item has input but output equals 0, that's a silent failure. The model was called, it returned success, but it produced nothing.

Check your code: does it handle this case explicitly?

# Without explicit handling (the dangerous version):
for item in batch:
    response = model.call(item)
    results.append(response)  # if response is empty, it still gets appended as "nothing"

# With explicit handling (safer):
for item in batch:
    response = model.call(item)
    if not response or response.tokens == 0:
        log_alert(f"Empty response for {item}")
        results.append({"error": "empty_response", "item": item})
    else:
        results.append(response)

The first version swallows the failure. The second surfaces it.

Step 4: Reason backward to the cause

Once you've identified which items failed (and how), ask:

Why did the model return empty?

Rate-limited? (Check for 429 errors in the logs, or long latency gaps.)
Malformed input? (What was actually sent to the model for that item? Log it.)
Model-specific quirk? (Some models return 200 with empty output on certain inputs; check your model's behavior.)
Timeout or partial read? (Did the connection drop mid-response?)

Why didn't the code catch it?

No explicit check for output_tokens == 0?
No retry logic?
The result was treated as valid because the API call didn't throw an exception?

Most of the time, the root cause is: your code assumes that HTTP 200 means success, and never checks whether actual output was produced.

What to do right now

Add explicit output validation. After every model call, check output_tokens > 0 (or whatever means "real output" for your use case). Log failures.
Count your batch completeness. Before shipping, compare input count to output count. If they don't match, you have a silent failure.
Log item-level decisions. Every time an item is skipped, processed, or returns empty, log it with the item ID. Then grep your logs later with full visibility.
Retry on empty. If an item returns empty output, retry it once or twice before giving up. Log the retry. Most transient failures clear on retry.

The deeper lesson

Batch processing is high-leverage and high-risk: process 100 items and one failure hides in plain sight. Process 1,000 and you won't notice until a customer complains or a metric suddenly drops.

The mechanic is always the same: success at the wrong layer. The orchestration layer (the loop) succeeds. The execution layer (the model call) fails silently. The mismatch never bubbles up because nobody's looking at both layers together.

This is why builders who catch these failures early don't rely on "the system will tell you if something's wrong." They instrument the intersection: input count, output count, per-item token accounting, and explicit checks for empty responses.

Once you reason through this mechanism, you can spot batch silent failures in your logs in minutes. It's not mysterious. It's just accounting.

Have you hit this in a batch job? What made you notice? Drop a comment, I'd like to hear what surfaced it.

The Hidden Math: Why "Cheapest Model" Doesn't Mean Cheapest Execution

Babar Hayat — Wed, 12 Aug 2026 10:20:37 +0000

The Hidden Math: Why "Cheapest Model" Doesn't Mean Cheapest Execution

You're building an AI agent. Smart cost strategy: route to GPT-4o when you need reasoning, Haiku for simple classification, Groq when it's available and fast. In your head, the math is simple: pick the cheapest model per task.

In practice, you ship code and never really know if it worked.

Here's the gap: cheapest model ≠ cheapest execution.

A single agent run might span multiple model calls. One task uses Anthropic's Sonnet ($3 / 1M input, $15 / 1M output). Another uses OpenAI's GPT-4o mini ($0.15 / 1M input, $0.60 / 1M output). A third falls back to Groq Llama 3.1 when latency spikes. By the time the execution finishes, you have no idea which model was actually called, how many tokens each burned, or whether that "cheap" fallback actually saved money — or whether a long output summary from one call inflated the token count beyond what you budgeted.

The assumption breaks immediately.

Why the math matters (and why it's invisible)

Token costs compound differently by model.

Take a real scenario: your agent needs to summarize a customer support transcript (2,000 input tokens). You budgeted for Haiku: 2,000 tokens × $0.80 / 1M = $0.0016 input cost. Cheap.

But Haiku hits a rate limit. Code routes to Sonnet as fallback. Same 2,000 input tokens, now $0.006 — nearly 4× more. Multiply that across 100 daily runs, and you've overrun your mental budget without knowing.

Output lengths are invisible.

Worse: you don't even know the output token count. An agent that "completes" a task could've produced 500 output tokens (brief response) or 5,000 (verbose reasoning). With Sonnet outputting at $15 / 1M, that's the difference between $0.0075 and $0.075 per run. Neither is huge. But if your agent runs 1,000 times a month and half your executions unexpectedly verbose?

You're now $37 over budget without seeing why.

Multi-model routing compounds the blind spot.

When you route across three providers with six fallback models, you lose the ability to reason about cost at a glance. Did GPT-4o actually run, or did it fail and drop to Sonnet? How many times did Groq get chosen? Which model produced the longest outputs? You shipped it all, but you're flying blind on the arithmetic that actually happened.

The unsexy solution: measure and account

Here's what works (and it's not flashy):

1. Log every model call with input + output token counts.

Before you route, log which model was selected. After the call returns, log the actual token counts from the response metadata. Not a guess — the real numbers.

Why: your mental model of "mostly cheap, sometimes fallback" gets replaced with actual data. You can group by model, see fallback rates, and measure whether your routing actually favors the cheap path.

2. Compute cost inline using per-model rates.

Every model has a published price per 1M tokens (input and output separate). Compute cost at execution time using actual tokens.

For example:

Input: 2,000 tokens
Output: 1,500 tokens
Model: Anthropic Sonnet ($3 / 1M input, $15 / 1M output)
Cost: (2,000 / 1M × $3) + (1,500 / 1M × $15) = $0.006 + $0.0225 = $0.0285 per run

Sixty runs a day? That's $1.71 / day, $51 / month. You now know the actual stake.

3. Bucket by model + routing path.

At the end of each day (or run batch), group executions by which model actually ran and whether it was your first choice or a fallback. Count, sum costs, measure fallback frequency.

If Groq was supposed to be your primary but it failed 20% of the time, you now have evidence to either increase fallback tolerance, switch providers, or accept that Sonnet is your real cost center.

4. Alert on outliers.

A single execution shouldn't surprise you. If one run's cost is 3σ above the 30-day average for that agent, you want to know why: did it produce an unexpectedly long output? Did it hit a fallback you didn't expect? Was there a retry loop?

The math stops being invisible.

What this buys you

You move from "I think this is cheap" to "I know what this costs, per execution, per model, per routing path."

That's the difference between budgeting in hope and budgeting in fact.

The math isn't exciting. But it's urgent, because runaway cost is usually a routing or retry problem hiding in production, and it stays hidden until the bill lands.

The Silent Failure Detection Framework: Catching Agents That "Succeed" and Do Nothing

Babar Hayat — Mon, 10 Aug 2026 14:01:59 +0000

Your LLM agent returned a response. No error, no exception. But did it actually do what you asked?

That's the silent failure problem. The system behaves normally — HTTP 200, status success — but the output is empty or nonsensical. Nothing alerts you. The customer complains first. To catch these, you need to know what makes a silent failure detectable in the first place. Here's the framework.

What you're actually hunting

A silent failure is execution that reports success but produces zero (or useless) output. The agent didn't crash — it just did nothing. The most common pattern:

Agent calls the model.
Model responds with ~0 output tokens, or a blank response.
No error is raised.
The calling code proceeds, unaware.

Two other variants show up often: an agent looping indefinitely without progress (tool calls that never advance state), and output that's structurally valid but semantically empty — "I don't know" when it should have searched.

Standard error monitoring misses all three, because there is no error. You're checking the wrong signal.

The three detection levers

1. Token counting — the baseline signal

Every LLM call produces input tokens (your prompt) and output tokens (the model's response). Measurable, deterministic, available in the response metadata from every provider.

The insight: a healthy agent call has a predictable token range, per agent. A ticket-summarizer might consume ~100-200 input tokens and produce ~50-150 output tokens, run after run. If an execution comes back with 100 input tokens and 0 output tokens, that's a signal worth flagging — not proof of failure on its own, but worth alerting on.

How to implement it: log input/output tokens for every execution, compute the median and standard deviation of output tokens over the past 30 days, flag anything below the 10th percentile or exactly zero.

baseline_median_output = median(past_30_days_output_tokens)
baseline_std = stdev(past_30_days_output_tokens)
is_anomaly = (today_output_tokens < baseline_median - 2*baseline_std) OR (today_output_tokens == 0)

2. Cost anomaly — a derived signal

Cost = input_tokens × input_rate + output_tokens × output_rate. When token consumption drops, cost drops with it. Cost is often easier to track than raw tokens because it's vendor-independent — you can aggregate across models.

If your agent normally costs $0.02-$0.05 per run and today it's $0.0001, something's very wrong. Same approach: baseline median/std over 30 days, alert below the 5th percentile. Cost lags token anomalies slightly since it's derived, but it's the easier number to reason about in business terms.

3. Output examination — the confirmatory signal

If tokens suggest an anomaly, the next question is what the model actually said. Store a short excerpt of the output (not the whole response, that's usually too large), and you can scan for empty strings, repetitive/looping output, or run a secondary AI judge against a correctness rubric.

Capture the first 500 characters, flag anything empty or suspiciously short against your baseline. Rubric-based judging is optional but valuable for high-stakes agents.

Putting it together

Every execution logs input_tokens, output_tokens, cost_usd, output_summary, executed_at.
On a schedule (nightly, or hourly at volume), compute per-agent baselines and flag anything below threshold.
On a flag: alert the engineer with the output summary attached so triage is immediate, and optionally auto-pause the agent if the cost anomaly is severe.

Why this works, and where it doesn't

Strengths: no false positives from error logs, since you're measuring execution quality directly rather than exceptions. Early signal — tokens are real-time, so you catch problems in minutes. Framework-agnostic — LangChain, CrewAI, raw SDK, every model call produces tokens. Low friction — logging tokens is a few lines of code.

Limitations: you need baseline history (a brand-new agent with five executions has no reliable range yet), the technique is context-dependent (agents with legitimately wide output variance need a wider baseline), and detection isn't prevention — flagging a failure doesn't stop the next expensive call, that needs an actual pause or rate limit on top.

The practical next step

Log tokens — add input_tokens, output_tokens, cost_usd to every execution record.
Compute a baseline once you have a week or two of history.
Set an alert on the 10th percentile of output tokens, or output_tokens == 0.
Triage once. When it fires, you'll either see a genuine empty response (actionable) or a legitimate edge case you exclude next round.

You won't catch every silent failure this way. You'll catch the common ones — the ones that hurt most because they're invisible until someone notices.

If you'd rather not build and maintain this plumbing yourself, that's the layer we built into the AI Agents Control Tower — token tracking, baselining, and anomaly alerting, so you define thresholds instead of infrastructure. Either way the principle holds: measure the tokens, establish the baseline, alert on the gap. Silent failures are only silent until you start listening.

Why Frameworks Don't Teach You How to Know If Your Automation Actually Worked

Babar Hayat — Sun, 09 Aug 2026 11:06:32 +0000

Frameworks like n8n, LangChain, and CrewAI are brilliant at teaching you how to build automations and agents. They're almost entirely silent on how to know if they worked.

This isn't an accident. It's a scope choice. And it's leaving builders stranded.

The assumption framework designers make

When you build a workflow in n8n or a multi-step agent in LangChain, the framework is optimizing for one thing: authoring. How do you express the logic? How do you chain steps? How do you test locally? Every tutorial, every example, every feature set is built around one question: can I describe what I want to happen?

Frameworks answer that brilliantly. You can build sophisticated, multi-step workflows in hours. You can define agents that reason, call tools, and loop. The developer experience is real.

But then you deploy it.

Where the scope ends

The moment your automation or agent goes live, the operating model switches. You no longer care how the logic was expressed, you care whether it actually did what it was supposed to do right now, in production, with real data, against real external systems.

This is where the framework's view of the world stops.

Look at the n8n docs. Search for "how do I know if a workflow succeeded?" You'll find status badges on the UI (green means success), logs you can click through if you log into the dashboard, and error messages if something exploded loudly.

Look at the LangChain docs. Search for "how do I monitor an agent in production?" You'll find examples of .invoke() calls in a notebook. Nothing about continuous visibility. Nothing about the silent failure class: an agent that returns success but produced empty output.

Look at CrewAI. Same story.

The blindspot is structural

Here's the key insight: frameworks optimize for the author's perspective, not the operator's.

An author needs to know if their logic executed. Yes, and the logs show the steps. An operator needs to know if it did the right thing, which is a completely different question. These require different instrumentation.

Authoring needs visibility into the logic: step-by-step execution, conditional branches, tool calls. Operating needs visibility into the outcome: did the expected side effect happen, did the external system behave, did the cost stay within bounds, did the agent loop.

Frameworks nail the first. They assume you'll handle the second.

What that gap looks like in practice

You ship a workflow that connects your CRM to your email service. It processes 50 contacts a day. The logs show green. Three weeks later, a customer complains they never got contacted. You dig into the logs and find the workflow "succeeded" 21 times. But the email service's API had silently started requiring an authentication header your workflow wasn't sending. The workflow called the API, got a 200 response because the API is polite about failures, and moved on. Zero errors. 21 failures, all silent.

You ship an agent that researches customers and generates summaries. It works perfectly in your test suite. In production, it runs fine for a week, then suddenly starts burning ten times the tokens per run. The logs show successful tool calls. But it's looping, calling the same tool over and over because it's misinterpreting the response. It "succeeds" at high cost until your budget alert fires.

In both cases, the framework did exactly what you asked. The automation authoring was sound. The operating visibility was zero.

It's not a flaw in the framework

This isn't a criticism of n8n, LangChain, or CrewAI. They're designed for a specific job: making it easy to express automation logic. That's genuinely hard, and they solved it.

But that job isn't the job of operations. Operations needs a different set of tools and patterns:

Baseline tracking: what does a normal run look like, in tokens, latency, and success rate
Anomaly detection: what looks weird compared to that baseline
Silent-failure detection: the agent returned success but output was empty
Cost tracking: not just total spend, but spend per agent and per run
Enforcement: if an agent starts looping, stop it before the bill explodes

Frameworks don't provide these because they don't operate the automation. You do.

What builders need to do

The gap isn't that frameworks are bad. The gap is that frameworks assume you'll wire in the operating layer yourself.

A few patterns that close it:

Instrument execution: every time an automation or agent runs, log or send structured telemetry: tokens, latency, status, and a small output sample. Just the facts, not the whole story.
Compare to baseline: after about 50 runs, you know what normal looks like. Cost should sit within roughly 20% of the median. Latency should stay within bounds. Alert on drift.
Catch silent failures explicitly: an automation that returns success but produces zero output, zero tokens, a blank response, is a silent failure. Most frameworks don't flag this. You have to.
Set and enforce limits: define a per-agent token ceiling. If an agent hits it, stop the run. Don't wait for the bill.

The real lesson

Frameworks are optimized for builders who are authoring. The moment you're operating, running that code against real data, at scale, continuously, you've moved into a domain the framework doesn't own.

That's not a flaw. It's a reminder that building and operating are different jobs, and they require different tools and visibility.

If your framework tells you your automation "succeeded," that's authoring feedback. It doesn't tell you if it worked. Figuring that out is the operator's job. The gap between "framework success" and "actual working" is exactly where silent failures hide.

Token budget guardrails: enforcement, not just visibility

Babar Hayat — Sat, 08 Aug 2026 11:39:57 +0000

You've been monitoring your agent's token spend for weeks. Then one run goes wild — 50x the usual token count — and by the time you notice, half your budget is gone.

Visibility is the first step. Enforcement is the second, and it's the one that actually stops the bleed.

The problem: seeing the spike after the damage

Most monitoring setups give you a dashboard. Your agent runs, consumes tokens, and at month end you see the bill. Even with a live dashboard, the response chain is slow: monitor, alert, human, decision, pause. That gap costs real money.

What you actually need is a guardrail that stops the agent from running the moment it crosses a ceiling you defined in advance. Not "alert the team" — "reject the call before it happens."

This matters because of how token explosions actually happen:

Retry loops — an agent keeps calling the same tool because it misreads the response. Each retry burns tokens. A single run can loop 10-50 times before anyone notices.
Context bloat — the agent accumulates conversation history or debug logs in its context window. By run 100, input tokens are 3x baseline.
Hallucinated retries — the model thinks a tool call failed when it actually succeeded, so it tries again. Logs look clean. Cost doesn't.

In every case, the agent still succeeds — HTTP 200. The token spend doesn't.

How to set per-agent token budgets

Start with a baseline. Run your agent 20-50 times under normal conditions and record the token count per execution. This is your reference point — not average monthly spend, but typical spend per single run.

Example: a document-retrieval agent
- Run 1: 3,200 input + 450 output = 3,650 total
- Run 2: 3,100 input + 480 output = 3,580 total
- Run 3: 3,400 input + 520 output = 3,920 total
Average per run: ~3,700 tokens

From that baseline, set two thresholds:

Per-run ceiling. Set it at 2-3x typical spend. Anomalies exist — a genuinely complex query might need more — but 3x is usually where something's actually wrong. If baseline is 3,700 tokens, ceiling lands around 10,000-11,000.

Monthly budget ceiling. Divide your monthly LLM budget by expected run count, then apply a safety margin. Budget $100/month, expect 100 runs — that's $1/run. At roughly $0.002 per 1K tokens for cheaper models, that's about 500 tokens/run. Set the monthly ceiling around $80 (a 20% margin) and enforce it across all agents.

The math is simple. The enforcement is what actually matters.

Implementation: the kill-switch pattern

Once you've defined ceilings, the agent needs to check them before each call, not after.

If you're using the AI Agents Control Tower, this is built in. The kill switch is an opt-in feature that pauses your agents the moment your org crosses its monthly budget:

Enable the kill switch in org settings and define your monthly token/cost limit.
Wrap your LLM client with the SDK's enforcement layer (wrap_langchain for LangChain models) — adds a lightweight status check (~3ms, cached) before each inference call.
If the limit's crossed, the SDK raises OpsVeritasKilledError instead of calling the model. The agent stops. No token burn, no surprise bill.

The kill switch fails open — if the status check itself fails (a network blip), the call proceeds normally. You're protected against silent failures, not against your own infrastructure breaking.

In code:

from opsveritas import init, wrap_langchain
from langchain.chat_models import ChatOpenAI

init(api_key="your-secret")

# Wrap the model for telemetry (which agent ran, how many tokens)
model = ChatOpenAI(model="gpt-4o-mini")

# Add enforcement: blocks the call if budget is exceeded
model = wrap_langchain(model, agent_name="my_agent")

# model.invoke() now raises OpsVeritasKilledError
# if the org's monthly limit is breached

For non-LangChain setups, use the Universal Webhook to POST your agent's token telemetry to the control tower after each run. Include cost_usd in the payload and the system tracks spend in real time across all agents. The kill switch still fires at the org level, but you own the decision of whether to actually call the model.

Per-agent ceilings: a second layer

The org-wide kill switch is insurance. For finer control, add per-agent ceilings inside your agent's own execution loop:

def run_agent(query):
    baseline_tokens = 3700  # your per-run baseline
    ceiling_tokens = 11000  # 3x baseline

    response = model.invoke(query)
    total_tokens = response.usage.input_tokens + response.usage.output_tokens

    if total_tokens > ceiling_tokens:
        log.error(f"Agent exceeded token ceiling: {total_tokens} > {ceiling_tokens}")
        # retry with a simpler query, alert the user, or flag for review
        return {"status": "budget_exceeded", "data": None}

    return response

This layer catches anomalies per execution, not per month. If one run gets expensive, you know immediately, and you can retry, degrade gracefully, or fail loud.

Measuring what matters

Once guardrails are in place, track three metrics:

Tokens per run (p50, p95) — tells you if the baseline is drifting. A climbing trend means the agent is degrading.
Cost per run — token count alone doesn't account for model differences; a gpt-4o run costs differently than gpt-4o-mini.
% of runs hitting the ceiling — if more than 1-2% of runs exceed the threshold, either the ceiling's wrong or the agent has a real problem.

Most monitoring tools show total spend. What you need is per-execution visibility — the individual run that went wild, not just the aggregate bill.

The real lesson

Visibility without enforcement is a dashboard you check after something breaks. Enforcement without visibility is a kill switch that fires mysteriously. You need both.

Set baselines from real data. Define ceilings at 2-3x normal. Enforce before the call, not after. Track per-run metrics, not just aggregate spend. When an anomaly hits, you'll catch it in milliseconds, not at month end.

Cost governance is reliability. Silent token loops are a failure mode just as real as crashes — they just hide longer.

The eval illusion: why passing tests doesn't mean safe in production

Babar Hayat — Fri, 07 Aug 2026 06:04:18 +0000

You built an AI agent. The reasoning holds up — your eval suite checks that it picks the right tool for each task, chains them logically, recovers from a bad step. Scores are high. You ship it.

Three days later, a customer says the agent returned a successful response and did nothing. No errors in the logs. Your monitoring flags something odd though: output tokens dropped to zero while input tokens looked totally normal. A silent failure your evals never had a chance to catch.

That's not a hole in your eval strategy. It's a different category of problem entirely.

Evals test reasoning in a controlled environment. Does it pick the right tool? Does it combine tools correctly? Does it recover when something looks wrong? Yes, mostly — because it's running with fresh context, clean input, and nothing else competing for its attention.

The reliability stack tests what happens once that same agent is loose in the real world. Did it actually execute what it decided to do? Did the output match what the model claimed? Did it quietly loop when a tool failed? These aren't reasoning questions. They're runtime questions, and evals were never built to answer them.

What evals miss

Evals run under close to ideal conditions — fresh context, known inputs, tools that behave the way you mocked them, one clean execution path.

Production runs under none of that. APIs time out at random. Rate limits show up mid-run. Real users send things your training data never saw. Context balloons across hundreds of runs. And sometimes the agent just quietly retries a failing tool call, over and over, with nothing logged as an error.

Here's a concrete version of that: your eval tests an agent that fetches user data from an API and summarizes it. It passes clean. In production, that API is slow one day. The timeout fires, the model reads that as a failed call, and decides to retry. Then retries again. Thirty attempts and 120,000 tokens later, it gives up and returns nothing — but logs "success" the whole way through, because technically nothing errored.

Your evals never saw a slow API. They never saw a third retry. They definitely never logged what the agent does when it finally gives up.

What the reliability stack actually watches

A few signals fill that gap, and none of them come from an eval suite:

Silent failures show up as a mismatch between input and output tokens — normal input, near-zero output means the agent processed the request and produced nothing.

Latency anomalies show up when success stays green but timing spikes. That's almost always a loop or a retry burning time in the background.

Token drift shows up when the same task starts costing more tokens over time on the same agent — usually prompt bloat or state quietly accumulating.

Cost spikes are the most obvious canary. A 10x jump in cost per run is rarely a coincidence.

And execution paths — which tools actually got called, in what order, did they all succeed — tell you whether the agent's recovery logic works under real failure, not just the happy path you tested.

Why this matters

The gap between shipped and safe isn't a code quality problem. It's an observability problem.

A well-reasoned agent with a clean eval score can still loop forever on a timeout, return silent success while doing nothing, or spike cost 100x from one bad retry cascade. None of those are logic errors — they only show up once the agent meets a messy, real execution environment.

Your evals proved it can reason. Something else has to prove it can run safely.

Getting started

You don't need to rebuild the agent or the eval suite. You need visibility into execution itself: wrap the LLM client so every run logs tokens, latency, cost, and output length. Run it long enough to know your baselines. Alert when something drifts from them. Log which tools actually got called and what happened when one failed.

Evals and the reliability stack aren't competing with each other. One proves the agent can think. The other proves that thinking survives contact with production.

The Silent Loop: Why Latency and Token Count Together Catch What Status Codes Miss

Babar Hayat — Thu, 06 Aug 2026 06:33:17 +0000

You deploy an AI agent that calls a flaky API. The API fails 10% of the time. Your agent is wired to retry on failure, good practice, right?

Then something goes wrong. The API stays broken for an hour. Your agent keeps retrying. Each retry consumes tokens. The loop never errors out, it just cycles. Success status, because the agent finished (it gave up after N retries). But the cost? Climbing in a straight line while no work got done.

This is the infinite retry loop, not a crash, not a timeout, just silent token bleed. Your logs look healthy. Your error rate is zero. Your bill is bleeding.

The reason most monitoring misses this: error rates and request counts don't catch it. The agent isn't erroring, it's succeeding at retrying. What does catch it is a pattern you can measure right now: latency and token count moving together, in a way that doesn't match your normal execution.

The Math of the Pattern

When a healthy agent runs, there's a relationship between how long it takes and how many tokens it uses. Call this the latency-to-token ratio.

A normal run: 2 seconds, 500 tokens. Ratio: 250 tokens/sec.
Another normal run: 3 seconds, 800 tokens. Ratio: ~267 tokens/sec.
Your baseline range: say, 200-300 tokens/sec.

Now the loop starts:

First 30 seconds: 15,000 tokens. Ratio: 500 tokens/sec. Way above baseline.
Next 30 seconds: another 15,000 tokens. Still climbing.

The signal is clear: latency is spiking, and token consumption is accelerating disproportionately. The agent is doing a lot of token work in a short time, characteristic of retry loops where the model is being re-invoked repeatedly against the same or similar inputs.

By contrast:

A normal, slow run (agent thinking hard): latency high, tokens high, but the ratio stays in your normal band. The model had a complex problem; it thought longer and used more tokens. Proportional.
A loop: latency high, tokens extremely high (disproportionate spike), ratio breaks baseline. The model is being called over and over in the same second window, burning tokens on retry cycles.

Building the Heuristic

Here's the threshold pattern builders should use:

Step 1: Establish your baseline

Run 20 normal agent executions. For each, compute:

ratio = output_tokens / (duration_ms / 1000)

Record the 50th and 95th percentile ratios. Call these p50_ratio and p95_ratio.

Example:

p50: 250 tokens/sec
p95: 400 tokens/sec (accounting for "thinking hard" runs)

Step 2: Set alert thresholds

Caution zone: ratio > p95_ratio * 1.5 (e.g., > 600 tokens/sec)
Alert zone: ratio > p95_ratio * 2.5 (e.g., > 1000 tokens/sec)

Step 3: Combine with duration

If latency is also above your normal range (e.g., > 2 standard deviations from mean), weight the alert higher. A 45-second run with 20,000 tokens (444 tokens/sec) might be fine. A 45-second run with 50,000 tokens (1111 tokens/sec) is almost certainly looping.

Step 4: Watch the trend, not the single run

A single run can be an outlier. Watch for the pattern across 3+ consecutive runs in a 5-minute window:

If 2 or more runs breach the alert threshold, fire the alert.
If the ratio stays elevated, the loop is ongoing.

Why This Catches Loops Before Cost Explodes

Consider the numbers:

Normal agent: 500 tokens/run, $0.00075 per run (at typical pricing).
Loop for 1 hour: 150 retries, ~5000 tokens per cycle, = 750,000 tokens, ~$1.13 per hour.
Loop for 4 hours undetected: $4.50. Small. Still bad.
Loop for a full day: $27. Big enough to be noticed in a weekly bill review. Too late.
Loop caught at 30 minutes via latency+token spike: ~$0.56. Caught, stopped, learning drawn.

The latency-token ratio detects the loop in the first 2-3 runs, not after hours or days. You catch it while the cost is still negligible.

How to Instrument This (No Vendor Lock-in)

You don't need a specialized tool to start. Just log three numbers per execution:

{
"agent_name": "my_agent",
"duration_ms": 45000,
"output_tokens": 22500,
"model": "gpt-4o",
"status": "success"
}

Then compute the ratio in your log aggregator (Datadog, New Relic, CloudWatch, or a simple script):

ratio = output_tokens / (duration_ms / 1000)
if ratio > 1000: # or your alert threshold
send_alert(f"High token-per-second ratio: {ratio}")

If you're wrapping an LLM SDK (OpenAI, Anthropic, etc.), you already have access to output_tokens and request duration. Logging that pair takes three lines.

The Deeper Pattern

This heuristic works because infinite loops have a structural signature: they burn tokens fast relative to the time they're supposed to be thinking. A model pondering a hard question runs long and uses tokens proportionally. A retry loop cycles the model through shallow re-attempts, burning tokens in bursts.

By watching both dimensions together, you're detecting the imbalance that characterizes the loop without needing to parse logs, understand retry logic, or set up complex alerting rules.

It's not perfect, a pathological case might fool it. But it catches the 99% case: the silent loop that's invisible when you're watching latency alone, and eats budget fast when you're not watching at all.

Start logging latency + output tokens today. Compute the ratio. Set the thresholds. You'll catch the next loop before your bill does.

If you're already running agents in production, pull 20 recent runs and compute your p50 and p95 right now. That baseline is your first line of defense against cost creep.

You Shipped the Agent. Now Build the Monitoring Layer.

Babar Hayat — Wed, 05 Aug 2026 07:54:18 +0000

You've shipped your first AI agent to production. The dashboard shows it running. But you have no real idea what it costs per request, or whether it's quietly failing in ways your error logs would never catch.

This is the gap most builders find out about the hard way: deployment isn't observability. An agent can return HTTP 200, log success, and produce zero useful output — all without a single error anywhere in the chain.

Why standard monitoring misses AI agents

Typical app observability — latency, error rates, uptime — was built for request/response workflows. It catches crashes, timeouts, explicit errors. AI agents fail differently. They fail silently.

Three ways this shows up in practice:

HTTP 200, logs show success, zero output tokens generated. The agent ran, the API call succeeded, the response came back empty. No error. Just nothing.
A hundred requests, ninety blank outputs. Your logs say 90% success. Your users see broken functionality.
Cost per request 10x what it was last week — the agent's looping, retrying tool calls, burning tokens, and the logs still say "success."

None of these are crashes. Nothing you're currently watching will flag them.

What you actually need to see

Four signals, once the agent's live:

Tokens per execution (input + output). Your canary — zero output tokens on a "successful" run means something's broken.
Cost per request, computed from tokens × model pricing. Aggregates hide loops; per-execution cost reveals them.
Did it actually produce output — separate from whether the HTTP call succeeded.
Latency and retry patterns — a 2-second call suddenly taking 30 usually means it's looping on a tool call.

Three ways to wire it in

SDK instrumentation

The easiest path if you're on OpenAI, Anthropic, or an OpenAI-compatible provider:

from opsveritas import opsveritas
opsveritas.init(secret="<your OpsVeritas SDK key>")  # from your OpsVeritas org, not OpenAI/Anthropic
opsveritas.wrap(client)

Three lines, and every call gets tracked — tokens, cost, output status, latency, model — without your API keys ever leaving your process.

Webhook

For frameworks without SDK support, or when you want full control over what gets sent. The x-agents-key here is a key OpsVeritas issues to your org specifically — it's separate from your OpenAI/Anthropic credentials, which never touch OpsVeritas in this flow:

curl -X POST https://ai-agents-control-tower.onrender.com/webhooks/agent-execution \
  -H "x-agents-key: <your OpsVeritas webhook key>" \
  -H "Content-Type: application/json" \
  -d '{
    "agent_name": "customer-support-bot",
    "status": "success",
    "input_tokens": 245,
    "output_tokens": 0,
    "cost_usd": 0.0042,
    "duration_ms": 1240,
    "executed_at": "2026-08-04T14:22:15Z"
  }'

Just log it locally

If you're not ready for a platform yet — capture tokens, cost, and whether there was output on every run, print it as structured JSON, and grep for output_tokens: 0 in your own logs. Fifteen minutes, no external dependency.

Set the thresholds

Cost per request under 2x your baseline — above that, something's looping.
Output tokens should never be zero on a run marked successful.
Latency spikes usually mean retries.

Most teams skip this because it feels like overhead — right up until an agent burns a day's budget in an hour, or fails silently for your highest-value customer. Start with one agent, capture tokens and cost, watch for zeros. That's most of the observability you actually need, and it's a few minutes of work.