DEV Community: OpsVeritas

Cost Governance for AI Agents: From Detection to Enforcement

Babar Hayat — Tue, 18 Aug 2026 19:23:56 +0000

Your AI agent is burning money. It hasn't crashed, it's running fine, and the only way you find out is when the bill shows up.

Here's the pattern: an agent gets confused and calls the same tool over and over. Or a model upgrade quietly triples the cost per token. Or one bad execution fires a hundred times in parallel. From the SDK's point of view, every one of those is a success. Status 200, output returned, nothing to flag.

We think about cost governance as three layers, and they build on each other.

Detection

You need per-execution visibility, not a monthly total. Token count, cost in dollars, latency, per run. That's how you find out which agent burned $500 yesterday instead of which month burned $10k.

Diagnosis

An agent that costs $5 one day and $500 the next isn't random. Either the cost per call spiked (something's asking for way more tokens than usual) or the call count spiked (it's looping). Knowing which one happened tells you what to actually fix.

Enforcement

This is the part most teams skip. Once you know an agent's normal cost profile, you can pause it automatically if it breaks that pattern hard enough, or if it crosses a budget you set. That check runs before the next API call goes out, so it stops the spend rather than just reporting it after the fact. It fails open too, so a network hiccup on the check itself never breaks your agent's normal operation. And restarting it once you've found the root cause takes seconds.

Detection and diagnosis should always be on. Enforcement is opt-in, because the policy is yours to set. One team wants a hard stop. Another wants an alert and a human in the loop.

None of this works without seeing every execution, though. Sampled metrics will miss the loop that cost you a thousand dollars in five minutes.

If you're running agents in production, the cost story shouldn't be a surprise invoice weeks later. It should be a dial you can turn today.

We Built Monitoring Into Our Own AI Agents. Here's What We Learned.

Babar Hayat — Mon, 17 Aug 2026 17:30:42 +0000

We run marketing workflows on an AI agent. When we tried to monitor it with existing tools, we found ourselves flying blind in ways we didn't expect.

The Problem We Didn't Know We Had

Three months ago, our marketing agent was supposed to draft social posts every morning. One Tuesday, it hadn't. We checked the logs. No errors. The LLM call succeeded. HTTP 200. The response came back. And yet: no draft. Just a blank.

This is the failure mode that most observability tooling doesn't catch. The agent executed. The infrastructure said "success." But the agent produced nothing — zero output tokens, an empty response, a request that returned 200 OK but accomplished zero.

We had to manually check the execution history to find it. By then, someone else had noticed the absence.

That's when we realized: we couldn't see what was actually happening in our own agents in real time. We needed to build visibility from scratch.

Starting from First Principles

We asked: what do we actually need to know about an AI agent execution?

Not just "did it error?" — because errors aren't the only way an agent fails. We needed:

Did it run at all? (HTTP status, latency)
Did it consume what we expected? (tokens, cost)
Did it produce anything? (output length, not just "success" status)
Did it do the right thing? (the harder one — we'll come back to this)

Most APM tools watch infrastructure: request latency, error rates, dependencies. None of those answer "was the output empty?" They watch the wrapper, not what happened inside.

So we wrote an SDK wrapper for our agents. Simple design:

Intercept the LLM client calls (OpenAI, Anthropic, etc.)
Capture the telemetry: tokens in/out, cost, latency, and — crucially — the output itself
Send it somewhere we could query and alert on it

The SDK runs inside our own environment, not between us and the LLM provider. Your API keys stay in your process. We only ever see the telemetry.

That design choice matters: it means we can capture what the model actually returned, but we can't see your proprietary prompt or system instructions. Read-only observability.

What We Built (And Why)

The first version just captured raw metrics. We logged tokens consumed (input/output), cost calculated from per-model pricing, latency, status (success/failure/timeout), model name (so we could auto-calculate cost from a built-in rate table), and a summary of the output.

We pushed it to a dashboard and set up basic alerts: if cost spiked, flag it. If latency crossed a threshold, flag it.

This caught infrastructure problems. It didn't catch silent failures.

So we added silent-failure detection: if status is "success" but output_tokens equals 0, that's an alert. The agent ran. It returned 200. It produced nothing.

We caught three more of them within a week using this rule alone.

Then came the harder question: what if the output looks fine but is actually wrong?

An agent returns a well-formed response. Tokens flow normally. Cost is where we'd expect. But the output is garbage — a recommendation that doesn't match the input, a calculation that's off, a response that's just not what was asked for.

No metric catches that. No baseline can. You need a human (or another model) to grade it.

The Correctness Problem

We built a "correctness check" feature: you write a rubric in plain English describing what a correct output should look like ("The response must include a specific name and a recommendation. The recommendation must be actionable."). Then we run every execution's output past an AI grader against that rubric.

The grader isn't perfect. That's the whole point of making it reviewable. On every alert, we let you give a thumbs-up or thumbs-down: "did the judge get this right?" Over time, that feedback becomes a record of accuracy.

And if the judge is systematically wrong about something, you can flag a disagreement and add it as an example the judge learns from. The correction is always manual — we don't auto-add examples from feedback — so a bad rubric can't teach itself to fail.

This catches the "200 OK but completely wrong" case. It doesn't solve the problem of an imperfect judge. It just makes the imperfection visible and reviewable, rather than hidden.

What We Learned

The infrastructure lies to you. HTTP 200 is not a guarantee of work done. Latency is not a guarantee of usefulness. Error rates don't tell you about silent failures. You need eyes on the actual output.

Secrets stay secrets. You can have observability without sharing your API keys or your prompt logic. An SDK that runs in your own environment sees everything your code sees, but only sends us aggregates and summaries. That's the design that lets us help you without breaking your security model.

The easiest failures to miss are the ones that don't error. An agent that times out, an API that 500s — those are obvious. An agent that returns 200 with no response? That's what sits in your queue unnoticed until a customer tells you.

Correctness is a human problem with AI help. You can't write a metric that "correctness" is. You can write a rubric and ask another model to grade it, and you can let humans correct the grader. The system becomes useful when the correction loop is tight and transparent.

The Gaps We Hit

When we were building this, we looked at what existed:

Application performance monitoring tools watch latency, errors, and infrastructure — not AI-specific problems like silent failures or cost anomalies.
LLM observability tools (the newer ones) watch tokens and cost, which is closer — but most don't distinguish between "agent returned success with no output" and "agent returned success with output." The metrics look the same.
Logging frameworks let you log whatever you want, but they don't automatically flag patterns. You're reading through logs manually or writing custom rules.

We needed something that:

Auto-detected which model you were using (so cost was automatic, no config)
Flagged silent failures specifically (not just errors)
Didn't require you to share your API keys with us
Let you define what "correct" means and alert when it's violated
Gave you real-time visibility without slowing down your agent

That's what we built. And we're eating our own dogfood: this very marketing system runs on an AI agent that we monitor with our own product.

What Comes Next

We're learning as we use it. The silent-failure detection is solid — it catches what the infrastructure can't see. The cost anomaly detection helps catch runaway loops. The correctness check is useful but imperfect by design: we don't claim to know what "correct" means for your use case, only to help you define and enforce it.

The thing we're still figuring out: how to make the correctness feedback loop even tighter. Right now, if the judge disagrees with you, you can promote that disagreement into a training example. But how many examples does the judge actually need to improve? When does a rubric become accurate enough to trust? How do you know when you've taught it enough?

Those are open questions. We're answering them live, with real agents, real failures, and real feedback from the builders using the system.

If you're building AI agents and you're wondering whether your monitoring is good enough, ask yourself: could you spot a silent failure in your own agent right now? Not an error — just an execution that returned success but produced nothing?

If the answer is "not without checking the logs manually," you've found the gap we're trying to close.

The Hidden Math: Why "Cheapest Model" Doesn't Mean Cheapest Execution

Babar Hayat — Wed, 12 Aug 2026 10:20:37 +0000

The Hidden Math: Why "Cheapest Model" Doesn't Mean Cheapest Execution

You're building an AI agent. Smart cost strategy: route to GPT-4o when you need reasoning, Haiku for simple classification, Groq when it's available and fast. In your head, the math is simple: pick the cheapest model per task.

In practice, you ship code and never really know if it worked.

Here's the gap: cheapest model ≠ cheapest execution.

A single agent run might span multiple model calls. One task uses Anthropic's Sonnet ($3 / 1M input, $15 / 1M output). Another uses OpenAI's GPT-4o mini ($0.15 / 1M input, $0.60 / 1M output). A third falls back to Groq Llama 3.1 when latency spikes. By the time the execution finishes, you have no idea which model was actually called, how many tokens each burned, or whether that "cheap" fallback actually saved money — or whether a long output summary from one call inflated the token count beyond what you budgeted.

The assumption breaks immediately.

Why the math matters (and why it's invisible)

Token costs compound differently by model.

Take a real scenario: your agent needs to summarize a customer support transcript (2,000 input tokens). You budgeted for Haiku: 2,000 tokens × $0.80 / 1M = $0.0016 input cost. Cheap.

But Haiku hits a rate limit. Code routes to Sonnet as fallback. Same 2,000 input tokens, now $0.006 — nearly 4× more. Multiply that across 100 daily runs, and you've overrun your mental budget without knowing.

Output lengths are invisible.

Worse: you don't even know the output token count. An agent that "completes" a task could've produced 500 output tokens (brief response) or 5,000 (verbose reasoning). With Sonnet outputting at $15 / 1M, that's the difference between $0.0075 and $0.075 per run. Neither is huge. But if your agent runs 1,000 times a month and half your executions unexpectedly verbose?

You're now $37 over budget without seeing why.

Multi-model routing compounds the blind spot.

When you route across three providers with six fallback models, you lose the ability to reason about cost at a glance. Did GPT-4o actually run, or did it fail and drop to Sonnet? How many times did Groq get chosen? Which model produced the longest outputs? You shipped it all, but you're flying blind on the arithmetic that actually happened.

The unsexy solution: measure and account

Here's what works (and it's not flashy):

1. Log every model call with input + output token counts.

Before you route, log which model was selected. After the call returns, log the actual token counts from the response metadata. Not a guess — the real numbers.

Why: your mental model of "mostly cheap, sometimes fallback" gets replaced with actual data. You can group by model, see fallback rates, and measure whether your routing actually favors the cheap path.

2. Compute cost inline using per-model rates.

Every model has a published price per 1M tokens (input and output separate). Compute cost at execution time using actual tokens.

For example:

Input: 2,000 tokens
Output: 1,500 tokens
Model: Anthropic Sonnet ($3 / 1M input, $15 / 1M output)
Cost: (2,000 / 1M × $3) + (1,500 / 1M × $15) = $0.006 + $0.0225 = $0.0285 per run

Sixty runs a day? That's $1.71 / day, $51 / month. You now know the actual stake.

3. Bucket by model + routing path.

At the end of each day (or run batch), group executions by which model actually ran and whether it was your first choice or a fallback. Count, sum costs, measure fallback frequency.

If Groq was supposed to be your primary but it failed 20% of the time, you now have evidence to either increase fallback tolerance, switch providers, or accept that Sonnet is your real cost center.

4. Alert on outliers.

A single execution shouldn't surprise you. If one run's cost is 3σ above the 30-day average for that agent, you want to know why: did it produce an unexpectedly long output? Did it hit a fallback you didn't expect? Was there a retry loop?

The math stops being invisible.

What this buys you

You move from "I think this is cheap" to "I know what this costs, per execution, per model, per routing path."

That's the difference between budgeting in hope and budgeting in fact.

The math isn't exciting. But it's urgent, because runaway cost is usually a routing or retry problem hiding in production, and it stays hidden until the bill lands.

The Silent Failure Detection Framework: Catching Agents That "Succeed" and Do Nothing

Babar Hayat — Mon, 10 Aug 2026 14:01:59 +0000

Your LLM agent returned a response. No error, no exception. But did it actually do what you asked?

That's the silent failure problem. The system behaves normally — HTTP 200, status success — but the output is empty or nonsensical. Nothing alerts you. The customer complains first. To catch these, you need to know what makes a silent failure detectable in the first place. Here's the framework.

What you're actually hunting

A silent failure is execution that reports success but produces zero (or useless) output. The agent didn't crash — it just did nothing. The most common pattern:

Agent calls the model.
Model responds with ~0 output tokens, or a blank response.
No error is raised.
The calling code proceeds, unaware.

Two other variants show up often: an agent looping indefinitely without progress (tool calls that never advance state), and output that's structurally valid but semantically empty — "I don't know" when it should have searched.

Standard error monitoring misses all three, because there is no error. You're checking the wrong signal.

The three detection levers

1. Token counting — the baseline signal

Every LLM call produces input tokens (your prompt) and output tokens (the model's response). Measurable, deterministic, available in the response metadata from every provider.

The insight: a healthy agent call has a predictable token range, per agent. A ticket-summarizer might consume ~100-200 input tokens and produce ~50-150 output tokens, run after run. If an execution comes back with 100 input tokens and 0 output tokens, that's a signal worth flagging — not proof of failure on its own, but worth alerting on.

How to implement it: log input/output tokens for every execution, compute the median and standard deviation of output tokens over the past 30 days, flag anything below the 10th percentile or exactly zero.

baseline_median_output = median(past_30_days_output_tokens)
baseline_std = stdev(past_30_days_output_tokens)
is_anomaly = (today_output_tokens < baseline_median - 2*baseline_std) OR (today_output_tokens == 0)

2. Cost anomaly — a derived signal

Cost = input_tokens × input_rate + output_tokens × output_rate. When token consumption drops, cost drops with it. Cost is often easier to track than raw tokens because it's vendor-independent — you can aggregate across models.

If your agent normally costs $0.02-$0.05 per run and today it's $0.0001, something's very wrong. Same approach: baseline median/std over 30 days, alert below the 5th percentile. Cost lags token anomalies slightly since it's derived, but it's the easier number to reason about in business terms.

3. Output examination — the confirmatory signal

If tokens suggest an anomaly, the next question is what the model actually said. Store a short excerpt of the output (not the whole response, that's usually too large), and you can scan for empty strings, repetitive/looping output, or run a secondary AI judge against a correctness rubric.

Capture the first 500 characters, flag anything empty or suspiciously short against your baseline. Rubric-based judging is optional but valuable for high-stakes agents.

Putting it together

Every execution logs input_tokens, output_tokens, cost_usd, output_summary, executed_at.
On a schedule (nightly, or hourly at volume), compute per-agent baselines and flag anything below threshold.
On a flag: alert the engineer with the output summary attached so triage is immediate, and optionally auto-pause the agent if the cost anomaly is severe.

Why this works, and where it doesn't

Strengths: no false positives from error logs, since you're measuring execution quality directly rather than exceptions. Early signal — tokens are real-time, so you catch problems in minutes. Framework-agnostic — LangChain, CrewAI, raw SDK, every model call produces tokens. Low friction — logging tokens is a few lines of code.

Limitations: you need baseline history (a brand-new agent with five executions has no reliable range yet), the technique is context-dependent (agents with legitimately wide output variance need a wider baseline), and detection isn't prevention — flagging a failure doesn't stop the next expensive call, that needs an actual pause or rate limit on top.

The practical next step

Log tokens — add input_tokens, output_tokens, cost_usd to every execution record.
Compute a baseline once you have a week or two of history.
Set an alert on the 10th percentile of output tokens, or output_tokens == 0.
Triage once. When it fires, you'll either see a genuine empty response (actionable) or a legitimate edge case you exclude next round.

You won't catch every silent failure this way. You'll catch the common ones — the ones that hurt most because they're invisible until someone notices.

If you'd rather not build and maintain this plumbing yourself, that's the layer we built into the AI Agents Control Tower — token tracking, baselining, and anomaly alerting, so you define thresholds instead of infrastructure. Either way the principle holds: measure the tokens, establish the baseline, alert on the gap. Silent failures are only silent until you start listening.

Token budget guardrails: enforcement, not just visibility

Babar Hayat — Sat, 08 Aug 2026 11:39:57 +0000

You've been monitoring your agent's token spend for weeks. Then one run goes wild — 50x the usual token count — and by the time you notice, half your budget is gone.

Visibility is the first step. Enforcement is the second, and it's the one that actually stops the bleed.

The problem: seeing the spike after the damage

Most monitoring setups give you a dashboard. Your agent runs, consumes tokens, and at month end you see the bill. Even with a live dashboard, the response chain is slow: monitor, alert, human, decision, pause. That gap costs real money.

What you actually need is a guardrail that stops the agent from running the moment it crosses a ceiling you defined in advance. Not "alert the team" — "reject the call before it happens."

This matters because of how token explosions actually happen:

Retry loops — an agent keeps calling the same tool because it misreads the response. Each retry burns tokens. A single run can loop 10-50 times before anyone notices.
Context bloat — the agent accumulates conversation history or debug logs in its context window. By run 100, input tokens are 3x baseline.
Hallucinated retries — the model thinks a tool call failed when it actually succeeded, so it tries again. Logs look clean. Cost doesn't.

In every case, the agent still succeeds — HTTP 200. The token spend doesn't.

How to set per-agent token budgets

Start with a baseline. Run your agent 20-50 times under normal conditions and record the token count per execution. This is your reference point — not average monthly spend, but typical spend per single run.

Example: a document-retrieval agent
- Run 1: 3,200 input + 450 output = 3,650 total
- Run 2: 3,100 input + 480 output = 3,580 total
- Run 3: 3,400 input + 520 output = 3,920 total
Average per run: ~3,700 tokens

From that baseline, set two thresholds:

Per-run ceiling. Set it at 2-3x typical spend. Anomalies exist — a genuinely complex query might need more — but 3x is usually where something's actually wrong. If baseline is 3,700 tokens, ceiling lands around 10,000-11,000.

Monthly budget ceiling. Divide your monthly LLM budget by expected run count, then apply a safety margin. Budget $100/month, expect 100 runs — that's $1/run. At roughly $0.002 per 1K tokens for cheaper models, that's about 500 tokens/run. Set the monthly ceiling around $80 (a 20% margin) and enforce it across all agents.

The math is simple. The enforcement is what actually matters.

Implementation: the kill-switch pattern

Once you've defined ceilings, the agent needs to check them before each call, not after.

If you're using the AI Agents Control Tower, this is built in. The kill switch is an opt-in feature that pauses your agents the moment your org crosses its monthly budget:

Enable the kill switch in org settings and define your monthly token/cost limit.
Wrap your LLM client with the SDK's enforcement layer (wrap_langchain for LangChain models) — adds a lightweight status check (~3ms, cached) before each inference call.
If the limit's crossed, the SDK raises OpsVeritasKilledError instead of calling the model. The agent stops. No token burn, no surprise bill.

The kill switch fails open — if the status check itself fails (a network blip), the call proceeds normally. You're protected against silent failures, not against your own infrastructure breaking.

In code:

from opsveritas import init, wrap_langchain
from langchain.chat_models import ChatOpenAI

init(api_key="your-secret")

# Wrap the model for telemetry (which agent ran, how many tokens)
model = ChatOpenAI(model="gpt-4o-mini")

# Add enforcement: blocks the call if budget is exceeded
model = wrap_langchain(model, agent_name="my_agent")

# model.invoke() now raises OpsVeritasKilledError
# if the org's monthly limit is breached

For non-LangChain setups, use the Universal Webhook to POST your agent's token telemetry to the control tower after each run. Include cost_usd in the payload and the system tracks spend in real time across all agents. The kill switch still fires at the org level, but you own the decision of whether to actually call the model.

Per-agent ceilings: a second layer

The org-wide kill switch is insurance. For finer control, add per-agent ceilings inside your agent's own execution loop:

def run_agent(query):
    baseline_tokens = 3700  # your per-run baseline
    ceiling_tokens = 11000  # 3x baseline

    response = model.invoke(query)
    total_tokens = response.usage.input_tokens + response.usage.output_tokens

    if total_tokens > ceiling_tokens:
        log.error(f"Agent exceeded token ceiling: {total_tokens} > {ceiling_tokens}")
        # retry with a simpler query, alert the user, or flag for review
        return {"status": "budget_exceeded", "data": None}

    return response

This layer catches anomalies per execution, not per month. If one run gets expensive, you know immediately, and you can retry, degrade gracefully, or fail loud.

Measuring what matters

Once guardrails are in place, track three metrics:

Tokens per run (p50, p95) — tells you if the baseline is drifting. A climbing trend means the agent is degrading.
Cost per run — token count alone doesn't account for model differences; a gpt-4o run costs differently than gpt-4o-mini.
% of runs hitting the ceiling — if more than 1-2% of runs exceed the threshold, either the ceiling's wrong or the agent has a real problem.

Most monitoring tools show total spend. What you need is per-execution visibility — the individual run that went wild, not just the aggregate bill.

The real lesson

Visibility without enforcement is a dashboard you check after something breaks. Enforcement without visibility is a kill switch that fires mysteriously. You need both.

Set baselines from real data. Define ceilings at 2-3x normal. Enforce before the call, not after. Track per-run metrics, not just aggregate spend. When an anomaly hits, you'll catch it in milliseconds, not at month end.

Cost governance is reliability. Silent token loops are a failure mode just as real as crashes — they just hide longer.

The eval illusion: why passing tests doesn't mean safe in production

Babar Hayat — Fri, 07 Aug 2026 06:04:18 +0000

You built an AI agent. The reasoning holds up — your eval suite checks that it picks the right tool for each task, chains them logically, recovers from a bad step. Scores are high. You ship it.

Three days later, a customer says the agent returned a successful response and did nothing. No errors in the logs. Your monitoring flags something odd though: output tokens dropped to zero while input tokens looked totally normal. A silent failure your evals never had a chance to catch.

That's not a hole in your eval strategy. It's a different category of problem entirely.

Evals test reasoning in a controlled environment. Does it pick the right tool? Does it combine tools correctly? Does it recover when something looks wrong? Yes, mostly — because it's running with fresh context, clean input, and nothing else competing for its attention.

The reliability stack tests what happens once that same agent is loose in the real world. Did it actually execute what it decided to do? Did the output match what the model claimed? Did it quietly loop when a tool failed? These aren't reasoning questions. They're runtime questions, and evals were never built to answer them.

What evals miss

Evals run under close to ideal conditions — fresh context, known inputs, tools that behave the way you mocked them, one clean execution path.

Production runs under none of that. APIs time out at random. Rate limits show up mid-run. Real users send things your training data never saw. Context balloons across hundreds of runs. And sometimes the agent just quietly retries a failing tool call, over and over, with nothing logged as an error.

Here's a concrete version of that: your eval tests an agent that fetches user data from an API and summarizes it. It passes clean. In production, that API is slow one day. The timeout fires, the model reads that as a failed call, and decides to retry. Then retries again. Thirty attempts and 120,000 tokens later, it gives up and returns nothing — but logs "success" the whole way through, because technically nothing errored.

Your evals never saw a slow API. They never saw a third retry. They definitely never logged what the agent does when it finally gives up.

What the reliability stack actually watches

A few signals fill that gap, and none of them come from an eval suite:

Silent failures show up as a mismatch between input and output tokens — normal input, near-zero output means the agent processed the request and produced nothing.

Latency anomalies show up when success stays green but timing spikes. That's almost always a loop or a retry burning time in the background.

Token drift shows up when the same task starts costing more tokens over time on the same agent — usually prompt bloat or state quietly accumulating.

Cost spikes are the most obvious canary. A 10x jump in cost per run is rarely a coincidence.

And execution paths — which tools actually got called, in what order, did they all succeed — tell you whether the agent's recovery logic works under real failure, not just the happy path you tested.

Why this matters

The gap between shipped and safe isn't a code quality problem. It's an observability problem.

A well-reasoned agent with a clean eval score can still loop forever on a timeout, return silent success while doing nothing, or spike cost 100x from one bad retry cascade. None of those are logic errors — they only show up once the agent meets a messy, real execution environment.

Your evals proved it can reason. Something else has to prove it can run safely.

Getting started

You don't need to rebuild the agent or the eval suite. You need visibility into execution itself: wrap the LLM client so every run logs tokens, latency, cost, and output length. Run it long enough to know your baselines. Alert when something drifts from them. Log which tools actually got called and what happened when one failed.

Evals and the reliability stack aren't competing with each other. One proves the agent can think. The other proves that thinking survives contact with production.

The Silent Loop: Why Latency and Token Count Together Catch What Status Codes Miss

Babar Hayat — Thu, 06 Aug 2026 06:33:17 +0000

You deploy an AI agent that calls a flaky API. The API fails 10% of the time. Your agent is wired to retry on failure, good practice, right?

Then something goes wrong. The API stays broken for an hour. Your agent keeps retrying. Each retry consumes tokens. The loop never errors out, it just cycles. Success status, because the agent finished (it gave up after N retries). But the cost? Climbing in a straight line while no work got done.

This is the infinite retry loop, not a crash, not a timeout, just silent token bleed. Your logs look healthy. Your error rate is zero. Your bill is bleeding.

The reason most monitoring misses this: error rates and request counts don't catch it. The agent isn't erroring, it's succeeding at retrying. What does catch it is a pattern you can measure right now: latency and token count moving together, in a way that doesn't match your normal execution.

The Math of the Pattern

When a healthy agent runs, there's a relationship between how long it takes and how many tokens it uses. Call this the latency-to-token ratio.

A normal run: 2 seconds, 500 tokens. Ratio: 250 tokens/sec.
Another normal run: 3 seconds, 800 tokens. Ratio: ~267 tokens/sec.
Your baseline range: say, 200-300 tokens/sec.

Now the loop starts:

First 30 seconds: 15,000 tokens. Ratio: 500 tokens/sec. Way above baseline.
Next 30 seconds: another 15,000 tokens. Still climbing.

The signal is clear: latency is spiking, and token consumption is accelerating disproportionately. The agent is doing a lot of token work in a short time, characteristic of retry loops where the model is being re-invoked repeatedly against the same or similar inputs.

By contrast:

A normal, slow run (agent thinking hard): latency high, tokens high, but the ratio stays in your normal band. The model had a complex problem; it thought longer and used more tokens. Proportional.
A loop: latency high, tokens extremely high (disproportionate spike), ratio breaks baseline. The model is being called over and over in the same second window, burning tokens on retry cycles.

Building the Heuristic

Here's the threshold pattern builders should use:

Step 1: Establish your baseline

Run 20 normal agent executions. For each, compute:

ratio = output_tokens / (duration_ms / 1000)

Record the 50th and 95th percentile ratios. Call these p50_ratio and p95_ratio.

Example:

p50: 250 tokens/sec
p95: 400 tokens/sec (accounting for "thinking hard" runs)

Step 2: Set alert thresholds

Caution zone: ratio > p95_ratio * 1.5 (e.g., > 600 tokens/sec)
Alert zone: ratio > p95_ratio * 2.5 (e.g., > 1000 tokens/sec)

Step 3: Combine with duration

If latency is also above your normal range (e.g., > 2 standard deviations from mean), weight the alert higher. A 45-second run with 20,000 tokens (444 tokens/sec) might be fine. A 45-second run with 50,000 tokens (1111 tokens/sec) is almost certainly looping.

Step 4: Watch the trend, not the single run

A single run can be an outlier. Watch for the pattern across 3+ consecutive runs in a 5-minute window:

If 2 or more runs breach the alert threshold, fire the alert.
If the ratio stays elevated, the loop is ongoing.

Why This Catches Loops Before Cost Explodes

Consider the numbers:

Normal agent: 500 tokens/run, $0.00075 per run (at typical pricing).
Loop for 1 hour: 150 retries, ~5000 tokens per cycle, = 750,000 tokens, ~$1.13 per hour.
Loop for 4 hours undetected: $4.50. Small. Still bad.
Loop for a full day: $27. Big enough to be noticed in a weekly bill review. Too late.
Loop caught at 30 minutes via latency+token spike: ~$0.56. Caught, stopped, learning drawn.

The latency-token ratio detects the loop in the first 2-3 runs, not after hours or days. You catch it while the cost is still negligible.

How to Instrument This (No Vendor Lock-in)

You don't need a specialized tool to start. Just log three numbers per execution:

{
"agent_name": "my_agent",
"duration_ms": 45000,
"output_tokens": 22500,
"model": "gpt-4o",
"status": "success"
}

Then compute the ratio in your log aggregator (Datadog, New Relic, CloudWatch, or a simple script):

ratio = output_tokens / (duration_ms / 1000)
if ratio > 1000: # or your alert threshold
send_alert(f"High token-per-second ratio: {ratio}")

If you're wrapping an LLM SDK (OpenAI, Anthropic, etc.), you already have access to output_tokens and request duration. Logging that pair takes three lines.

The Deeper Pattern

This heuristic works because infinite loops have a structural signature: they burn tokens fast relative to the time they're supposed to be thinking. A model pondering a hard question runs long and uses tokens proportionally. A retry loop cycles the model through shallow re-attempts, burning tokens in bursts.

By watching both dimensions together, you're detecting the imbalance that characterizes the loop without needing to parse logs, understand retry logic, or set up complex alerting rules.

It's not perfect, a pathological case might fool it. But it catches the 99% case: the silent loop that's invisible when you're watching latency alone, and eats budget fast when you're not watching at all.

Start logging latency + output tokens today. Compute the ratio. Set the thresholds. You'll catch the next loop before your bill does.

If you're already running agents in production, pull 20 recent runs and compute your p50 and p95 right now. That baseline is your first line of defense against cost creep.

You Shipped the Agent. Now Build the Monitoring Layer.

Babar Hayat — Wed, 05 Aug 2026 07:54:18 +0000

You've shipped your first AI agent to production. The dashboard shows it running. But you have no real idea what it costs per request, or whether it's quietly failing in ways your error logs would never catch.

This is the gap most builders find out about the hard way: deployment isn't observability. An agent can return HTTP 200, log success, and produce zero useful output — all without a single error anywhere in the chain.

Why standard monitoring misses AI agents

Typical app observability — latency, error rates, uptime — was built for request/response workflows. It catches crashes, timeouts, explicit errors. AI agents fail differently. They fail silently.

Three ways this shows up in practice:

HTTP 200, logs show success, zero output tokens generated. The agent ran, the API call succeeded, the response came back empty. No error. Just nothing.
A hundred requests, ninety blank outputs. Your logs say 90% success. Your users see broken functionality.
Cost per request 10x what it was last week — the agent's looping, retrying tool calls, burning tokens, and the logs still say "success."

None of these are crashes. Nothing you're currently watching will flag them.

What you actually need to see

Four signals, once the agent's live:

Tokens per execution (input + output). Your canary — zero output tokens on a "successful" run means something's broken.
Cost per request, computed from tokens × model pricing. Aggregates hide loops; per-execution cost reveals them.
Did it actually produce output — separate from whether the HTTP call succeeded.
Latency and retry patterns — a 2-second call suddenly taking 30 usually means it's looping on a tool call.

Three ways to wire it in

SDK instrumentation

The easiest path if you're on OpenAI, Anthropic, or an OpenAI-compatible provider:

from opsveritas import opsveritas
opsveritas.init(secret="<your OpsVeritas SDK key>")  # from your OpsVeritas org, not OpenAI/Anthropic
opsveritas.wrap(client)

Three lines, and every call gets tracked — tokens, cost, output status, latency, model — without your API keys ever leaving your process.

Webhook

For frameworks without SDK support, or when you want full control over what gets sent. The x-agents-key here is a key OpsVeritas issues to your org specifically — it's separate from your OpenAI/Anthropic credentials, which never touch OpsVeritas in this flow:

curl -X POST https://ai-agents-control-tower.onrender.com/webhooks/agent-execution \
  -H "x-agents-key: <your OpsVeritas webhook key>" \
  -H "Content-Type: application/json" \
  -d '{
    "agent_name": "customer-support-bot",
    "status": "success",
    "input_tokens": 245,
    "output_tokens": 0,
    "cost_usd": 0.0042,
    "duration_ms": 1240,
    "executed_at": "2026-08-04T14:22:15Z"
  }'

Just log it locally

If you're not ready for a platform yet — capture tokens, cost, and whether there was output on every run, print it as structured JSON, and grep for output_tokens: 0 in your own logs. Fifteen minutes, no external dependency.

Set the thresholds

Cost per request under 2x your baseline — above that, something's looping.
Output tokens should never be zero on a run marked successful.
Latency spikes usually mean retries.

Most teams skip this because it feels like overhead — right up until an agent burns a day's budget in an hour, or fails silently for your highest-value customer. Start with one agent, capture tokens and cost, watch for zeros. That's most of the observability you actually need, and it's a few minutes of work.

Your Agent Returned 200 and Did Nothing: Detecting Silent Failures with Token Counts

Babar Hayat — Mon, 03 Aug 2026 13:02:00 +0000

Your agent returned HTTP 200, zero errors in the logs, and then did nothing. The failure wasn't loud — it was silent.

This happens more than you'd think. An agent invokes a model, gets back a response, processes it, returns success. But somewhere in that flow the actual output — the thing that was supposed to happen — vanished. No exception. No timeout. Just an empty response where a decision or action should have been.

Standard monitoring catches loud failures: connection timeouts, rate limits, exceptions. It misses the silent ones. Here's why, and how to spot them.

The three signals most monitors ignore

When an LLM agent runs, three things happen:

The HTTP call completes (status 200 or 400+)
Tokens are consumed (input, then output)
Actual work occurs — a response is generated, a decision is made, text comes back

Most observability tools watch #1. Some logging catches parts of #2. Almost nobody watches #3 in a way that actually matters.

The mechanical truth: HTTP 200 and a nonzero token count do not guarantee real output.

The model refuses silently

An agent asks Claude (or GPT, or any model) to do something that brushes up against a boundary — political, violent, ambiguous. The refusal policy triggers. It comes back:

HTTP 200
input_tokens: 450
output_tokens: 127
response: "" // or a short refusal like "[No response]"

The call succeeded. Tokens were spent. Nothing usable came out. If your downstream logic checks if response: do_something(), you've got a silent failure.

The model generates emptiness

Less common, but real: the model returns a completion that's all whitespace or filler with no actual content. Output tokens go up. Meaning doesn't.

HTTP 200
input_tokens: 300
output_tokens: 89
response: " \n\n " // whitespace, no content

Tokens spent, call succeeded, nothing happened.

Tool use fails quietly

An agent calls a tool — database lookup, API call, calculation. The tool errors or returns nothing. The model, correctly, generates a response explaining it couldn't do the thing. Except your downstream code expected a value, not an apology.

HTTP 200
input_tokens: 520
output_tokens: 212
response: "I attempted to fetch the user but the database returned no match."
expected: { user_id: 12345, status: "active" }

The agent succeeded at speaking. It failed at acting.

Why token count is the canary

Output tokens measure how much the model actually generated, independent of quality. A genuinely empty response will have an output token count near zero. A response that merely looks empty — but has some content, even noise — will burn tokens.

That's why token count matters more than HTTP status:

HTTP 200 tells you the infrastructure worked.
Output tokens > 0 tells you the model generated something.
Actual content length tells you if there's signal or just noise.

Rough heuristic: if output tokens are suspiciously low for the task, or nonzero but the response is empty/whitespace, you're looking at a silent failure.

A simple detector

def detect_silent_failure(agent_run):
    http_ok = agent_run.status_code == 200
    tokens_spent = agent_run.output_tokens > 0
    has_content = len(agent_run.response.strip()) > min_threshold  # e.g. 10 chars

    if http_ok and tokens_spent and not has_content:
        return "SILENT_FAILURE"
    if http_ok and tokens_spent < expected_tokens - threshold:
        return "UNDERSHOOTING"
    return "OK"

The key is comparing actual output against token count. If a model spent 150 output tokens and your response is empty, something broke between generation and delivery.

Why it's worth catching

A crash tells you something broke — you debug it. A "success" that produced nothing is worse: it quietly corrupts pipelines, skips records, or leaves a user waiting on a response that's never coming. HTTP status and token count both look fine, so plain logging won't catch it. You need something that checks whether the response actually matches what was supposed to happen.

In practice

If you're running agents and not instrumenting for this, you're flying blind. Three things help:

Log output token count alongside every response. Put it in your metrics, not just your logs.
Flag runs where tokens are spent but response length is suspiciously small.
Trace downstream failures back to the source — did the agent really produce output, or did it just look like it did?

The green light doesn't mean it's working. The token count tells you if anyone's actually home.

6 Failure Modes to Instrument Before You Ship an AI Agent

Babar Hayat — Sat, 01 Aug 2026 11:44:24 +0000

Before you ship an AI agent to production, you need to know what happens when it breaks silently, and most teams don't find out until a customer complains.

The reason is structural: your agent can return HTTP 200, your logs can show success, your tests can pass, and the agent still did nothing. No tokens consumed, no output, no action taken. You asked it for a summary and it gave you an empty string. You asked it to classify an email and it returned null. The call succeeded. The agent failed.

This is different from errors. Errors are loud, your monitoring catches them, your alerting screams, you fix them. Silent failures are quieter, and they're more expensive because they hide.

What you need is a checklist, a set of things to instrument before production that would catch the failures that actually matter. Not every metric, just the ones that predict a broken agent.

The 6 failure modes to watch for

1. Zero output tokens

An LLM call that consumed input tokens but produced zero output tokens is a red flag. The model was invoked, it ran, and it produced nothing. Maybe it hit a length limit, maybe it looped and timed out mid-generation, maybe it rejected the prompt entirely.

The check is simple: output_tokens equals 0, input_tokens is greater than 0, and status is success. If this happens once, it's probably noise. If it happens twice in a row, something is wrong.

Track this per agent, per model, per day. If your agent's zero-output rate climbs above 5% on a Wednesday, you have a problem before your customer does.

2. Output summary is blank or obviously wrong

Your agent returns text, but the text is empty, or it's a hallucinated placeholder like "[No response]" or "ERROR: could not generate output." The agent didn't fail to run, it ran and produced garbage.

Capture a short summary of the output, the first 50 to 100 characters, or a key field if your agent returns structured data. Then set an alert: if more than 2 consecutive runs produce blank or obviously-empty summaries, page someone.

This is a heuristic, not a law. But it's the difference between finding out at 3am because your customer's support queue is stuck versus finding out during your afternoon standup.

3. Latency spikes (model is timing out or looping)

An agent call that takes 60 seconds when it normally takes 2 seconds is a symptom. Usually it means one of two things: the model is in a retry loop, or it's stuck generating tokens and hit your timeout.

Track p50 and p95 latency per agent. If p95 suddenly jumps 10x, it's not noise. Something changed.

4. Token consumption is wildly higher than expected

You ship an agent that normally uses 500 tokens per run. On Tuesday, the same agent is consuming 15,000 tokens per run, same input, same queries, same model. But the token use is exploding.

This usually means the agent got stuck in a loop, it's calling the model multiple times in a single run, or it's feeding its own output back into itself, or the prompt got injected with adversarial content that forces token-heavy generation.

Track input and output tokens per run. Set a ceiling: if a single run exceeds 10x the 90th percentile of that agent's normal token use, that's a signal to investigate.

5. Success rate drops below your baseline

You know what your agent's success rate is most of the time. Maybe it's 97%, maybe it's 87%. Whatever it is, that's your baseline. If it drops to 70%, you have a problem.

Most monitoring tools measure errors, exceptions, 500s, timeouts. But an agent can have a very high "success" rate (all calls return 200) while having a low "useful" rate (half the calls produce empty output). Track both.

Success rate should be defined as calls that returned a non-empty, non-hallucinated result divided by total calls. If it drops 10 points or more day-over-day, alert.

6. Cost per run is spiking

Your agent cost $0.02 per run last week. This week it's $0.50 per run, same input distribution, same model. Something is bloating the token use.

Usually this is the loop problem again, the agent calling the model multiple times per run, using a more expensive model, or the prompt got bloated. But you won't know until you have the number.

Compute cost per run from input tokens plus output tokens times model pricing. If the per-run cost jumps 5x, that's your smoke signal.

How to instrument for these checks

You don't need a fancy platform to catch these. You need:

Capture these fields per run: status (success, error, timeout), input_tokens and output_tokens, latency_ms, a short summary of the output (first 100 characters, or null if empty), cost_usd (computed from tokens times model pricing), and timestamp.

Store them anywhere, a database, a log aggregate, a monitoring tool. Even a CSV file updated every 5 minutes will work to start.

Set four alerts: zero-output rate above 5% in the last hour, two consecutive runs with blank summaries, p95 latency above 10x baseline, and per-run cost above 5x the 90th percentile.

Check them daily, or set up a dashboard to watch them during launch week.

That's it. You don't need ML, you don't need anomaly detection, you don't need sophisticated alerting. You need a checklist and the discipline to look at it before your customer's error budget is exhausted.

Why this matters before production

Silent failures are cheap to fix early and expensive to fix late. Catch a zero-output loop before production and you tweak the prompt and redeploy. Catch it at 2am because your customer can't process their data and you're in firefighting mode.

The difference is observability. Not just knowing that the agent ran, but knowing what it did when it ran.

Run through this checklist before you ship. Instrument these six signals. Check them the day you launch. You'll find problems you didn't know you had, and you'll sleep better knowing you'll find the next ones before they become incidents.

The agent that fails silently is not the one you instrumented, it's the one you shipped without looking.

Delegation Masking: Why Your LangChain Callbacks Lie About Sub-Agent Failures

Babar Hayat — Tue, 28 Jul 2026 08:38:58 +0000

You delegate a task from Agent A to Agent B in LangChain. Agent B fails. Agent A's callback chain fires 'success' anyway.

This is the observability blind spot most builders miss in agentic workflows: delegation masking. A sub-agent fails silently, but the parent agent's callback layer never knows because it only watches the delegation call itself, not what the delegated agent actually did.

Let's walk the mechanism.

The Delegation Pattern in LangChain

When you wire up agent-to-agent delegation in LangChain, you're typically using the tool decorator to wrap a sub-agent invocation:

@tool
def delegate_to_classification_agent(task: str) -> str:
    """Delegate classification to a specialized sub-agent."""
    result = classification_agent.invoke({"input": task})
    return result.get("output", "")

The parent agent treats this as just another tool. It calls it, gets a result, moves on. The parent agent's callback chain (the layer that logs success/failure, fires alerts, measures latency) only sees the function return value, not the internal state of the sub-agent.

Where the Failure Hides

Here's what can happen:

Agent B (sub-agent) fails to produce valid output. Its internal chain breaks, maybe a tool call failed, or output parsing broke, or the LLM went silent. Agent B's callback chain logs the failure.
But the delegation function still returns something. Maybe it returns an empty string, a cached fallback, or a generic error message. It doesn't raise an exception, it just returns.
Agent A's callback layer sees HTTP 200. The delegation tool returned something, so the callback fires on_tool_end with status: "success". The parent agent logs success, moves on.
The actual failure is buried two layers deep. Agent A's monitoring sees "delegation succeeded." Only if someone digs into Agent B's logs does the failure surface.

This is a callback visibility boundary. The parent agent's instrumentation layer is one level too high to catch delegation failures.

The Signal That Catches It

Standard token-counting observability misses this because both agents might report partial token usage (Agent B started, burned some tokens, then failed). A success callback fired, so metrics look nominal.

What actually catches delegation failures:

1. Output validation at the delegation boundary

Track what the delegation function actually returned:

@tool
def delegate_to_agent(task: str) -> str:
    """Delegate with observability."""
    result = sub_agent.invoke({"input": task})
    output = result.get("output", "")

    # Signal: validate the output exists and is non-empty
    if not output or output.strip() == "":
        # This is a silent failure, the sub-agent ran but produced nothing
        log_event("delegation_produced_empty_output", {
            "delegated_agent": "classification_agent",
            "input": task,
            "sub_agent_status": result.get("status"),
            "token_count": result.get("usage", {}).get("output_tokens", 0)
        })

    return output

The signal is: a delegation that returned empty or unchanged input. The parent agent's callback sees "success," but observability knows something went wrong.

2. Cross-agent execution correlation

When Agent A calls Agent B, log a correlation ID that links both agents' execution traces:

import uuid

correlation_id = str(uuid.uuid4())

# In Agent A's tool:
result = sub_agent.invoke(
    {"input": task, "correlation_id": correlation_id},
    metadata={"correlation_id": correlation_id}
)

# In sub-agent's callback handler:
def on_tool_end(self, output: str, **kwargs):
    corr_id = kwargs.get("metadata", {}).get("correlation_id")
    if corr_id:
        log_event("agent_execution", {
            "correlation_id": corr_id,
            "agent": "sub_agent",
            "output_tokens": ...,
            "status": "success" or "failed"
        })

Now you can query: "What sub-agent executions have a correlation_id but show empty output or error status?" That's where delegation failures hide.

3. Aggregate success rate per agent pair

Track success rates at the delegation edge, not just per-agent:

delegation_success_rate = (
    executions where parent_agent=A and delegated_agent=B and output_validation_passed
) / (total executions where parent_agent=A and delegated_agent=B)

If Agent A to Agent B delegation shows 95% success in parent logs but only 70% pass output validation, you've found the callback masking.

Why Frameworks Don't Catch This

LangChain's callback layer is designed to instrument the calling agent's perspective, not the called agent's internal state. That's by design, clean separation of concerns. But it means delegation failures are invisible until they propagate (or don't).

Most frameworks have the same boundary. CrewAI's task delegation, AutoGen's sub-agent calls, they all fire success callbacks when the call itself succeeds, regardless of what the called agent actually did.

The Observability Fix

You need one layer deeper:

Instrument sub-agents independently. Log their execution status, output validity, token usage. Don't rely on the parent agent's callback to know what happened.
Validate delegation outputs. Don't trust that a delegation function returning a value means the sub-agent actually succeeded. Check the output.
Correlate across agents. Link parent and child agent executions so failures propagate upward in your observability dashboard, not just downward in logs.

The callback chain is essential, but it's not enough. Delegation visibility requires you to see both sides of the boundary.

The takeaway: When agents delegate to other agents, callback success is not the same as execution success. Standard monitoring stays silent because the parent agent's callbacks only see the delegation call, not the delegated agent's actual work. Catch it by validating outputs, correlating executions across agents, and measuring success rates at the delegation edge.

Instrumentation Patterns for AI Agents: SDK vs Webhook

Babar Hayat — Mon, 27 Jul 2026 05:30:28 +0000

When you instrument a distributed system — a microservice mesh, a backend job queue, a real-time event pipeline — you don't ask "should we?" You ask "how?" And you know the playbook: wrap your client, push telemetry, choose your transport, decide on sampling.

AI agents need the same discipline. But right now, most builders either skip instrumentation entirely or bolt it on as an afterthought. The gap between "my agent runs" and "I know what my agent actually did" is where silent failures hide, cost spikes live invisible, and production incidents start.

There are two proven patterns for wiring observability into AI agents: SDK-based instrumentation and webhook-based telemetry. Neither is universally better, each trades off deployment simplicity, latency impact, privacy scope, and operational control. Understanding those tradeoffs matters: it determines whether you catch silent failures before your customers do.

Pattern 1: SDK Instrumentation

With the SDK pattern, you install a lightweight library into your agent's runtime and wrap your model client, the OpenAI, Anthropic, or Gemini instance your agent actually calls.

# Install: pip install opsveritas
from opsveritas import init, wrap
import openai

init(secret="YOUR_AGENT_SECRET")
client = openai.Client(api_key="...")
wrapped_client = wrap(client)  # That's it

# Now your agent uses wrapped_client instead of client
response = wrapped_client.chat.completions.create(...)

The SDK intercepts the call before it leaves your process, reads the request metadata and response (tokens, latency, cost, parsed output), and ships that telemetry asynchronously. Your agent's latency is unaffected; the SDK's overhead is a few milliseconds of serialization.

The tradeoffs:

Low latency impact. Telemetry is pushed in the background, so your agent's response time doesn't change.
In-process visibility. The SDK sees the raw request and response before they leave your Python or Node process, capturing token counts, model name, and optionally a summary of the output without re-parsing.
Framework coverage. SDKs can auto-instrument specific client libraries (OpenAI, Anthropic, Gemini) and frameworks (LangChain callbacks, CrewAI integration). Each integration is narrow but deep.
Operational cost. You manage telemetry transport, meaning SDK retries, buffering, batching. If your network is flaky, telemetry may queue or drop.
Privacy scope. The SDK runs in your environment; you control whether to strip output text, run in metadata-only mode, or send full details.
Framework coupling. You depend on SDK updates to support new models or client libraries. An obscure or internal LLM client won't be auto-instrumented.

Pattern 2: Webhook Instrumentation

With the webhook pattern, you don't install a library. Instead, you POST telemetry directly to an observability service from your agent code.

import requests
from datetime import datetime

# After your agent runs
response = agent.run(user_input)

payload = {
    "agent_name": "document-processor",
    "status": "success",
    "executed_at": datetime.utcnow().isoformat(),
    "input_tokens": 500,
    "output_tokens": 150,
    "model": "gpt-4o",
    "cost_usd": 0.0075,
    "duration_ms": 2400,
}

requests.post(
    "https://ai-agents-control-tower.onrender.com/webhooks/agent-execution",
    json=payload,
    headers={"x-agents-key": "YOUR_ORG_SECRET"}
)

You decide what to capture and POST it yourself. There's no magic, just HTTP.

The tradeoffs:

Framework-agnostic. Works with any agent framework, any LLM client, even custom scripts. You're not locked into SDK coverage.
Operational control. You own the payload shape, so you can capture custom fields (user ID, feature flags, request context) that matter to your business.
Network latency. The webhook is an HTTP request. If your observability service is slow or the network is congested, it adds latency to your agent's response time unless you fire-and-forget with an async task.
Manual instrumentation. You have to write the code to collect and POST telemetry. It's not automatically captured the way the SDK auto-patches a client.
Privacy-first. You decide exactly what data gets shipped. No SDK auto-capturing output text or summarizing responses unless you code it.
Operational resilience. If the observability service is down, your webhook requests will fail. You need retry logic and queueing to avoid blocking your agent.

How they differ in practice

Capture scope: the SDK automatically captures tokens, latency, model, and output (configurable). With webhooks, you decide, and minimal setup means only the fields you code.

Latency cost: SDK overhead is negligible, async telemetry serialization. Webhooks add 50 to 500ms per request unless you async-queue them.

Time to first signal: with the SDK it's immediate, since telemetry is already in your code. With webhooks you add instrumentation per agent or per framework, which takes more planning.

Handling new models: SDK updates add support and you upgrade. With webhooks you handle it yourself, usually just adding the cost calculation.

Privacy: the SDK is configurable, with a metadata-only mode that strips all output content. Webhooks send whatever you choose to POST.

When to use each

Use the SDK if you have a small number of well-known model clients (OpenAI, Anthropic, Gemini), want observability with minimal code changes, your agent is latency-sensitive and can't afford webhook round-trips, or you're using a supported framework like LangChain or CrewAI and want callbacks wired automatically.

Use webhooks if you have a heterogeneous stack (internal LLM API, third-party models, multiple clients), need custom telemetry fields (user context, feature flags, request metadata), want to avoid SDK dependencies and keep your deployment simple, or you're comfortable managing retry logic and async queueing.

Use both if you have a hybrid setup: SDKs for critical paths like real-time APIs, webhooks for background jobs and batch processing.

The implementation reality

In practice, the pattern you choose shapes your observability architecture for months. The SDK path is faster to ship but locks you into SDK coverage. The webhook path requires more upfront design but gives you more flexibility.

Most production AI systems end up using both: the SDK for OpenAI/Anthropic agents in hot paths where latency matters, webhooks for heterogeneous or custom setups. The tradeoff isn't binary, it's contextual.

The core insight is that instrumentation isn't optional. Whether you choose SDK or webhooks, the choice forces you to think about what you need to observe, and that discipline is what catches silent failures before production users do.

Pick the pattern that matches your architecture. Wire it in before you ship to production. And don't wait until a cost spike or a failed task to realize you have no visibility into what actually ran.