DEV Community: OpsVeritas

The Real Cost of Not Knowing

Babar Hayat — Mon, 07 Sep 2026 14:44:13 +0000

The Real Cost of Not Knowing

Most automation teams think about monitoring backwards. They see the $99/month price tag and do a quick mental math: "That's expensive. We've been running workflows fine for two years without it."

Then a workflow fails silently for 6 hours. Not erroring, just doing nothing. Candidates don't get scored. Payment records don't sync. Leads don't get tagged. Everything looks green in the logs.

The cost of that 6-hour window is almost never $99.

Why "Silent" Failures Are Expensive

A silent failure is a workflow that returns HTTP 200 and exits cleanly, but processed zero items. Your monitoring dashboard doesn't see an error. Your error tracking doesn't ping. Your team finds out when a customer says "I applied three days ago and haven't heard back."

Here's the math that makes this expensive:

Revenue workflows (payment reconciliation, lead capture, subscription processing):

If your workflow touches 50 transactions/hour at an average value of $20 each: 6 hours x 50 x $20 = $6,000 in unprocessed value.
Detection delay if you're checking manually once a day: lose another 18 hours.
Cost of recovery (manual backfill, customer support, retry fee): $500-$2,000.
Total: $6,500-$8,000 for one undetected 6-hour gap.

Compliance workflows (audit logs, data sync, regulatory reporting):

Silent failure = missing logs or records.
Audit discovery cost: compliance review + documentation recovery: $1,000-$5,000.
Regulatory fine risk if the gap affects reporting deadlines: $5,000-$50,000+.

Candidate/hiring workflows (resume screening, interview scheduling, panel coordination):

Hiring round stalls because candidates aren't being moved through the pipeline.
Each day of stall costs hiring velocity: 1-2 strong candidates who lose interest.
Cost of replacing a failed hire due to slow pipeline: $20,000-$100,000.
Cost of filling a role one week late: $30,000-$50,000 in productivity loss.

The pattern: if your workflow touches anything with business value (revenue, compliance, hiring, customer data), a 6-hour detection window usually costs more than a year of monitoring.

The Break-Even Framework

Here's a framework you can run in a spreadsheet:

Cost of one silent-failure incident:
= (detection_delay_hours / 24) x daily_workflow_value
+ recovery_cost
+ business_impact_cost

Daily workflow value =
(transactions_per_day) x (average_value_per_transaction)
OR (revenue_at_risk_per_day)
OR (compliance_penalty_risk_per_day)

Annual monitoring cost = $99/mo x 12 = $1,188

Break-even point =
Annual monitoring cost / Cost of one incident
(If the cost is high, break-even is often < 1 incident per year)

Example: a Make scenario that syncs 500 customers/day to your CRM

Transactions per day: 500
Value per transaction: customer relationship data
(if sync fails, support team has to manually fix entries = $50/incident)
Detection delay if manual: 24 hours
Recovery overhead: 8 hours x $150/hr labor = $1,200

Cost of one 24-hour undetected sync failure:
= (24 / 24) x (500 x $50) + $1,200
= $25,000 + $1,200
= $26,200

Annual monitoring cost: $1,188

Break-even: $1,188 / $26,200 = 0.045 incidents/year
(Meaning: one incident every 22 years)

If you've run this workflow for 2 years without a silent failure, congratulations, you've been lucky. Monitoring is insurance. The question is whether you can afford a $26,000 loss to save $1,188 in premiums.

What "Monitoring" Actually Means Here

When we say monitoring catches silent failures, the mechanism is simple:

Your workflow completes without error.
Monitoring logs: status: success, items_processed: 0.
Alert fires immediately: "Workflow succeeded but produced zero output."
You know in 10 minutes instead of 24 hours.

For n8n, Make, Zapier, and GitHub Actions, OpsVeritas connects via a read-only API key or a monitoring node. No changes to your workflows, no rip-and-replace. You get per-run visibility: status, items processed, run time, and execution logs. Silent failures, stale workflows, cost spikes, and performance degradation all surface before your customers notice.

The monthly cost is the same regardless of platform: $99 (Starter) covers 100 workflows. If you're running 10+ workflows that touch revenue, data, or compliance, your break-even is usually less than one incident.

The Real Question

Ask yourself: "What's the business value of the workflows I'm monitoring? How long can one of them stay silent before it costs me more than I'm spending on monitoring?"

If that number is less than 30 days, monitoring isn't a cost center. It's insurance. And like all insurance, you don't buy it because you expect a payout. You buy it because the cost of being wrong is higher than the cost of peace of mind.

The best time to add monitoring is before you need it. The second-best time is right after you find out why you needed it the first time.

Building a DIY Health Check for Your Zapier Webhooks (And Where It Falls Apart)

Babar Hayat — Thu, 03 Sep 2026 06:18:03 +0000

If you run anything business-critical through Zapier webhooks, you've probably had the moment: a zap that looked fine in the dashboard, that had been quietly not firing for two days. Here's a simple health check you can build yourself in about twenty minutes, and the point at which it stops being enough.

The basic pattern: a dead man's switch

The core idea is old and reliable. Instead of asking "did my zap run correctly," ask "has my zap checked in recently."

Write a heartbeat. At the end of your critical zap, add a final step that writes a timestamp somewhere durable: a row in a Google Sheet, a key in Airtable, a record in your own database. It doesn't matter what the zap actually did upstream; this step just proves the zap reached its last action.
Poll the heartbeat. A second, independent zap (or a simple cron job, or an n8n schedule trigger, whatever you already have running) checks that timestamp every 15 or 30 minutes.
Alert on staleness, not on error. If the timestamp hasn't moved in longer than your zap's expected interval, say double the normal gap, fire a Slack message or an email. You're not catching an error. You're catching silence.

This catches the most obvious failure mode: a zap that stops triggering entirely, because a connected account's token expired, a filter got misconfigured, or the upstream service changed its webhook payload shape.

Where it breaks down

The dead man's switch only tells you the zap is still executing. It says nothing about whether each individual run did anything useful. That's the gap that actually causes the most damage, because it's invisible even to this fix.

Concretely: your webhook-triggered zap runs at 9am, 10am, 11am, every hour, checking in on schedule. But three of those runs processed zero records because the incoming payload's field names changed slightly and your filter step silently excluded everything. The heartbeat updates right on schedule. The zap is "healthy." It's also doing nothing.

Catching that requires a different kind of check: not "did the zap run," but "did the zap's output match what it normally produces." That means tracking volume per run, not just liveness, and comparing it against a baseline instead of a fixed rule, since a genuinely quiet Sunday and a broken filter look identical from a single data point.

When the DIY version stops scaling

The heartbeat-and-poll pattern works fine for one or two critical zaps. It gets unwieldy fast once you're running a dozen workflows across Zapier, Make, and n8n, each needing its own heartbeat step, its own polling job, its own alerting logic, and its own baseline for what "normal volume" even looks like.

At that point, a tool like OpsVeritas can help. It sits outside your workflows and watches for silent failures at scale: zero-output runs, silent stalls, volume drops against a rolling baseline, so your team isn't rebuilding health checks in every zap by hand.

Why "HTTP 200" Doesn't Mean Your Automation Worked

Babar Hayat — Thu, 03 Sep 2026 06:17:48 +0000

Every automation platform, n8n, Make, Zapier, and the custom pipelines teams build on top of them, tells you when a run "succeeded." The problem is what that word actually means under the hood.

Two different definitions of success

In application code, success usually means a specific outcome happened: a row got inserted, a record got returned, a message got delivered. If it didn't happen, you get an exception, a non-200 status, something loud.

In workflow automation, "success" almost always means something narrower: the run finished executing every node without one of them throwing an unhandled error. That's it. Nothing about what the nodes actually produced.

These two definitions look similar from a distance. They diverge exactly where it matters.

Where the gap opens

Take a typical n8n workflow: a trigger, a filter node, an HTTP request node, a database write. If the filter node's condition is slightly wrong, say it excludes every record instead of the ones you meant to exclude, the workflow doesn't error. It just filters everything out, so the HTTP request and database write nodes never fire, or fire with zero items. The execution log shows green across every node. Nothing crashed. Nothing timed out.

Same shape of problem in Make: a scenario polls an API, transforms the response, and syncs it to a destination. If the source query is malformed in a way the API accepts (wrong filter value, wrong date range), you get a 200 response with an empty result set. The scenario "succeeds" at processing zero records.

Zapier is the same pattern with a friendlier UI. A trigger that's scoped too narrowly fires never, or fires without the fields downstream steps expect. The zap's history shows a clean run. It just ran on nothing.

Why this isn't a platform bug

It's tempting to file this under "these tools should be smarter." They can't be, not without knowing your intent. A workflow engine has no way to know that your lead-sync scenario is supposed to move roughly 40 to 80 records a day. It only knows whether each node's own operation completed. Whether the operation was meaningful is business context the platform was never given.

That's the actual architectural gap: execution-level success and outcome-level success are checked at different layers, and most teams only instrument the first one.

What validating at the boundary actually looks like

The fix isn't more error handling inside the workflow. It's a check that sits outside the workflow and compares what actually happened against what should have happened:

Item count, not just status code. A sync that normally moves 50-plus records and today moved zero is a signal, even though every node returned 200.
Baseline comparison, not a fixed threshold. "Zero is always bad" is too blunt for workflows with legitimately idle days. Compare against a rolling baseline instead.
Alerting on the absence of activity, not just the presence of errors. Most monitoring is built to catch exceptions. It has to be deliberately built to catch silence.

None of this requires touching the workflow's own logic. It's a layer that watches outputs, independent of whether the workflow thinks it succeeded.

That's the pattern behind OpsVeritas: watching for the runs that finish green and produce nothing, the failures automation platforms aren't designed to see. If you're running n8n, Make, or Zapier at any real volume, it's worth a look.

The Silent Wrong Answer

Babar Hayat — Sun, 23 Aug 2026 14:53:05 +0000

Your agent returned a well-formed response. The output looks right. Then your customer replies: "That's not what I asked for."

This is the failure class that hits hardest.

The Silent Wrong Answer

There's a hierarchy of agent failures, and most of the attention goes to the loud ones: crashes, timeouts, exceptions in the logs. Your monitoring catches those instantly. But there's a quieter failure — the one that slips through because everything looks fine — and it can damage your reputation faster than any outage.

Your agent completes successfully. HTTP 200. No errors. The output parses. It's well-formed JSON or clean text. Your logs show a successful execution. By every standard measure, the agent did its job.

Except it didn't.

The output is plausible. It has the right structure. It contains relevant information. But when your customer reads it, they realize it doesn't actually answer what they asked. Maybe it's an off-by-one error in a calculation. Maybe it misunderstood a constraint and applied the wrong business rule. Maybe it hallucinated a confident-sounding answer to a question it didn't actually know how to solve.

Your customer doesn't see "the monitoring missed this." They see: "Your automation is broken" — and they're not wrong. The damage is immediate: lost trust, delayed work, questions about whether they should even be using your automation.

Why This Failure Class Is Invisible

Standard observability catches:

Empty output (silent failures proper) — when an agent returns success but produces zero useful tokens.
Format errors — when the JSON doesn't parse or the response structure is wrong.
Execution failures — timeouts, API errors, exceptions.

None of those catch a response that is structurally correct but semantically wrong. The agent produced output. It's well-formed. It ran to completion. Everything your logs and dashboards show you says "success."

This is the gap most builders discover by accident — when a customer tells you about it.

The Real Cost

If you're building automations or AI agents for clients, for a product, or for yourself at scale, this failure class compounds:

Discovery delay. You don't find out until someone downstream uses the output and realizes it's wrong. Could be hours, could be days.
Attribution chaos. Your customer doesn't report "the agent output was wrong" — they report "your automation didn't work," and now you're debugging blind. Was it the agent? The integration? The underlying data? You have to work backward from the complaint.
Reputation hit. For agencies and builders, each wrong output erodes trust in your entire automation. If the agent gets the details wrong once, why should they trust it next time?
Compounding failures. If the wrong output feeds into a downstream process (another workflow, a database update, a customer communication), the damage multiplies before you even notice.

Catching It Before Your Customer Does

You can't prevent wrong outputs entirely — that's a problem in AI that outlives any monitoring tool. But you can detect them automatically, before they reach a customer.

The mechanism is straightforward: write a rubric — a short sentence or checklist describing what a correct output should look like for your agent. Then grade every execution's output against that rubric, using an AI judge that applies your criteria.

For example:

"The total must equal the sum of line items" (finance automation)
"All three fields must be present: name, email, phone" (data extraction)
"The response must cite at least one source from the provided documents" (research agent)
"The date must be in YYYY-MM-DD format and must not be in the future" (scheduling automation)

Every time your agent runs, the output is graded against your rubric. If it fails, you get an alert: "Wrong Output" — before that output goes downstream, before your customer sees it.

The rubric is your criteria, not an external fact-check. This matters: you're not asking the system to verify whether the answer is universally true; you're checking whether it meets your specific requirements. That's a much tighter problem.

And because the judge is imperfect (it can miss things, it can misapply your rubric), the tool is alert-only: it flags the execution for review, it doesn't silently block the agent. You can test your rubric against your agent's recent real executions before enabling it, so a badly written rubric gets caught by you, not by a false-positive flood.

Why This Matters For Builders

If you're shipping an automation or an AI agent into production — whether it's a client's workflow, a product feature, or your own internal system — the wrong output isn't just a technical problem. It's a trust problem.

Your customer doesn't care that the agent's logs looked good. They care that the output was wrong. Every undetected wrong output is a moment you didn't catch your own mistake before it reached them.

Catching them automatically, before they're visible to anyone downstream, is insurance. It's the observability layer that sits above the code and above the framework — the layer that answers the question most builders forget to ask: not "did the agent run," but "did the agent produce the right thing?"

The difference, when your customer is about to find out, is everything.

Cost Governance for AI Agents: From Detection to Enforcement

Babar Hayat — Tue, 18 Aug 2026 19:23:56 +0000

Your AI agent is burning money. It hasn't crashed, it's running fine, and the only way you find out is when the bill shows up.

Here's the pattern: an agent gets confused and calls the same tool over and over. Or a model upgrade quietly triples the cost per token. Or one bad execution fires a hundred times in parallel. From the SDK's point of view, every one of those is a success. Status 200, output returned, nothing to flag.

We think about cost governance as three layers, and they build on each other.

Detection

You need per-execution visibility, not a monthly total. Token count, cost in dollars, latency, per run. That's how you find out which agent burned $500 yesterday instead of which month burned $10k.

Diagnosis

An agent that costs $5 one day and $500 the next isn't random. Either the cost per call spiked (something's asking for way more tokens than usual) or the call count spiked (it's looping). Knowing which one happened tells you what to actually fix.

Enforcement

This is the part most teams skip. Once you know an agent's normal cost profile, you can pause it automatically if it breaks that pattern hard enough, or if it crosses a budget you set. That check runs before the next API call goes out, so it stops the spend rather than just reporting it after the fact. It fails open too, so a network hiccup on the check itself never breaks your agent's normal operation. And restarting it once you've found the root cause takes seconds.

Detection and diagnosis should always be on. Enforcement is opt-in, because the policy is yours to set. One team wants a hard stop. Another wants an alert and a human in the loop.

None of this works without seeing every execution, though. Sampled metrics will miss the loop that cost you a thousand dollars in five minutes.

If you're running agents in production, the cost story shouldn't be a surprise invoice weeks later. It should be a dial you can turn today.

We Built Monitoring Into Our Own AI Agents. Here's What We Learned.

Babar Hayat — Mon, 17 Aug 2026 17:30:42 +0000

We run marketing workflows on an AI agent. When we tried to monitor it with existing tools, we found ourselves flying blind in ways we didn't expect.

The Problem We Didn't Know We Had

Three months ago, our marketing agent was supposed to draft social posts every morning. One Tuesday, it hadn't. We checked the logs. No errors. The LLM call succeeded. HTTP 200. The response came back. And yet: no draft. Just a blank.

This is the failure mode that most observability tooling doesn't catch. The agent executed. The infrastructure said "success." But the agent produced nothing — zero output tokens, an empty response, a request that returned 200 OK but accomplished zero.

We had to manually check the execution history to find it. By then, someone else had noticed the absence.

That's when we realized: we couldn't see what was actually happening in our own agents in real time. We needed to build visibility from scratch.

Starting from First Principles

We asked: what do we actually need to know about an AI agent execution?

Not just "did it error?" — because errors aren't the only way an agent fails. We needed:

Did it run at all? (HTTP status, latency)
Did it consume what we expected? (tokens, cost)
Did it produce anything? (output length, not just "success" status)
Did it do the right thing? (the harder one — we'll come back to this)

Most APM tools watch infrastructure: request latency, error rates, dependencies. None of those answer "was the output empty?" They watch the wrapper, not what happened inside.

So we wrote an SDK wrapper for our agents. Simple design:

Intercept the LLM client calls (OpenAI, Anthropic, etc.)
Capture the telemetry: tokens in/out, cost, latency, and — crucially — the output itself
Send it somewhere we could query and alert on it

The SDK runs inside our own environment, not between us and the LLM provider. Your API keys stay in your process. We only ever see the telemetry.

That design choice matters: it means we can capture what the model actually returned, but we can't see your proprietary prompt or system instructions. Read-only observability.

What We Built (And Why)

The first version just captured raw metrics. We logged tokens consumed (input/output), cost calculated from per-model pricing, latency, status (success/failure/timeout), model name (so we could auto-calculate cost from a built-in rate table), and a summary of the output.

We pushed it to a dashboard and set up basic alerts: if cost spiked, flag it. If latency crossed a threshold, flag it.

This caught infrastructure problems. It didn't catch silent failures.

So we added silent-failure detection: if status is "success" but output_tokens equals 0, that's an alert. The agent ran. It returned 200. It produced nothing.

We caught three more of them within a week using this rule alone.

Then came the harder question: what if the output looks fine but is actually wrong?

An agent returns a well-formed response. Tokens flow normally. Cost is where we'd expect. But the output is garbage — a recommendation that doesn't match the input, a calculation that's off, a response that's just not what was asked for.

No metric catches that. No baseline can. You need a human (or another model) to grade it.

The Correctness Problem

We built a "correctness check" feature: you write a rubric in plain English describing what a correct output should look like ("The response must include a specific name and a recommendation. The recommendation must be actionable."). Then we run every execution's output past an AI grader against that rubric.

The grader isn't perfect. That's the whole point of making it reviewable. On every alert, we let you give a thumbs-up or thumbs-down: "did the judge get this right?" Over time, that feedback becomes a record of accuracy.

And if the judge is systematically wrong about something, you can flag a disagreement and add it as an example the judge learns from. The correction is always manual — we don't auto-add examples from feedback — so a bad rubric can't teach itself to fail.

This catches the "200 OK but completely wrong" case. It doesn't solve the problem of an imperfect judge. It just makes the imperfection visible and reviewable, rather than hidden.

What We Learned

The infrastructure lies to you. HTTP 200 is not a guarantee of work done. Latency is not a guarantee of usefulness. Error rates don't tell you about silent failures. You need eyes on the actual output.

Secrets stay secrets. You can have observability without sharing your API keys or your prompt logic. An SDK that runs in your own environment sees everything your code sees, but only sends us aggregates and summaries. That's the design that lets us help you without breaking your security model.

The easiest failures to miss are the ones that don't error. An agent that times out, an API that 500s — those are obvious. An agent that returns 200 with no response? That's what sits in your queue unnoticed until a customer tells you.

Correctness is a human problem with AI help. You can't write a metric that "correctness" is. You can write a rubric and ask another model to grade it, and you can let humans correct the grader. The system becomes useful when the correction loop is tight and transparent.

The Gaps We Hit

When we were building this, we looked at what existed:

Application performance monitoring tools watch latency, errors, and infrastructure — not AI-specific problems like silent failures or cost anomalies.
LLM observability tools (the newer ones) watch tokens and cost, which is closer — but most don't distinguish between "agent returned success with no output" and "agent returned success with output." The metrics look the same.
Logging frameworks let you log whatever you want, but they don't automatically flag patterns. You're reading through logs manually or writing custom rules.

We needed something that:

Auto-detected which model you were using (so cost was automatic, no config)
Flagged silent failures specifically (not just errors)
Didn't require you to share your API keys with us
Let you define what "correct" means and alert when it's violated
Gave you real-time visibility without slowing down your agent

That's what we built. And we're eating our own dogfood: this very marketing system runs on an AI agent that we monitor with our own product.

What Comes Next

We're learning as we use it. The silent-failure detection is solid — it catches what the infrastructure can't see. The cost anomaly detection helps catch runaway loops. The correctness check is useful but imperfect by design: we don't claim to know what "correct" means for your use case, only to help you define and enforce it.

The thing we're still figuring out: how to make the correctness feedback loop even tighter. Right now, if the judge disagrees with you, you can promote that disagreement into a training example. But how many examples does the judge actually need to improve? When does a rubric become accurate enough to trust? How do you know when you've taught it enough?

Those are open questions. We're answering them live, with real agents, real failures, and real feedback from the builders using the system.

If you're building AI agents and you're wondering whether your monitoring is good enough, ask yourself: could you spot a silent failure in your own agent right now? Not an error — just an execution that returned success but produced nothing?

If the answer is "not without checking the logs manually," you've found the gap we're trying to close.

The Hidden Math: Why "Cheapest Model" Doesn't Mean Cheapest Execution

Babar Hayat — Wed, 12 Aug 2026 10:20:37 +0000

The Hidden Math: Why "Cheapest Model" Doesn't Mean Cheapest Execution

You're building an AI agent. Smart cost strategy: route to GPT-4o when you need reasoning, Haiku for simple classification, Groq when it's available and fast. In your head, the math is simple: pick the cheapest model per task.

In practice, you ship code and never really know if it worked.

Here's the gap: cheapest model ≠ cheapest execution.

A single agent run might span multiple model calls. One task uses Anthropic's Sonnet ($3 / 1M input, $15 / 1M output). Another uses OpenAI's GPT-4o mini ($0.15 / 1M input, $0.60 / 1M output). A third falls back to Groq Llama 3.1 when latency spikes. By the time the execution finishes, you have no idea which model was actually called, how many tokens each burned, or whether that "cheap" fallback actually saved money — or whether a long output summary from one call inflated the token count beyond what you budgeted.

The assumption breaks immediately.

Why the math matters (and why it's invisible)

Token costs compound differently by model.

Take a real scenario: your agent needs to summarize a customer support transcript (2,000 input tokens). You budgeted for Haiku: 2,000 tokens × $0.80 / 1M = $0.0016 input cost. Cheap.

But Haiku hits a rate limit. Code routes to Sonnet as fallback. Same 2,000 input tokens, now $0.006 — nearly 4× more. Multiply that across 100 daily runs, and you've overrun your mental budget without knowing.

Output lengths are invisible.

Worse: you don't even know the output token count. An agent that "completes" a task could've produced 500 output tokens (brief response) or 5,000 (verbose reasoning). With Sonnet outputting at $15 / 1M, that's the difference between $0.0075 and $0.075 per run. Neither is huge. But if your agent runs 1,000 times a month and half your executions unexpectedly verbose?

You're now $37 over budget without seeing why.

Multi-model routing compounds the blind spot.

When you route across three providers with six fallback models, you lose the ability to reason about cost at a glance. Did GPT-4o actually run, or did it fail and drop to Sonnet? How many times did Groq get chosen? Which model produced the longest outputs? You shipped it all, but you're flying blind on the arithmetic that actually happened.

The unsexy solution: measure and account

Here's what works (and it's not flashy):

1. Log every model call with input + output token counts.

Before you route, log which model was selected. After the call returns, log the actual token counts from the response metadata. Not a guess — the real numbers.

Why: your mental model of "mostly cheap, sometimes fallback" gets replaced with actual data. You can group by model, see fallback rates, and measure whether your routing actually favors the cheap path.

2. Compute cost inline using per-model rates.

Every model has a published price per 1M tokens (input and output separate). Compute cost at execution time using actual tokens.

For example:

Input: 2,000 tokens
Output: 1,500 tokens
Model: Anthropic Sonnet ($3 / 1M input, $15 / 1M output)
Cost: (2,000 / 1M × $3) + (1,500 / 1M × $15) = $0.006 + $0.0225 = $0.0285 per run

Sixty runs a day? That's $1.71 / day, $51 / month. You now know the actual stake.

3. Bucket by model + routing path.

At the end of each day (or run batch), group executions by which model actually ran and whether it was your first choice or a fallback. Count, sum costs, measure fallback frequency.

If Groq was supposed to be your primary but it failed 20% of the time, you now have evidence to either increase fallback tolerance, switch providers, or accept that Sonnet is your real cost center.

4. Alert on outliers.

A single execution shouldn't surprise you. If one run's cost is 3σ above the 30-day average for that agent, you want to know why: did it produce an unexpectedly long output? Did it hit a fallback you didn't expect? Was there a retry loop?

The math stops being invisible.

What this buys you

You move from "I think this is cheap" to "I know what this costs, per execution, per model, per routing path."

That's the difference between budgeting in hope and budgeting in fact.

The math isn't exciting. But it's urgent, because runaway cost is usually a routing or retry problem hiding in production, and it stays hidden until the bill lands.

The Silent Failure Detection Framework: Catching Agents That "Succeed" and Do Nothing

Babar Hayat — Mon, 10 Aug 2026 14:01:59 +0000

Your LLM agent returned a response. No error, no exception. But did it actually do what you asked?

That's the silent failure problem. The system behaves normally — HTTP 200, status success — but the output is empty or nonsensical. Nothing alerts you. The customer complains first. To catch these, you need to know what makes a silent failure detectable in the first place. Here's the framework.

What you're actually hunting

A silent failure is execution that reports success but produces zero (or useless) output. The agent didn't crash — it just did nothing. The most common pattern:

Agent calls the model.
Model responds with ~0 output tokens, or a blank response.
No error is raised.
The calling code proceeds, unaware.

Two other variants show up often: an agent looping indefinitely without progress (tool calls that never advance state), and output that's structurally valid but semantically empty — "I don't know" when it should have searched.

Standard error monitoring misses all three, because there is no error. You're checking the wrong signal.

The three detection levers

1. Token counting — the baseline signal

Every LLM call produces input tokens (your prompt) and output tokens (the model's response). Measurable, deterministic, available in the response metadata from every provider.

The insight: a healthy agent call has a predictable token range, per agent. A ticket-summarizer might consume ~100-200 input tokens and produce ~50-150 output tokens, run after run. If an execution comes back with 100 input tokens and 0 output tokens, that's a signal worth flagging — not proof of failure on its own, but worth alerting on.

How to implement it: log input/output tokens for every execution, compute the median and standard deviation of output tokens over the past 30 days, flag anything below the 10th percentile or exactly zero.

baseline_median_output = median(past_30_days_output_tokens)
baseline_std = stdev(past_30_days_output_tokens)
is_anomaly = (today_output_tokens < baseline_median - 2*baseline_std) OR (today_output_tokens == 0)

2. Cost anomaly — a derived signal

Cost = input_tokens × input_rate + output_tokens × output_rate. When token consumption drops, cost drops with it. Cost is often easier to track than raw tokens because it's vendor-independent — you can aggregate across models.

If your agent normally costs $0.02-$0.05 per run and today it's $0.0001, something's very wrong. Same approach: baseline median/std over 30 days, alert below the 5th percentile. Cost lags token anomalies slightly since it's derived, but it's the easier number to reason about in business terms.

3. Output examination — the confirmatory signal

If tokens suggest an anomaly, the next question is what the model actually said. Store a short excerpt of the output (not the whole response, that's usually too large), and you can scan for empty strings, repetitive/looping output, or run a secondary AI judge against a correctness rubric.

Capture the first 500 characters, flag anything empty or suspiciously short against your baseline. Rubric-based judging is optional but valuable for high-stakes agents.

Putting it together

Every execution logs input_tokens, output_tokens, cost_usd, output_summary, executed_at.
On a schedule (nightly, or hourly at volume), compute per-agent baselines and flag anything below threshold.
On a flag: alert the engineer with the output summary attached so triage is immediate, and optionally auto-pause the agent if the cost anomaly is severe.

Why this works, and where it doesn't

Strengths: no false positives from error logs, since you're measuring execution quality directly rather than exceptions. Early signal — tokens are real-time, so you catch problems in minutes. Framework-agnostic — LangChain, CrewAI, raw SDK, every model call produces tokens. Low friction — logging tokens is a few lines of code.

Limitations: you need baseline history (a brand-new agent with five executions has no reliable range yet), the technique is context-dependent (agents with legitimately wide output variance need a wider baseline), and detection isn't prevention — flagging a failure doesn't stop the next expensive call, that needs an actual pause or rate limit on top.

The practical next step

Log tokens — add input_tokens, output_tokens, cost_usd to every execution record.
Compute a baseline once you have a week or two of history.
Set an alert on the 10th percentile of output tokens, or output_tokens == 0.
Triage once. When it fires, you'll either see a genuine empty response (actionable) or a legitimate edge case you exclude next round.

You won't catch every silent failure this way. You'll catch the common ones — the ones that hurt most because they're invisible until someone notices.

If you'd rather not build and maintain this plumbing yourself, that's the layer we built into the AI Agents Control Tower — token tracking, baselining, and anomaly alerting, so you define thresholds instead of infrastructure. Either way the principle holds: measure the tokens, establish the baseline, alert on the gap. Silent failures are only silent until you start listening.

Token budget guardrails: enforcement, not just visibility

Babar Hayat — Sat, 08 Aug 2026 11:39:57 +0000

You've been monitoring your agent's token spend for weeks. Then one run goes wild — 50x the usual token count — and by the time you notice, half your budget is gone.

Visibility is the first step. Enforcement is the second, and it's the one that actually stops the bleed.

The problem: seeing the spike after the damage

Most monitoring setups give you a dashboard. Your agent runs, consumes tokens, and at month end you see the bill. Even with a live dashboard, the response chain is slow: monitor, alert, human, decision, pause. That gap costs real money.

What you actually need is a guardrail that stops the agent from running the moment it crosses a ceiling you defined in advance. Not "alert the team" — "reject the call before it happens."

This matters because of how token explosions actually happen:

Retry loops — an agent keeps calling the same tool because it misreads the response. Each retry burns tokens. A single run can loop 10-50 times before anyone notices.
Context bloat — the agent accumulates conversation history or debug logs in its context window. By run 100, input tokens are 3x baseline.
Hallucinated retries — the model thinks a tool call failed when it actually succeeded, so it tries again. Logs look clean. Cost doesn't.

In every case, the agent still succeeds — HTTP 200. The token spend doesn't.

How to set per-agent token budgets

Start with a baseline. Run your agent 20-50 times under normal conditions and record the token count per execution. This is your reference point — not average monthly spend, but typical spend per single run.

Example: a document-retrieval agent
- Run 1: 3,200 input + 450 output = 3,650 total
- Run 2: 3,100 input + 480 output = 3,580 total
- Run 3: 3,400 input + 520 output = 3,920 total
Average per run: ~3,700 tokens

From that baseline, set two thresholds:

Per-run ceiling. Set it at 2-3x typical spend. Anomalies exist — a genuinely complex query might need more — but 3x is usually where something's actually wrong. If baseline is 3,700 tokens, ceiling lands around 10,000-11,000.

Monthly budget ceiling. Divide your monthly LLM budget by expected run count, then apply a safety margin. Budget $100/month, expect 100 runs — that's $1/run. At roughly $0.002 per 1K tokens for cheaper models, that's about 500 tokens/run. Set the monthly ceiling around $80 (a 20% margin) and enforce it across all agents.

The math is simple. The enforcement is what actually matters.

Implementation: the kill-switch pattern

Once you've defined ceilings, the agent needs to check them before each call, not after.

If you're using the AI Agents Control Tower, this is built in. The kill switch is an opt-in feature that pauses your agents the moment your org crosses its monthly budget:

Enable the kill switch in org settings and define your monthly token/cost limit.
Wrap your LLM client with the SDK's enforcement layer (wrap_langchain for LangChain models) — adds a lightweight status check (~3ms, cached) before each inference call.
If the limit's crossed, the SDK raises OpsVeritasKilledError instead of calling the model. The agent stops. No token burn, no surprise bill.

The kill switch fails open — if the status check itself fails (a network blip), the call proceeds normally. You're protected against silent failures, not against your own infrastructure breaking.

In code:

from opsveritas import init, wrap_langchain
from langchain.chat_models import ChatOpenAI

init(api_key="your-secret")

# Wrap the model for telemetry (which agent ran, how many tokens)
model = ChatOpenAI(model="gpt-4o-mini")

# Add enforcement: blocks the call if budget is exceeded
model = wrap_langchain(model, agent_name="my_agent")

# model.invoke() now raises OpsVeritasKilledError
# if the org's monthly limit is breached

For non-LangChain setups, use the Universal Webhook to POST your agent's token telemetry to the control tower after each run. Include cost_usd in the payload and the system tracks spend in real time across all agents. The kill switch still fires at the org level, but you own the decision of whether to actually call the model.

Per-agent ceilings: a second layer

The org-wide kill switch is insurance. For finer control, add per-agent ceilings inside your agent's own execution loop:

def run_agent(query):
    baseline_tokens = 3700  # your per-run baseline
    ceiling_tokens = 11000  # 3x baseline

    response = model.invoke(query)
    total_tokens = response.usage.input_tokens + response.usage.output_tokens

    if total_tokens > ceiling_tokens:
        log.error(f"Agent exceeded token ceiling: {total_tokens} > {ceiling_tokens}")
        # retry with a simpler query, alert the user, or flag for review
        return {"status": "budget_exceeded", "data": None}

    return response

This layer catches anomalies per execution, not per month. If one run gets expensive, you know immediately, and you can retry, degrade gracefully, or fail loud.

Measuring what matters

Once guardrails are in place, track three metrics:

Tokens per run (p50, p95) — tells you if the baseline is drifting. A climbing trend means the agent is degrading.
Cost per run — token count alone doesn't account for model differences; a gpt-4o run costs differently than gpt-4o-mini.
% of runs hitting the ceiling — if more than 1-2% of runs exceed the threshold, either the ceiling's wrong or the agent has a real problem.

Most monitoring tools show total spend. What you need is per-execution visibility — the individual run that went wild, not just the aggregate bill.

The real lesson

Visibility without enforcement is a dashboard you check after something breaks. Enforcement without visibility is a kill switch that fires mysteriously. You need both.

Set baselines from real data. Define ceilings at 2-3x normal. Enforce before the call, not after. Track per-run metrics, not just aggregate spend. When an anomaly hits, you'll catch it in milliseconds, not at month end.

Cost governance is reliability. Silent token loops are a failure mode just as real as crashes — they just hide longer.

The eval illusion: why passing tests doesn't mean safe in production

Babar Hayat — Fri, 07 Aug 2026 06:04:18 +0000

You built an AI agent. The reasoning holds up — your eval suite checks that it picks the right tool for each task, chains them logically, recovers from a bad step. Scores are high. You ship it.

Three days later, a customer says the agent returned a successful response and did nothing. No errors in the logs. Your monitoring flags something odd though: output tokens dropped to zero while input tokens looked totally normal. A silent failure your evals never had a chance to catch.

That's not a hole in your eval strategy. It's a different category of problem entirely.

Evals test reasoning in a controlled environment. Does it pick the right tool? Does it combine tools correctly? Does it recover when something looks wrong? Yes, mostly — because it's running with fresh context, clean input, and nothing else competing for its attention.

The reliability stack tests what happens once that same agent is loose in the real world. Did it actually execute what it decided to do? Did the output match what the model claimed? Did it quietly loop when a tool failed? These aren't reasoning questions. They're runtime questions, and evals were never built to answer them.

What evals miss

Evals run under close to ideal conditions — fresh context, known inputs, tools that behave the way you mocked them, one clean execution path.

Production runs under none of that. APIs time out at random. Rate limits show up mid-run. Real users send things your training data never saw. Context balloons across hundreds of runs. And sometimes the agent just quietly retries a failing tool call, over and over, with nothing logged as an error.

Here's a concrete version of that: your eval tests an agent that fetches user data from an API and summarizes it. It passes clean. In production, that API is slow one day. The timeout fires, the model reads that as a failed call, and decides to retry. Then retries again. Thirty attempts and 120,000 tokens later, it gives up and returns nothing — but logs "success" the whole way through, because technically nothing errored.

Your evals never saw a slow API. They never saw a third retry. They definitely never logged what the agent does when it finally gives up.

What the reliability stack actually watches

A few signals fill that gap, and none of them come from an eval suite:

Silent failures show up as a mismatch between input and output tokens — normal input, near-zero output means the agent processed the request and produced nothing.

Latency anomalies show up when success stays green but timing spikes. That's almost always a loop or a retry burning time in the background.

Token drift shows up when the same task starts costing more tokens over time on the same agent — usually prompt bloat or state quietly accumulating.

Cost spikes are the most obvious canary. A 10x jump in cost per run is rarely a coincidence.

And execution paths — which tools actually got called, in what order, did they all succeed — tell you whether the agent's recovery logic works under real failure, not just the happy path you tested.

Why this matters

The gap between shipped and safe isn't a code quality problem. It's an observability problem.

A well-reasoned agent with a clean eval score can still loop forever on a timeout, return silent success while doing nothing, or spike cost 100x from one bad retry cascade. None of those are logic errors — they only show up once the agent meets a messy, real execution environment.

Your evals proved it can reason. Something else has to prove it can run safely.

Getting started

You don't need to rebuild the agent or the eval suite. You need visibility into execution itself: wrap the LLM client so every run logs tokens, latency, cost, and output length. Run it long enough to know your baselines. Alert when something drifts from them. Log which tools actually got called and what happened when one failed.

Evals and the reliability stack aren't competing with each other. One proves the agent can think. The other proves that thinking survives contact with production.

The Silent Loop: Why Latency and Token Count Together Catch What Status Codes Miss

Babar Hayat — Thu, 06 Aug 2026 06:33:17 +0000

You deploy an AI agent that calls a flaky API. The API fails 10% of the time. Your agent is wired to retry on failure, good practice, right?

Then something goes wrong. The API stays broken for an hour. Your agent keeps retrying. Each retry consumes tokens. The loop never errors out, it just cycles. Success status, because the agent finished (it gave up after N retries). But the cost? Climbing in a straight line while no work got done.

This is the infinite retry loop, not a crash, not a timeout, just silent token bleed. Your logs look healthy. Your error rate is zero. Your bill is bleeding.

The reason most monitoring misses this: error rates and request counts don't catch it. The agent isn't erroring, it's succeeding at retrying. What does catch it is a pattern you can measure right now: latency and token count moving together, in a way that doesn't match your normal execution.

The Math of the Pattern

When a healthy agent runs, there's a relationship between how long it takes and how many tokens it uses. Call this the latency-to-token ratio.

A normal run: 2 seconds, 500 tokens. Ratio: 250 tokens/sec.
Another normal run: 3 seconds, 800 tokens. Ratio: ~267 tokens/sec.
Your baseline range: say, 200-300 tokens/sec.

Now the loop starts:

First 30 seconds: 15,000 tokens. Ratio: 500 tokens/sec. Way above baseline.
Next 30 seconds: another 15,000 tokens. Still climbing.

The signal is clear: latency is spiking, and token consumption is accelerating disproportionately. The agent is doing a lot of token work in a short time, characteristic of retry loops where the model is being re-invoked repeatedly against the same or similar inputs.

By contrast:

A normal, slow run (agent thinking hard): latency high, tokens high, but the ratio stays in your normal band. The model had a complex problem; it thought longer and used more tokens. Proportional.
A loop: latency high, tokens extremely high (disproportionate spike), ratio breaks baseline. The model is being called over and over in the same second window, burning tokens on retry cycles.

Building the Heuristic

Here's the threshold pattern builders should use:

Step 1: Establish your baseline

Run 20 normal agent executions. For each, compute:

ratio = output_tokens / (duration_ms / 1000)

Record the 50th and 95th percentile ratios. Call these p50_ratio and p95_ratio.

Example:

p50: 250 tokens/sec
p95: 400 tokens/sec (accounting for "thinking hard" runs)

Step 2: Set alert thresholds

Caution zone: ratio > p95_ratio * 1.5 (e.g., > 600 tokens/sec)
Alert zone: ratio > p95_ratio * 2.5 (e.g., > 1000 tokens/sec)

Step 3: Combine with duration

If latency is also above your normal range (e.g., > 2 standard deviations from mean), weight the alert higher. A 45-second run with 20,000 tokens (444 tokens/sec) might be fine. A 45-second run with 50,000 tokens (1111 tokens/sec) is almost certainly looping.

Step 4: Watch the trend, not the single run

A single run can be an outlier. Watch for the pattern across 3+ consecutive runs in a 5-minute window:

If 2 or more runs breach the alert threshold, fire the alert.
If the ratio stays elevated, the loop is ongoing.

Why This Catches Loops Before Cost Explodes

Consider the numbers:

Normal agent: 500 tokens/run, $0.00075 per run (at typical pricing).
Loop for 1 hour: 150 retries, ~5000 tokens per cycle, = 750,000 tokens, ~$1.13 per hour.
Loop for 4 hours undetected: $4.50. Small. Still bad.
Loop for a full day: $27. Big enough to be noticed in a weekly bill review. Too late.
Loop caught at 30 minutes via latency+token spike: ~$0.56. Caught, stopped, learning drawn.

The latency-token ratio detects the loop in the first 2-3 runs, not after hours or days. You catch it while the cost is still negligible.

How to Instrument This (No Vendor Lock-in)

You don't need a specialized tool to start. Just log three numbers per execution:

{
"agent_name": "my_agent",
"duration_ms": 45000,
"output_tokens": 22500,
"model": "gpt-4o",
"status": "success"
}

Then compute the ratio in your log aggregator (Datadog, New Relic, CloudWatch, or a simple script):

ratio = output_tokens / (duration_ms / 1000)
if ratio > 1000: # or your alert threshold
send_alert(f"High token-per-second ratio: {ratio}")

If you're wrapping an LLM SDK (OpenAI, Anthropic, etc.), you already have access to output_tokens and request duration. Logging that pair takes three lines.

The Deeper Pattern

This heuristic works because infinite loops have a structural signature: they burn tokens fast relative to the time they're supposed to be thinking. A model pondering a hard question runs long and uses tokens proportionally. A retry loop cycles the model through shallow re-attempts, burning tokens in bursts.

By watching both dimensions together, you're detecting the imbalance that characterizes the loop without needing to parse logs, understand retry logic, or set up complex alerting rules.

It's not perfect, a pathological case might fool it. But it catches the 99% case: the silent loop that's invisible when you're watching latency alone, and eats budget fast when you're not watching at all.

Start logging latency + output tokens today. Compute the ratio. Set the thresholds. You'll catch the next loop before your bill does.

If you're already running agents in production, pull 20 recent runs and compute your p50 and p95 right now. That baseline is your first line of defense against cost creep.

You Shipped the Agent. Now Build the Monitoring Layer.

Babar Hayat — Wed, 05 Aug 2026 07:54:18 +0000

You've shipped your first AI agent to production. The dashboard shows it running. But you have no real idea what it costs per request, or whether it's quietly failing in ways your error logs would never catch.

This is the gap most builders find out about the hard way: deployment isn't observability. An agent can return HTTP 200, log success, and produce zero useful output — all without a single error anywhere in the chain.

Why standard monitoring misses AI agents

Typical app observability — latency, error rates, uptime — was built for request/response workflows. It catches crashes, timeouts, explicit errors. AI agents fail differently. They fail silently.

Three ways this shows up in practice:

HTTP 200, logs show success, zero output tokens generated. The agent ran, the API call succeeded, the response came back empty. No error. Just nothing.
A hundred requests, ninety blank outputs. Your logs say 90% success. Your users see broken functionality.
Cost per request 10x what it was last week — the agent's looping, retrying tool calls, burning tokens, and the logs still say "success."

None of these are crashes. Nothing you're currently watching will flag them.

What you actually need to see

Four signals, once the agent's live:

Tokens per execution (input + output). Your canary — zero output tokens on a "successful" run means something's broken.
Cost per request, computed from tokens × model pricing. Aggregates hide loops; per-execution cost reveals them.
Did it actually produce output — separate from whether the HTTP call succeeded.
Latency and retry patterns — a 2-second call suddenly taking 30 usually means it's looping on a tool call.

Three ways to wire it in

SDK instrumentation

The easiest path if you're on OpenAI, Anthropic, or an OpenAI-compatible provider:

from opsveritas import opsveritas
opsveritas.init(secret="<your OpsVeritas SDK key>")  # from your OpsVeritas org, not OpenAI/Anthropic
opsveritas.wrap(client)

Three lines, and every call gets tracked — tokens, cost, output status, latency, model — without your API keys ever leaving your process.

Webhook

For frameworks without SDK support, or when you want full control over what gets sent. The x-agents-key here is a key OpsVeritas issues to your org specifically — it's separate from your OpenAI/Anthropic credentials, which never touch OpsVeritas in this flow:

curl -X POST https://ai-agents-control-tower.onrender.com/webhooks/agent-execution \
  -H "x-agents-key: <your OpsVeritas webhook key>" \
  -H "Content-Type: application/json" \
  -d '{
    "agent_name": "customer-support-bot",
    "status": "success",
    "input_tokens": 245,
    "output_tokens": 0,
    "cost_usd": 0.0042,
    "duration_ms": 1240,
    "executed_at": "2026-08-04T14:22:15Z"
  }'

Just log it locally

If you're not ready for a platform yet — capture tokens, cost, and whether there was output on every run, print it as structured JSON, and grep for output_tokens: 0 in your own logs. Fifteen minutes, no external dependency.

Set the thresholds

Cost per request under 2x your baseline — above that, something's looping.
Output tokens should never be zero on a run marked successful.
Latency spikes usually mean retries.

Most teams skip this because it feels like overhead — right up until an agent burns a day's budget in an hour, or fails silently for your highest-value customer. Start with one agent, capture tokens and cost, watch for zeros. That's most of the observability you actually need, and it's a few minutes of work.