We Built Monitoring Into Our Own AI Agents. Here's What We Learned.

We run marketing workflows on an AI agent. When we tried to monitor it with existing tools, we found ourselves flying blind in ways we didn't expect.

The Problem We Didn't Know We Had

Three months ago, our marketing agent was supposed to draft social posts every morning. One Tuesday, it hadn't. We checked the logs. No errors. The LLM call succeeded. HTTP 200. The response came back. And yet: no draft. Just a blank.

This is the failure mode that most observability tooling doesn't catch. The agent executed. The infrastructure said "success." But the agent produced nothing — zero output tokens, an empty response, a request that returned 200 OK but accomplished zero.

We had to manually check the execution history to find it. By then, someone else had noticed the absence.

That's when we realized: we couldn't see what was actually happening in our own agents in real time. We needed to build visibility from scratch.

Starting from First Principles

We asked: what do we actually need to know about an AI agent execution?

Not just "did it error?" — because errors aren't the only way an agent fails. We needed:

• Did it run at all? (HTTP status, latency)
• Did it consume what we expected? (tokens, cost)
• Did it produce anything? (output length, not just "success" status)
• Did it do the right thing? (the harder one — we'll come back to this)

Most APM tools watch infrastructure: request latency, error rates, dependencies. None of those answer "was the output empty?" They watch the wrapper, not what happened inside.

So we wrote an SDK wrapper for our agents. Simple design:

1. Intercept the LLM client calls (OpenAI, Anthropic, etc.)
2. Capture the telemetry: tokens in/out, cost, latency, and — crucially — the output itself
3. Send it somewhere we could query and alert on it

The SDK runs inside our own environment, not between us and the LLM provider. Your API keys stay in your process. We only ever see the telemetry.

That design choice matters: it means we can capture what the model actually returned, but we can't see your proprietary prompt or system instructions. Read-only observability.

What We Built (And Why)

The first version just captured raw metrics. We logged tokens consumed (input/output), cost calculated from per-model pricing, latency, status (success/failure/timeout), model name (so we could auto-calculate cost from a built-in rate table), and a summary of the output.

We pushed it to a dashboard and set up basic alerts: if cost spiked, flag it. If latency crossed a threshold, flag it.

This caught infrastructure problems. It didn't catch silent failures.

So we added silent-failure detection: if status is "success" but output_tokens equals 0, that's an alert. The agent ran. It returned 200. It produced nothing.

We caught three more of them within a week using this rule alone.

Then came the harder question: what if the output looks fine but is actually wrong?

An agent returns a well-formed response. Tokens flow normally. Cost is where we'd expect. But the output is garbage — a recommendation that doesn't match the input, a calculation that's off, a response that's just not what was asked for.

No metric catches that. No baseline can. You need a human (or another model) to grade it.

The Correctness Problem

We built a "correctness check" feature: you write a rubric in plain English describing what a correct output should look like ("The response must include a specific name and a recommendation. The recommendation must be actionable."). Then we run every execution's output past an AI grader against that rubric.

The grader isn't perfect. That's the whole point of making it reviewable. On every alert, we let you give a thumbs-up or thumbs-down: "did the judge get this right?" Over time, that feedback becomes a record of accuracy.

And if the judge is systematically wrong about something, you can flag a disagreement and add it as an example the judge learns from. The correction is always manual — we don't auto-add examples from feedback — so a bad rubric can't teach itself to fail.

This catches the "200 OK but completely wrong" case. It doesn't solve the problem of an imperfect judge. It just makes the imperfection visible and reviewable, rather than hidden.

What We Learned

The infrastructure lies to you. HTTP 200 is not a guarantee of work done. Latency is not a guarantee of usefulness. Error rates don't tell you about silent failures. You need eyes on the actual output.

Secrets stay secrets. You can have observability without sharing your API keys or your prompt logic. An SDK that runs in your own environment sees everything your code sees, but only sends us aggregates and summaries. That's the design that lets us help you without breaking your security model.

The easiest failures to miss are the ones that don't error. An agent that times out, an API that 500s — those are obvious. An agent that returns 200 with no response? That's what sits in your queue unnoticed until a customer tells you.

Correctness is a human problem with AI help. You can't write a metric that "correctness" is. You can write a rubric and ask another model to grade it, and you can let humans correct the grader. The system becomes useful when the correction loop is tight and transparent.

The Gaps We Hit

When we were building this, we looked at what existed:

• Application performance monitoring tools watch latency, errors, and infrastructure — not AI-specific problems like silent failures or cost anomalies.
• LLM observability tools (the newer ones) watch tokens and cost, which is closer — but most don't distinguish between "agent returned success with no output" and "agent returned success with output." The metrics look the same.
• Logging frameworks let you log whatever you want, but they don't automatically flag patterns. You're reading through logs manually or writing custom rules.

We needed something that:

1. Auto-detected which model you were using (so cost was automatic, no config)
2. Flagged silent failures specifically (not just errors)
3. Didn't require you to share your API keys with us
4. Let you define what "correct" means and alert when it's violated
5. Gave you real-time visibility without slowing down your agent

That's what we built. And we're eating our own dogfood: this very marketing system runs on an AI agent that we monitor with our own product.

What Comes Next

We're learning as we use it. The silent-failure detection is solid — it catches what the infrastructure can't see. The cost anomaly detection helps catch runaway loops. The correctness check is useful but imperfect by design: we don't claim to know what "correct" means for your use case, only to help you define and enforce it.

The thing we're still figuring out: how to make the correctness feedback loop even tighter. Right now, if the judge disagrees with you, you can promote that disagreement into a training example. But how many examples does the judge actually need to improve? When does a rubric become accurate enough to trust? How do you know when you've taught it enough?

Those are open questions. We're answering them live, with real agents, real failures, and real feedback from the builders using the system.

If you're building AI agents and you're wondering whether your monitoring is good enough, ask yourself: could you spot a silent failure in your own agent right now? Not an error — just an execution that returned success but produced nothing?

If the answer is "not without checking the logs manually," you've found the gap we're trying to close.