Everything Returned 200. The AI Agent Was Still Wrong 🙄

The dashboard said the agent was fine.

Every span completed. Every tool call returned 200. Latency sat inside the budget line. The trace tree looked like a textbook healthy run. And the model still told the user something that was not true.

I know because two people wrote the same thing into my comment thread hours apart, from opposite directions. One asked for denied transitions in the evidence line. The other asked for throttle responses recorded as events instead of being absorbed at the transport layer. They had never met, they did not agree on the vocabulary, and they both described the same shape: the evidence of a failure existed for a moment, and then the system moved on without it.

That is the gap this article is about. A green dashboard and a confident wrong answer are not an anomaly. They are the default state of most agent deployments, because the telemetry that most teams run was built to catch machines that stop, and an agent never stops. It keeps going, politely, and the answer is wrong in a way no span can see.

---

Let's see if everything is really fine...

A traditional service fails when something stops. The database is unreachable. The queue backs up. The certificate expires. The failure is a deviation from an expected mechanical path, and telemetry was built, over forty years, to catch deviations from expected mechanical paths.

An agent fails when everything keeps running.

The model called the tool, the tool returned data, the loop continued, and the final answer was built on a wrong assumption that never surfaced as an error. The failure is semantic. It lives in what the answer means, not in what the call returned.

The practical consequence is brutal if you have ever owned the pager: an error budget built on HTTP status codes goes blind at exactly the moment the agent stops working correctly and starts working confidently. You can put the agent on a page, set an alert for p95 latency, and watch a week of quiet dashboards while the agent answers questions wrong.

The missing signal is evidential, not mechanical. What did the agent attempt? What was blocked? What was verified? What failed the checks?

Here is the part that should worry you. When you start looking for that signal, you discover that most teams never recorded it, because nobody told them the dashboard was supposed to. The traces were always healthy. The failures were always invisible. The two strangers in my comments were describing the fix from opposite ends of the same hole.

Now we can look into the Mental Model Shift

The fastest way to see why this is a new discipline is to put the two monitoring worlds side by side.

Traditional monitoring	Agent observability
Failure is mechanical: something stops	Failure is semantic: everything keeps running
Error codes mark the problem	The problem never produces an error
A red span is the signal	A green span can be the lie
You watch for deviations from the happy path	The happy path itself can be wrong
The unit of truth is the request	The unit of truth is the decision chain
One trace tells you what failed	One trace cannot tell you why the answer is wrong
Metrics you trust because they measure machines	Metrics you must verify because the output is judged, not measured

The unlearning is the point. Every instinct that serves you in a normal service, trust the green, page on the red, watch the p95, ships the wrong behavior for an agent. The dashboard was built to tell you when a machine stopped. An agent does not stop. It finishes, incorrectly, and the finish looks exactly like the success you are trained to trust.

A Run That Looks Healthy ... Don't dare trust this 😂

Let me make the failure concrete before the fix. This is a real shape of run, simplified until the bones show.

Your agent is a support triage bot. A customer writes in with an account problem. The agent decides it needs to look up the account, so it calls the account service with a customer id. The call goes through a gate that checks the agent's permission scope. The gate approves. The service returns the account record. The agent then calls the billing service, and that call gets throttled, because the billing team set a per-minute limit and the agent has been busy. The transport layer retries with backoff. Attempt two returns 200. The billing data comes back.

Then the agent writes the answer. It is wrong, because the account record it read was for a different region, and the billing data it finally got belonged to a stale session. The customer follows up, angrier. The agent apologizes and fixes it.

Now the observability side. Every span completed. Every call returned 200. Latency sat inside budget. The trace tree looked like a healthy run.

Nothing in that trace can answer the two questions that matter. Was the agent told it could not do something? Yes, it was throttled, and the evidence was erased by the retry. Was the answer checked against anything? No, because there was no verdict layer, so the wrong answer sailed through with the same telemetry shape as a right one.

That is the entire problem in one run: the failure was invisible because the events that would have named it were never recorded.

The Retry Trap That Deletes the Evidence

Take the most boring failure in distributed systems: a throttled call.

Your agent asks the provider for something, and the provider says it has hit the rate limit. Under a plain OpenTelemetry setup, the agent's SDK or the transport layer retries with backoff. Attempt two returns 200. The trace records a successful call with a slightly higher duration.

The throttle never appears anywhere.

The 429 existed for a moment, was absorbed by the retry, and the observability layer saw only the retry's success. Your rate-limit budget is a real constraint on how the agent behaves, but the evidence that the constraint was touched was deleted before it reached your telemetry.

If you build where power and bandwidth are not guaranteed, this trap is personal. A 429 and a brownout look the same from the client side, so you retry by instinct. The habit that keeps a service alive in a hostile network is the same habit that deletes your observability. I have watched a week of near-identical error shapes and could not tell the network from the limit, because the evidence had been erased by the very code that was trying to help.

This is the same class of problem as a clean approval log that records only the approvals. A state history with no denials is an untested one. A retry-with-backoff removes the only sample of a remote limit. If your evidence line only records what was allowed, it cannot prove what was prevented. You are watching the agent succeed and calling that observability.

The fix is to make the retry honest about what it touched. The transport layer is the last place that sees the throttle, so it has to emit the evidence before it retries:

// retry.ts: record the limit before the retry erases it
export async function callWithEvidence<T>(
  action: string,
  attempt: () => Promise<T>,
  emit: (e: AgentEvent) => void,
  limitName = "provider",
): Promise<T> {
  let attemptNumber = 0
  for (;;) {
    try {
      const result = await attempt()
      return result
    } catch (err) {
      const status = err instanceof ProviderError ? err.status : undefined
      if (status === 429) {
        attemptNumber += 1
        emit({ kind: "agent_action.throttled", action, attempt: attemptNumber, limit: limitName, ts: Date.now() })
        if (attemptNumber >= 3) throw err
        await backoff(attemptNumber)
        continue
      }
      throw err
    }
  }
}

The throttle event is emitted on the first 429, before any retry runs. If the retry succeeds, the trace shows a successful call and the evidence line shows the limit that was touched. If the retry gives up, the event is already in the store. Either way the constraint becomes a fact of the run.

The emit call lives inside the catch, next to the backoff, because that is the only place the constraint is visible. Put it anywhere else, and you are back to a trace that cannot prove what the transport layer touched. This wrapper is not a special case. It is the pattern every remote call should follow.

The Vocabulary Already Exists

Before the schema, the vocabulary. OpenTelemetry's GenAI semantic conventions give you a vendor-neutral wire format for agent telemetry: the model that was called, input and output token counts, operation names, and, when you opt in, the content of prompts, completions, tool calls, and tool results.

The core attributes are gen_ai.system, gen_ai.operation.name, gen_ai.request.model, gen_ai.response.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.response.finish_reasons, and gen_ai.request.temperature. Signals split into four categories: events, metrics, model spans, and agent spans.

Two details matter for the argument.

Content is opt-in. By default, no prompt text or tool arguments are captured, and the spec deliberately places content in span events rather than span attributes, so a Collector processor can strip or redact it before it leaves your network. Any post or tool that tells you to write the prompt into a span attribute is teaching you a production incident.

Status is still moving. As of mid-2026, no GenAI-specific span, event, metric, or attribute is marked Stable. The conventions moved to a dedicated repository in May 2026, and the last versioned cut of core semconv (v1.42.0) deprecated the old gen_ai.* attributes in favor of it. Adopt the vocabulary now, but expect renames. Dashboards reading stable metrics survive renames. Dashboards reading span-level attribute keys do not.

The conventions give you the wire format. They do not tell you which events to record. That is the contribution this article is about.

What the Vendors Give You and What They Do Not

It is worth naming the tooling reality, because it explains why the schema is not a solved problem that someone else will hand you.

The platforms have done the plumbing well. SigNoz shipped agent-native observability in May 2026, with automatic instrumentation and a published case study of an AI agent monitoring setup. Datadog mapped the GenAI semantic conventions into its LLM Observability schema. Langfuse runs a native OpenTelemetry endpoint. Arize Phoenix is built on OpenTelemetry and supports dozens of frameworks. VS Code Copilot, OpenAI Codex, and Claude Code all emit OpenTelemetry traces, metrics, or events for agent interactions.

What they give you is the same thing: the trace, the token counts, the operation names. What none of them decide for you is whether the run was checked, what was prevented, and which failures matter enough to record. The platform will store the denied event if you emit it. It will not invent the denial for you, and it cannot tell a meaningful denial rate from a silence it never saw.

That is the gap this article fills. The tooling is the tape; the schema is the decision about what the tape records. The decision is yours, and it is the part the vendors cannot sell you.

The Evidence-Event Schema

Here is the schema that makes a trace mean something. Five events, each emitted by code at a specific boundary, none of them on the happy path only:

Event	Fires when	Fields that matter
agent_action.approved	the gate allows the call	action, actor
agent_action.denied	the gate rejects the call	action, reason
agent_action.throttled	a provider limit is touched	action, attempt, limit
verdict.passed	a deterministic check passes	check, evidence
verdict.failed	a deterministic check fails	check, cause

// evidence.ts: the event line that makes a trace meaningful
export type AgentEvent =
  | { kind: "agent_action.approved";  action: string; actor: string; ts: number }
  | { kind: "agent_action.denied";    action: string; reason: string; ts: number }
  | { kind: "agent_action.throttled"; action: string; attempt: number; limit: string; ts: number }
  | { kind: "verdict.passed";         check: string; evidence: string[]; ts: number }
  | { kind: "verdict.failed";         check: string; evidence: string[]; cause: string; ts: number }

Three rules govern the schema.

Denials and throttles are recorded where the retry cannot erase them. The gate layer emits the event at the moment it denies, before any retry logic runs. If your transport retries, the denied event is already in the store. The throttle becomes a fact of the run, not a swallowed error.

The denial event belongs in the same line as the approval. A trace that shows two approvals and one denial tells you the gate is alive. A trace that shows two approvals and zero denials tells you nothing, because you cannot distinguish a gate that never needed to fire from a gate that never fires. This is the denial-first principle: the absence of denials is only meaningful if denials are recorded when they happen.

Verdicts carry their evidence. A verdict.failed event names the check and the cause, so you can answer why the agent's confidence was overruled without opening the trace. The evidence field is the list of citations, tool results, or assertions the check used. Without it, a failed verdict is a bare assertion from another layer of the stack.

Instrumenting the Gate

The schema plugs into an OTel span at the tool boundary. Minimal instrumentation looks like this:

import { trace } from "@opentelemetry/api"

const tracer = trace.getTracer("agent-runtime")

export function recordGate(action: string, gate: GateResult) {
  const span = tracer.startSpan(`agent.${action}`, {
    attributes: {
      "gen_ai.operation.name": action,
      "gen_ai.system": "custom-agent",
    },
  })

  // The denial is evidence, not an error to swallow
  if (gate.status === "denied") {
    span.addEvent("agent_action.denied", {
      action,
      reason: gate.reason,
    })
  }

  if (gate.status === "throttled") {
    span.addEvent("agent_action.throttled", {
      action,
      attempt: gate.attempt,
      limit: gate.limitName,
    })
  }

  span.end()
}

The point of putting the event on the span is that it stays correlated with the run that produced it. You can ask: which agent, which run, which action, what was blocked, and why, and you get the answer from one store instead of four dashboards.

When you do capture content, redact at the Collector, not in the app. The spec records prompt and completion text as span events (gen_ai.content.prompt and gen_ai.content.completion), separate from the core attributes, precisely so a Collector can drop them before they reach storage:

# collector-config.yaml: drop GenAI content events before they leave your network
processors:
  filter:
    error_mode: ignore
    traces:
      spanevent:
        - 'name == "gen_ai.content.prompt" or name == "gen_ai.content.completion"'

The privacy rule is simple: content is PII until you have a reason to keep it, and the reason has to be written down. Token counts are telemetry. The text of a customer's email is not.

Deterministic Verdicts Over Judges

The event schema records whether checks ran. The second half of the discipline is what the checks are.

The current fashion is LLM-as-judge: a second model scores faithfulness, coherence, and tool-call quality. Judges are flexible and need no golden set, but they have three documented blind spots: silence detection (did the agent check for absence before claiming it does not know), perspective alignment (whether the evidence was available at decision time), and counterfactual causality (a plausible answer can rest on a wrong causal chain). A judge that cannot verify when information was absent will bless an answer it cannot verify.

The stronger pattern is deterministic first. Wherever a rule exists, enforce it with a rule. A grounding check is the cleanest example: if the answer cites a document, verify the citation exists in the corpus the agent was permitted to access.

// verdict.ts: grounding check against the source of truth
export function verdictGrounded(answer: string, citations: string[], corpus: Corpus) {
  const missing = citations.filter((c) => !corpus.contains(c))
  return missing.length === 0
    ? { passed: true, evidence: citations }
    : { passed: false, evidence: citations, cause: `citation not in corpus: ${missing[0]}` }
}

This is the same argument my verification article made for approval gates, applied at the monitoring layer: verification should live in code, outside the model's reach. A deterministic check is a unit test for the agent's output. It either passes or it fails, and the failure names its cause.

The strongest published version of this idea is GroundEval (arXiv 2606.22737, June 2026). It evaluates what the agent searched, fetched, cited, and was permitted to access, with rule-based scoring against the recorded trace. A failure reads like a failing unit test instead of a vague "confidence 0.62."

The deterministic check is the part I trust most. A draft either meets the standard or it does not, and when it fails, the reason is on the record. The judge, where a judge is still needed, is the part I watch. A score you did not calibrate is a rumor.

Use judges where no deterministic test exists, with three cautions. Pick a judge from a different model family than the agent, because a judge from the same family grades its own kind kindly. Calibrate against human-rated samples before trusting the numbers. And on high-stakes decisions, ensemble two judges and treat disagreement as a case for a human.

The decision between the two is not a taste question. It is a testability question.

	Deterministic verdict	LLM judge
What it checks	A rule the code can apply	A quality a model scores
Failure output	Names the cause	Returns a number
Reproducible	Yes, same input, same result	No, sampling and drift
Cost	A function call	A model call
Blind spots	Only what the rule encodes	Silence, perspective, causality
Needs calibration	No	Yes, or the score is a rumor
When to use	Wherever a rule exists	Only for the gaps

The order is the discipline: deterministic first, judges for the residue, humans for the long tail. A verdict layer built the other way around is a confidence layer, and a confidence layer is exactly what the green dashboard already is, wearing a different name.

What to Instrument First

If you are starting from a dashboard that cannot see semantic failure, build in this order.

The 48-Hour Path

Record denials and throttles before anything else. This is the cheapest fix with the largest effect. Your gate layer already knows when it denies. Emit the event today, and your clean traces become meaningful within a day. You will learn things about your own agent within the first week that no dashboard has ever shown you: which actions it tries that get blocked, which limits it touches, how often it is one retry away from a constraint it never sees.

This is the payoff the two commenters were describing. Neither of them built a new dashboard. Both of them asked for the same thing: stop deleting the evidence when it happens. The 48-hour path is that, and it is a day of work.

The Weeklong Path

Add one deterministic verdict per high-cost output. Pick the failure that costs the most when it slips through, write the rule that catches it, and make sure your evidence line records when the rule fires. For grounded answers, that is the citation check above. For structured output, that is schema conformance.

One rule that actually fires is worth more than a judge scoring everything at 0.87. A single verdict that caught one real incident last week is doing more for your trust in the system than a scoring layer that has never disagreed with the agent once, because a scorer that never disagrees is either perfect or blind, and you cannot tell which.

The Monthlong Path

Instrument the evaluation layer itself. Judge models drift, and a drifting judge is indistinguishable from a drifting agent without calibration records. Record the judge's score distribution weekly and alert when it shifts. You are only as reliable as the layer that decides you are reliable.

This is the habit the operation that publishes this article keeps: every review gate's scores stay on file, and a gate whose score history you cannot inspect is a gate you cannot trust. The practice transfers directly. If your evaluation stack has no history, you have no way to know it has gone soft, and the first sign of a soft judge is a slow rise in wrong answers that all score fine.

The Habit That Binds Them

Put the evidence line in front of humans, not just dashboards. You should see: approved, denied, throttled, verdict passed, verdict failed, with reasons and causes. That view is the audit trail. Dashboards summarize. Evidence lines prove.

Common First-Week Traps

Every team that adopts this schema makes the same five mistakes in the first week. If you know them in advance, you skip a week of confusion.

Trap one: recording denials as error spans. The instinct is to treat a blocked action like a failed request, mark the span red, and page someone. A denial is not an incident. It is the gate working. If you page on every denial, you will tune the alert to silence within a week, and then the denials are invisible again. Record them as events with reasons, and only alert on the rate or the pattern, not the single occurrence.

Trap two: emitting only the happy path. It is easy to instrument the approval and forget the denial, because the denial path runs less often and the code is a few lines further from the main flow. Test the denial path. If you cannot produce a denial in staging, you have not actually wired the event.

Trap three: the retry wrapper that still swallows the 429. The most common implementation of the retry wrapper catches the error, backs off, retries, and never emits anything. That is the original bug, rebuilt in the fix. The emit call belongs inside the catch, before the backoff, and it must fire on the first 429, not the last.

Trap four: putting content in span attributes anyway. The spec puts prompt and completion text in span events so a Collector can strip them. Teams that skip the Collector config and log content as attributes are writing customer PII into their telemetry store. If you capture content at all, run the filter from this article, and write down why you kept it.

Trap five: judging everything and verifying nothing. A judge layer that scores every output at 0.8 to 0.9 creates the feeling of coverage without any of its substance, because the score never disagrees with the agent. The discipline is backward: one deterministic rule that actually fires is worth more than a judge that never says no. Build the rules first, then add judges for the gaps the rules cannot reach.

What the Schema Does Not Do

Now the honest part, because this discipline has a real ceiling and pretending otherwise is how teams get burned.

No production setup catches every hallucination. The realistic bands are directional: a frontier model with RAG and structured output sits around 1-3% factual hallucination on grounded Q&A; open-domain answers run 5-15%; a multi-step agent with weak grounding can be wrong on 10-30% of tasks; a heavily guarded domain pipeline can get under 1%.

Two things to do with those numbers. First, treat them as directional, not gospel. They come from practice surveys, not a controlled benchmark, and your agent will land somewhere in its own range depending on your domain, your grounding, and your guardrails. Measure your own rate; do not quote someone else's.

Second, use them to set the target. You do not aim for zero hallucinations. You aim for misses that are cheap to detect and bounded in harm, and when a miss does slip through, you want the record of the check that was supposed to catch it.

The reasoning visibility gap stays open. Tool calls are observable; the decision process that produced them is not. The schema records what happened at the boundaries: what was called, what was blocked, what was checked. It does not record why the agent chose those calls, because the reasoning was never emitted in the first place.

Trajectory-level tracing narrows the gap, and deterministic verdicts evaluate the evidence the agent used, not its hidden reasoning. That is the honest ceiling of the discipline, and designing against it is better than pretending it is closed. What the schema does is give you the strongest possible picture of the agent's behavior without access to its hidden reasoning, which is the difference between trusting a run and trusting a guess.

There is also a social ceiling. The evidence-event schema only works if the team that runs the agent treats a denial as data, not as a defect to hide. I have seen teams ship the schema and then quietly stop recording denials because a stakeholder asked why the agent was being blocked so often. The absence of denials then tells you nothing again, except that somebody is embarrassed. The discipline is as much about the culture that keeps the evidence as the code that emits it.

Why This Discipline Matters More Outside the Big Labs

One more observation before the closing, because it changes who should read this.

The teams that most need the evidence-event schema are not the ones building frontier models. Those teams have evaluation engineering as a job title. The teams that need it are the ones deploying agents on top of models they do not control, with a small platform team and a pager rotation: the startup running a support bot, the mid-size company wiring an agent into its CRM, the developer in an emerging market building for customers whose internet and power are not guaranteed.

That last group is my own context, and it is why the retry trap is not an abstraction to me. When your network is hostile, retry-by-instinct is survival. The same instinct is what deletes your evidence.

Teams in this position cannot afford a dedicated evaluation engineer. They can afford the discipline in this article: a gate that emits events, a retry that records what it touched, one deterministic check on the highest-cost output. None of it requires a model, a budget, or a title. It requires paying attention.

Agent observability is one of the rare skills where the reliability layer costs discipline instead of capital. That is exactly the kind of work builders in emerging markets can do better than the labs, because they have been living with unreliable infrastructure for longer and have the scars to prove it.

What Changes for a Team

The concrete difference shows up in the first incident review after the schema is live.

Before, the review started with a trace that looked fine, and the conversation stalled on a shrug: the model was probably confused, the context was probably stale, we should probably add more examples to the prompt. Nobody could prove anything, because the evidence had never been recorded.

After, the review starts with an evidence line. The agent was approved here, denied there, throttled on the billing call, and the verdict check failed on a missing citation. The wrong answer is no longer a mystery to be argued about. It is a trail to be walked, and every step of the trail names a specific fix: the gate rule was too loose, the retry hid the limit, the verdict needs a broader corpus.

That is the whole value of the discipline. It moves agent debugging from discussion to inspection. The trace told you the agent ran. The evidence line tells you what was prevented, what was checked, and what slipped past the check. When a miss happens, you do not argue about it. You read it.

The Shape of the Argument

Put it together and the pattern is simple.

The dashboard tells you the agent ran. The evidence-event schema tells you it was checked.

If your telemetry only records what was allowed, it cannot prove what was prevented, and a green trace tree stays perfectly compatible with a confident wrong answer. Record the denied, the throttled, and the failed verdict, and the green trace tree finally has something to be right about.

The two strangers in my comments were not asking for a feature. They were describing the missing layer between a trace and a proof. One of them saw it from the gate side, the other from the transport side, and they landed on the same event line. That convergence is why I believe this is the next layer of the discipline, not a personal preference: people who build agents for a living are hitting the same wall from different rooms.

---

A green dashboard and a wrong answer used to be a story I told about other people's systems. Then the comments arrived, and I realized the same wall was in my own evidence line, in a quieter form: a history of accepted work that could not tell me what had been rejected.

Have you ever watched a green dashboard hide a wrong answer? What signal was missing?

---

Cross-link: This is the monitoring-layer companion to How to Verify AI Agent Work: State Machines, Approval Gates, and Least-Privilege Access. The verification article built the gates; this article records what they prevent.

Further reading:

• The comment thread on How to Verify AI Agent Work, where the denial-first and throttle-event design converged