Silent failures in AI agents: how to detect a wrong answer

Silent failures in AI agents are the cases where everything looks successful: the request returns HTTP 200, no exception is raised, no span is in error, and yet the answer is wrong, incomplete or built on a bad tool result. A classic dashboard stays green. This is the main trap when operating agents.

This article offers a method to detect a wrong answer: trace every agent step, separate technical errors from content errors, then attach an evaluation result to the trace. Attribute names come from the OpenTelemetry GenAI specification, read at the source. They apply to any OTLP backend, including Dynatrace SaaS. No percentage is given here, because none was read in a primary source.

Silent failures: why alerts do not see them

The OpenTelemetry specification asks for the span status to be set to Error when the operation ends in an error, along with error.type. It also makes clear there is no fixed rule for « valid but wrong » results: whether an outcome counts as an error depends on context. For an agent, a well-formed but wrong answer therefore raises no error signal by default.

Common shapes of silent failure, to adapt to your agent:

• a tool returns an empty or stale result and the agent ignores it;
• the model stops for an unexpected reason (length limit, filter) and the answer is truncated;
• the agent picks the wrong tool or repeats the same call;
• the answer reads well but rests on no retrieved data.

None of these raises an HTTP error. For the cost side of an agent loop, see the token cost of an AI agent; the signals are the same spans.

Tracing the agent: one span per step

The GenAI conventions define agent spans, with Development status (so subject to change). The invocation span is named invoke_agent {gen_ai.agent.name}: kind CLIENT for a remote agent, INTERNAL when it runs in the same process. Each tool call produces an execute_tool {gen_ai.tool.name} span of kind INTERNAL, with gen_ai.tool.name required.

This structure answers the first diagnostic question: which step went wrong? An execute_tool span with an empty result points at the tool. An inference span with an unexpected gen_ai.response.finish_reasons value (a recommended attribute, an array of reasons the model stopped) points at the model. To link these spans across services, the OpenTelemetry Collector remains the usual entry point.

Capturing content, with care

To judge an answer, you must see it. The specification provides the OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENT variable, with values NO_CONTENT (default), SPAN_ONLY, EVENT_ONLY and SPAN_AND_EVENT. An unrecognised value falls back to NO_CONTENT.

Tool arguments and results (gen_ai.tool.call.arguments, gen_ai.tool.call.result) are Opt-In: the documentation warns that some attributes may contain sensitive information. Enable capture on a sample, with personal data masked, rather than everywhere. Tool-side abuse detection is covered in the article on MCP tool poisoning.

Detecting a wrong answer: the evaluation event

The specification defines the gen_ai.evaluation.result event (Development status, Recommended level). It carries the result of evaluating model output for quality, accuracy or other characteristics. It should be parented to the evaluated span when possible; otherwise set gen_ai.response.id.

The evaluation name (gen_ai.evaluation.name, for example Relevance) is required. The numeric score (gen_ai.evaluation.score.value) and the label (gen_ai.evaluation.score.label) are required « if applicable »; the label must stay low-cardinality. If the evaluator itself fails, error.type says so: a broken evaluator is not an answer judged correct.

Who computes the score? The specification does not say: it only describes how to carry it. A deterministic check (does the answer cite a retrieved source? is the JSON valid?), human review on a sample or a second model are all possible. A judge model is itself fallible and must be calibrated on annotated cases.

On Dynatrace SaaS, the « Dynatrace for AI Observability » page states, as read in the documentation, online LLM-as-a-judge evaluations on production traces, with scores written as business events linked to the source traces, and OpenTelemetry ingestion using the GenAI conventions. No equivalence with the attributes above is claimed: check in your tenant.

In practice: a four-step plan

1. Instrument the agent with a library that emits invoke_agent and execute_tool; check the names actually produced, as the specification is in development.
2. Define a success criterion per task (answer grounded in a source, tool called, valid format) before hunting for anomalies.
3. Emit an evaluation result on a sample of requests, attached to the span or to gen_ai.response.id.
4. Track the share of « fail » results per agent version, like a quality SLI, with an objective (SLO) set by the team rather than borrowed from elsewhere.

To see why a raw failure rate is not enough, read the difference between observability and monitoring. And if trace volume gets heavy, tail sampling can keep the traces with bad evaluations first.

How to check it works

Replay a request whose expected answer is known, forcing a tool failure (a tool returning an empty list). The trace should show the execute_tool span, and the evaluation should come out as a failure. If it stays « pass », the criterion is too loose. If no event arrives, check the link to the span or the gen_ai.response.id.

Good practices

• Keep the technical error rate apart from the rate of answers judged wrong.
• Store the evaluator’s explanation (gen_ai.evaluation.explanation) for human review.
• Limit content capture and mask sensitive data.
• Version the agent (gen_ai.agent.version) to compare before and after a prompt change.
• Keep a set of annotated cases to recalibrate the evaluator.

Frequently asked questions

What is a silent failure of an AI agent?

It is a run that ends without a technical error but whose result is wrong or unusable. Spans carry no error status, hence the need to evaluate the content.

Are the OpenTelemetry GenAI conventions stable?

No: the pages read show Development status for agent spans, the evaluation event and most attributes. Names may change; only error.type is marked stable there.

Should every prompt and answer be captured?

No. Capture is off by default (NO_CONTENT) and content can be sensitive. Enable it on a sample and protect access.

Conclusion

Detecting a wrong answer needs a signal that availability metrics do not provide: an evaluation of the content, attached to the trace. OpenTelemetry offers a format to carry it; the quality criterion remains for the team to define. Next, see the guardrails for AI agents in production.

Sources: semantic-conventions-genai repository (Apache 2.0), OpenTelemetry: Recording errors, Dynatrace documentation (SaaS).

This is an independent blog, not affiliated with Dynatrace LLC. Product names belong to their owners. The conventions cited are in development: check the specification version before relying on them.

---

Originally published on span.observer. French version: https://span.observer/fr/echecs-silencieux-agents-ia-detecter/.