Agent Black Box: Teaching an AI Agent to Explain Its Own Failures

Banka Mani Bhargava — Sun, 26 Jul 2026 08:03:09 +0000

How I built a record-guard-explain pipeline for AI agents on self-hosted SigNoz, and the three-layer metrics bug that taught me more about OpenTelemetry than the docs did.

Key Takeaways

Most agent observability stops at "send some spans to a dashboard." Agent Black Box adds live guardrails that kill runaway runs and an automated postmortem agent that explains failures after the fact, using SigNoz's own MCP server.

Getting agent metrics to show up correctly took fixing three separate, independent bugs: a missing flush, an instance-ID cardinality explosion, and a temporality mismatch, each one looking identical from the dashboard ("No Data" or a wrong zero) but with a genuinely different root cause.

A SigNoz alert form silently saved two thresholds as 0 and inverted a comparison operator, no error shown. Lesson: verify anything configured through a UI by reading it back through the API, not by trusting the screen.

The hook

My test agent called the same search tool 47 times in a row and burned real API cost before anything told me. Nothing crashed. No log said anything was wrong. It just kept going, the way a car with a stuck accelerator keeps going, right up until it doesn't.

That's the actual failure mode of AI agents in 2026, and it's the one almost nobody instruments for. Most "agent observability" means shipping a few spans to a dashboard and hoping you're watching it at the moment it breaks. You're not. You're asleep, or in a meeting, or building something else, and the agent is quietly turning your token budget into nothing.

I built Agent Black Box to fix that. It records every agent run as real OpenTelemetry spans, guards the run live so a runaway loop or cost spike gets killed before it does more damage, and, the part that took the longest and mattered the most, explains what happened afterward, by having a second agent query the observability backend itself and write a root-cause report.

The problem, precisely

Three things are true about AI agents that make them hard to watch:

They fail silently. A stuck loop doesn't throw an exception. A cost spiral doesn't page anyone. The agent just keeps doing the wrong thing, successfully, from its own point of view.
The failure signal and the root cause are different things. A guardrail catching a loop tells you a loop happened. It doesn't tell you why the agent decided to repeat itself, that's upstream, in the reasoning, and you have to go dig for it.
Nobody wants to dig. After an incident, the actual investigative work, pulling the trace, cross-referencing logs, checking token counts, is exactly the tedious work a second AI agent is good at, if you give it the tools to do it.

So the design became three pieces, in one sentence: record everything properly, kill bad runs live, and have an agent read the wreckage back and explain it.

Architecture

                    ┌──────────────────────────────┐
  your agent  ──►   │  agent_blackbox.trace()      │
  (any LLM /        │   • gen_ai.* semconv spans   │   OTLP
   framework)       │   • token + cost attributes  │  ──────►  SigNoz (Foundry, :8080)
                    │   • guardrail span processor │  :4318         │
                    └──────────────────────────────┘                │
                                    │                               │
                          RunawayAgentError                         │
                          (loop / cost / error cascade)             │
                                                                    │
  $ blackbox postmortem <trace_id>                                  │
        └─► LLM agent ──MCP──► signoz_get_trace_details ────────────┘
             (localhost:8000)    signoz_search_logs
                                 signoz_query_metrics
             └─► Root-cause report (markdown, printed + saved)

Everything runs against self-hosted SigNoz, deployed with Foundry. One YAML file (casting.yaml) and one command (foundryctl cast) brings up SigNoz and its MCP server together. That MCP server is the piece that makes the "explain" step possible: it's a real, first-party interface an LLM can call to query traces, logs, and metrics like any other tool.

The recorder: making agents speak OpenTelemetry

The public API is deliberately tiny: two lines to wrap an agent.

from agent_blackbox import trace, traced_llm, traced_tool

with trace(agent="research-agent") as run:
    resp = traced_llm(client.messages.create)(model="claude-haiku-4-5-20251001", ...)
    result = traced_tool("web_search")(search_fn)(query="...")

Under the hood, this emits proper OpenTelemetry GenAI semantic-convention spans, invoke_agent, chat, execute_tool, plus a small custom agent.* namespace for what the spec doesn't cover yet: loop signatures, run cost, guardrail trips. Getting the semconv attributes right mattered more than I expected going in. A trace that looks like a GenAI trace but doesn't use the standard attribute names is a trace SigNoz's Query Builder can't group or aggregate on, and the whole dashboard depends on that grouping working.

Guardrails: making the kill itself observable

Four detectors run as a SpanProcessor watching the span stream live: loop detection (same tool and same arguments, repeated), a cost ceiling, an error cascade, and a depth limit for nested agents. Each one raises a RunawayAgentError, but the interesting design decision was making the kill itself show up in SigNoz, not just in a Python traceback:

tracer = otel.get_tracer()
with tracer.start_as_current_span("guardrail.trip") as span:
    span.set_attribute(semconv.AGENT_GUARDRAIL_NAME, guardrail_name)
    span.set_attribute(semconv.AGENT_GUARDRAIL_REASON, reason)
    span.set_status(Status(StatusCode.ERROR, reason))
raise RunawayAgentError(guardrail_name, reason, trace_id)

A guardrail that fires silently in application logs is a guardrail nobody trusts. This one leaves a guardrail.trip span and a metric increment, so the kill is exactly as visible as the failure it prevented.

The differentiator: a postmortem agent that reads its own trace

This is the part I'm actually proud of. blackbox postmortem <trace_id> connects to the SigNoz MCP server as a genuine MCP client, gives Claude the real tool list (signoz_get_trace_details, signoz_search_logs, signoz_query_metrics), and lets it investigate: not summarize a template, actually look at the evidence and reason about it.

Here's an unedited excerpt from a real report, generated against a trace where the loop-detection guardrail fired:

Root cause
The guardrail (guardrail.trip) is a symptom detector, not the cause, it worked as designed, catching a runaway pattern and aborting the run before it could burn more resources. The actual root cause is upstream in the agent's decision loop: the agent issued the exact same web_search tool call three times in a row... Notably, the trace contains no LLM/completion span between the tool calls, so the instrumentation does not show what reasoning/prompt state produced the repeated call. This is an observability gap that prevents confirming whether the repetition originated from the LLM output itself or from an orchestration bug.

Nobody told it to distinguish "symptom" from "cause." Nobody told it to notice the missing LLM span and flag it as its own limitation. That's what evidence-grounded reasoning over real telemetry looks like when it works.

What it looks like when it's all wired together

Seven panels, all built with SigNoz's Query Builder: runs by outcome, latency percentiles, token burn and cost by model, tool success rate, slowest tool calls, and guardrail trips. Four alert rules watch cost spikes, tool error rates, guardrail trips, and, the one I like most, an absent-data alert that catches the agent going silent instead of erroring loudly.

What went wrong (the honest part)

The recorder, the guardrails, the postmortem agent, all of that came together in a day. The dashboard metrics took three separate bugs to get right, and each one taught me something that isn't obvious from the OpenTelemetry docs.

Bug 1: metrics never left the process. Every dashboard panel built on traces worked immediately. Every panel built on metrics said "No Data." The cause: my demo scripts are short-lived CLI processes that finish in a couple of seconds, well under PeriodicExportingMetricReader's default 15-second export interval, so the reader's own timer never got a chance to fire before the process exited. shutdown() alone doesn't force a final flush; you need an explicit force_flush() first. Traces looked fine because BatchSpanProcessor.shutdown() does flush reliably. The same assumption just doesn't hold for the metrics reader.

Bug 2: metrics arrived, then fragmented into uselessness. After the flush fix, "No Data" became sparse, near-empty results. The SigNoz Metrics Explorer is what actually cracked this one: one metric had 16 samples spread across 14 separate time series, nearly one series per sample. The cause was OpenTelemetry auto-generating a random service.instance.id per process, since I never set one explicitly. Every CLI invocation is a new process, so every run got its own random instance ID and its own brand-new, single-point series. Increase/Rate need at least two points within the same series to compute a delta, and a series with one point has nothing to compute against. Fix: pin service.instance.id to a constant string, so repeated runs of the same script accumulate into one continuous series instead of scattering.

Bug 3: consolidated, but still computing the wrong number. Even after fixing cardinality, the "Guardrail trips" panel stubbornly showed 0, despite the guardrail having tripped at least five times across my test runs. This is the one that actually taught me something about OpenTelemetry's default assumptions: cumulative temporality assumes one long-running process. Each of my short-lived processes started its own counter at zero and typically reported exactly one value, "1 trip", before exiting. Across three separate processes that each tripped once, the raw exported values looked like [1, 1, 1]. A naive last-minus-first delta across those points is 1 - 1 = 0, which is correct math for a single continuously-running counter and wrong for three independent ones. The fix wasn't in any query, it was one environment variable: OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta, so each export reports "+1 this interval" directly instead of a cumulative-since-start value that doesn't compose across processes.

Three bugs, each one masquerading as "still no data," each with a genuinely different root cause. The lesson that generalizes: OpenTelemetry's defaults are tuned for one long-running server, not a fleet of short CLI invocations. If that's your workload, expect to fight the flush timing, the resource cardinality, and the temporality model, in that order.

The smaller lessons

A few more from the build log, because they're the kind of thing you only learn by hitting them:

SigNoz's IAM has a two-step trap. Creating an API key under a service account doesn't grant that account read access to observability data. You also need to assign it a role (signoz-admin or signoz-viewer) under Settings -> Roles, which is a separate page the API-key creation flow never mentions.
A UI form can silently save the wrong value. Two of my four alert rules saved with target: 0 instead of the real threshold, and a third saved with the comparison operator inverted, no error, no warning, the UI just reported success. I only caught it by checking the saved rule against the API (GET /api/v2/rules) instead of trusting what the form showed. That's now a standing habit: verify anything you configure through a UI by reading it back, not by trusting the screen.
Two guardrails can both be "correct" about the same symptom. My loop detector and error-cascade detector both watch the same tool-call stream, and when a tool is both looping and failing, whichever check runs first in the code wins the race. That's an implementation detail, not a judgment call, and it's worth being honest about instead of pretending the system is smarter than it is.
Reproducibility is worth testing for real, not assuming. I tore down the whole deployment, cloned the repo fresh, and ran foundryctl cast -f casting.yaml against nothing but the two committed files. It came back up clean: UI at 200, MCP correctly demanding auth at 401. That's a much stronger claim than "it should be reproducible."

FAQ

Why not just use SigNoz Cloud instead of self-hosting?
Self-hosting via Foundry means the whole deployment, SigNoz plus its MCP server, is defined in one committed YAML file. That makes the setup reproducible from a clean clone with a single command, which matters if you want anyone else to verify it actually works rather than take your word for it.

What's the difference between this and just asking an LLM to look at a SigNoz dashboard?
The guardrails run live, during the agent's execution, not after. A dashboard only tells you what already happened; the guardrail span processor can kill a run mid-flight, before it burns more cost or keeps looping. The postmortem step is separate and happens after the fact.

Does the postmortem agent ever get it wrong?
It's bounded by what the trace, logs, and metrics actually contain. In one real run, it correctly flagged that it couldn't confirm the root cause because the trace had no LLM span to inspect, rather than guessing. That's the behavior I wanted: reasoning grounded in the available evidence, with the gaps stated as gaps.

Can this work with agent frameworks other than a raw Anthropic client?
The recorder wraps any callable, traced_llm and traced_tool are decorators around whatever function makes the actual call, so it isn't tied to one SDK. Anything that returns token usage in its response object can be adapted.

Conclusion

The technically hardest part of this project wasn't the AI agent. It was making sure short-lived processes report metrics correctly, a problem OpenTelemetry's defaults quietly assume you don't have. The most interesting part was watching a postmortem agent correctly reason that a guardrail is a symptom detector, not a root cause, without being told to think that way.

Record. Guard. Explain. Full-circle observability tells you what happened. Agent Black Box stops it happening, and explains it afterward.

Code, dashboard export, alert rules, and the full build log are on GitHub: agent-blackbox.

DEV Community: Banka Mani Bhargava