DEV Community: Samip Ghimire

Agent Black Box: a flight recorder and SRE copilot for AI agents, built on SigNoz

Samip Ghimire — Sun, 26 Jul 2026 02:59:17 +0000

An AI agent that fails in production is hard to debug, because the interesting part happened inside a chain of LLM calls and tool calls you never saw. My hackathon project records every one of those steps into SigNoz, then has a second agent read the telemetry back and explain what went wrong. This is what I built, what broke along the way, and what the telemetry looked like.

What I built

Agent Black Box has two halves. The first is a customer-support agent (a small LangGraph graph) that handles refund tickets by calling tools: look up the order, retrieve the refund policy from a vector store, decide, and reply. Every LLM call, tool call, retry, and RAG lookup is instrumented with OpenTelemetry and shipped to a self-hosted SigNoz.

The second half is an SRE copilot. When the agent misbehaves, SigNoz fires an alert to a webhook, and the copilot turns that alert plus the surrounding telemetry into a root-cause report with a suggested fix. The whole thing runs against SigNoz installed via Foundry, and the casting.yaml + lockfile are in the repo so the deployment reproduces.

The agent task itself is deliberately boring. The point is the observability, so I gave the agent three ways to break on purpose.

How the agent breaks

Each failure is a deterministic switch, so I can reproduce the same incident before and after a fix:

Retry storm: the order-lookup tool returns HTTP 503 and the agent retries with no budget, burning time and tokens.
Bad RAG: the vector search returns the wrong policy document with a low confidence score, and a valid refund gets denied.
Tool loop: ambiguous retrieval keeps confidence low, so the agent re-queries the same policy in a loop.

These are the failure classes the hackathon brief calls out, and they map cleanly onto telemetry: a retry storm is a span that repeats, a bad retrieval is a low rag.confidence attribute, a loop is one span name appearing over and over in a trace.

What SigNoz records

Each support session becomes a single trace, support_agent.session, with the LLM turns and tool calls nested underneath. The LLM spans follow the OpenTelemetry GenAI semantic conventions, so gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and a computed gen_ai.usage.cost_usd all sit on the span. A tool-loop session shows up as a tall waterfall of tool.retrieve_policy spans, which is exactly the shape of the bug.

Metrics feed a dashboard I called the Agent Flight Deck: token cost, retries by tool, failed sessions by failure mode, tool calls, and low-confidence retrievals.

Logs are bridged through the OTel logging handler, so every structured log line the agent writes carries its trace and span IDs. Clicking a spike on a trace lands on the log that explains it, without matching timestamps by hand.

The instrumentation setup is small. A resource, three OTLP exporters, and a logging handler:

resource = Resource.create({
    "service.name": "agent-black-box",
    "service.version": AGENT_VERSION,
    "service.instance.id": "agent-black-box-agent-1",  # see "what surprised me"
})
tracer_provider = TracerProvider(resource=resource)
tracer_provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint=f"{ENDPOINT}/v1/traces")))

From an alert to a root cause

I built a metric alert in SigNoz on agent.retries grouped by tool.name, routed to a webhook channel pointed at the copilot. When a retry storm happens, SigNoz posts an Alertmanager-style payload to the copilot, carrying the failure mode, the tool, and the observed value in the annotation text.

The copilot reads that payload, pulls the evidence, and writes a report. Here is a real one, from a real alert:

Incident: Retry Storm in tool lookup_order
- Observed value: 11 (threshold 5)
- Tool: lookup_order
- Failing span: tool.lookup_order
Likely root cause: lookup_order retried an upstream 503 with no
retry budget and no fallback path.
Suggested fixes:
1. Add max_retries=3 with exponential backoff to lookup_order.
2. Add a fallback: create_support_ticket when the order API is unavailable.

The rule I held myself to: every line in the report has to trace back to telemetry the copilot actually read. The LLM writes the prose; it does not invent the number 11 or the span name. If evidence is missing, the report says so.

The fix, then a replay

The copilot suggested a retry budget with a fallback, so I shipped exactly that as v2 behavior, controlled by one config value. RETRY_BUDGET=0 is the buggy v1 that storms. RETRY_BUDGET=3 is the fixed v2 that gives up after three tries and opens a support ticket instead.

Replaying the same scenario on v2, the tool.lookup_order span now records three retries instead of eleven, the session ends with failure_mode=none, and the retry-storm alert stops firing because the peak retry count drops below the threshold. In the trace, the loud red span becomes a short one with a graceful fallback next to it. That before-and-after, driven by the same seed and visible in the same trace view, is the story I wanted the tool to tell.

What surprised me

The instrumentation was the easy part. Getting a metric alert to actually fire taught me the most, and none of it is in a quickstart.

SigNoz's rate() and increase() returned zero for my retry counter, even though ClickHouse clearly showed eleven retries. Two things were wrong. First, every agent run is a short-lived process, and the OpenTelemetry SDK stamps a fresh service.instance.id on each one, so a single metric fractured into thirty-one separate short series. Pinning a stable instance id fixed that. Second, increase needs a series that rises across the window, but these counters jump to eleven and vanish in thirteen seconds, so the delta reads as zero. Switching the alert to max (the peak value in the window) gave me the number I wanted, and it reads naturally: the worst session had more than five retries.

I also learned that a high-cardinality label like session_id on a metric is a trap. It makes every session its own series and breaks aggregation. Session id belongs on traces and logs, where it does the correlation work; the metrics stay low-cardinality.

Two more that cost real time: SigNoz evaluates alerts on a delayed window (about two minutes behind now), so fresh data sits in a blind spot and the rule reads "query result is nil" until you widen the window. And after a Docker restart, SigNoz's replicated ClickHouse tables came up read-only, which looks exactly like "no data" in the UI until you run SYSTEM RESTORE REPLICA on them. Both are written up in the repo so the next person does not lose the afternoon I did.

Wrapping up

The build proves a simple claim: with OpenTelemetry and SigNoz, an AI agent stops being a black box. You can watch a session as a trace, chart its cost and retries, get paged when it storms, and read a root-cause report grounded in the telemetry instead of a guess. Code, the reproducible Foundry deployment, and the full debugging notes are in the repo.

Repository: https://github.com/SamipSGz/agent-black-box
SigNoz install docs: https://signoz.io/docs/install/docker/
OpenTelemetry GenAI semantic conventions: https://opentelemetry.io/docs/specs/semconv/gen-ai/

Jumping from a log line to the exact trace that caused it, with self-hosted SigNoz

Samip Ghimire — Tue, 14 Jul 2026 12:43:38 +0000

A service in front of me was slow maybe one request in ten, and the logs just said 503, retrying. Retrying how many times? Inside which request? The logs and the traces lived in two different tabs and I was matching them up by hand, squinting at timestamps. This is a walkthrough of how I self-hosted SigNoz, sent traces and logs from a small Python service, and got a log line to link straight to the trace that produced it. That one link is the feature I ended up liking most.

What I was trying to do

SigNoz is an OpenTelemetry-native observability tool you can run on your own machine, so traces, logs, and metrics land in one place instead of three vendors. I did not want the full production setup. I wanted the smallest thing that would answer one question: when a log says a request retried, can I click that log and see the whole request as a trace?

For that I needed two pieces sending data to SigNoz over OTLP: spans for the request, and logs emitted while a span is active. If the logs carry the trace ID, SigNoz can connect them.

Installing SigNoz (the part where my first instructions were wrong)

Most guides I found said to clone the repo and run docker compose up from deploy/docker. I did that, and the folder was not there. The main branch has deprecated the Compose manifests. deploy/README.md now says so directly and points to a tool called Foundry. That was my first real lesson: check the current install docs, because the internet is a few months behind.

The current path is three steps. Install the CLI:

curl -fsSL https://signoz.io/foundry.sh | bash

Write a casting.yaml:

apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    flavor: compose
    mode: docker

Then deploy:

foundryctl cast -f casting.yaml

Foundry generates the Compose files, validates Docker, and starts the containers. A minute later docker ps showed the stack running: a signoz container publishing the UI on 8080, and an ingester publishing OTLP on 4317 (gRPC) and 4318 (HTTP). I used HTTP because it is one less thing to think about.

The UI at http://localhost:8080 sent me to a signup page on first run. It creates a local admin account, and the password rule is stricter than I expected: at least 12 characters with an uppercase letter, a lowercase letter, a number, and a symbol. My first two attempts bounced before I read that.

Reference: SigNoz install docs.

Instrumenting a small Python service

I wrote a script that pretends to be an order service. The lookup fails with a 503 most of the time and retries until it succeeds, which is the behavior I wanted to see in a trace. The whole thing is about 40 lines.

import logging
import random
import time

from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

from opentelemetry._logs import set_logger_provider
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter

ENDPOINT = "http://localhost:4318"
resource = Resource.create({"service.name": "order-service"})

# traces
tracer_provider = TracerProvider(resource=resource)
tracer_provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint=f"{ENDPOINT}/v1/traces"))
)
trace.set_tracer_provider(tracer_provider)

# logs, bridged into the standard logging module
logger_provider = LoggerProvider(resource=resource)
logger_provider.add_log_record_processor(
    BatchLogRecordProcessor(OTLPLogExporter(endpoint=f"{ENDPOINT}/v1/logs"))
)
set_logger_provider(logger_provider)
logging.getLogger().addHandler(LoggingHandler(logger_provider=logger_provider))
logging.getLogger().setLevel(logging.INFO)

tracer = trace.get_tracer("order-service")
log = logging.getLogger("orders")


def lookup_order(order_id: str) -> dict:
    with tracer.start_as_current_span("lookup_order") as span:
        span.set_attribute("order.id", order_id)
        attempts = 0
        while True:
            attempts += 1
            if random.random() < 0.7 and attempts < 12:
                log.warning("order service 503, retrying (attempt %d)", attempts)
                span.add_event("retry", {"attempt": attempts})
                time.sleep(0.2)
                continue
            span.set_attribute("lookup.attempts", attempts)
            log.info("order %s resolved after %d attempts", order_id, attempts)
            return {"id": order_id}


if __name__ == "__main__":
    for order_id in ["A-1042", "A-2091", "A-3157", "A-4480", "A-5561"]:
        with tracer.start_as_current_span("handle_request") as root:
            root.set_attribute("request.id", f"req-{order_id}")
            lookup_order(order_id)
        time.sleep(0.5)
    tracer_provider.shutdown()   # flush the last batch before exit
    logger_provider.shutdown()

Install the two packages and run it:

pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
python retry_demo.py

The piece that makes the rest work is the LoggingHandler. It routes Python's normal logging output to SigNoz, and because the log calls happen inside start_as_current_span, each log record picks up the active trace and span IDs. The OpenTelemetry Python docs cover this under logs.

Finding the traces

In the Traces explorer each run showed up as handle_request with a nested lookup_order. The default time range is the last 30 minutes, and the first load came up empty for a few seconds before the spans appeared, so give it a moment or hit refresh.

Opening one trace shows the waterfall. The trace I picked took 1.02 seconds, almost all of it inside lookup_order, and the span carried lookup.attempts: 6. The retry events I recorded sit on the span as five markers. So a retry storm looks like a retry storm here, without me counting log lines.

The part I came for

On the span details panel there is a Logs tab. Opening it showed the log lines recorded during that span: five 503, retrying warnings and the final resolved after 6 attempts, already scoped to this one request. I did not search by timestamp. The trace ID rode along on the log records, so SigNoz connected the two views instead of me doing it by eye.

There is an Open in Logs Explorer button on that panel too, which drops you into the full Logs view filtered to the same context. Going the other way, the Logs explorer shows every line the service emitted, and each one carries the trace and span IDs that make the jump back possible.

One trace is a story; a metric is the pattern

A single trace tells me why one request was slow. It does not tell me whether retries are getting worse across all requests. For that I added one metric: a counter that ticks up every time the lookup retries.

from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter

meter_provider = MeterProvider(
    resource=resource,
    metric_readers=[
        PeriodicExportingMetricReader(
            OTLPMetricExporter(endpoint=f"{ENDPOINT}/v1/metrics"),
            export_interval_millis=5000,
        )
    ],
)
metrics.set_meter_provider(meter_provider)
retries_counter = metrics.get_meter("order-service").create_counter(
    "order.retries", unit="1", description="order lookup retries"
)

Then retries_counter.add(1, {"order.id": order_id}) on each retry. Metrics export on their own timer rather than per-span, so a script that fires a handful of requests and exits barely registers. I wrapped the request loop in a while that ran for a few minutes so the counter had time to climb across several export intervals.

In SigNoz I made a dashboard, added a Time Series panel, set the data source to Metrics, and picked order.retries with a Rate aggregation. The build flow is worth noting: the panel does not query until you press Stage and Run Query, which caught me out when the graph sat on No Data for a moment.

Now the same failure reads three ways in one tool: the trace shows a single slow request, the logs explain each retry, and the metric shows the retry rate climbing across all of them. That is the whole pitch of keeping the three signals in one place.

What tripped me up

The deprecated Compose folder was the first thing, covered above. A few more were not obvious.

The exporter endpoint needs the signal path when you pass it to the constructor. OTLPSpanExporter(endpoint="http://localhost:4318") sends nothing useful. It has to be .../v1/traces, and logs go to .../v1/logs. If instead you set the OTEL_EXPORTER_OTLP_ENDPOINT environment variable, the SDK appends the path for you, which is the opposite of the constructor behavior and easy to mix up.

Short scripts exit before the data ships. BatchSpanProcessor and BatchLogRecordProcessor send on a timer, so a script that finishes in half a second is gone before the first batch leaves. Calling shutdown() on both providers at the end forces a flush. Without it I saw nothing in the UI and wrongly assumed the setup was broken.

Logs only correlate if they run inside an active span and go through the OTel handler. A print(), or a logger writing straight to stdout, produces a line with no trace ID, and then SigNoz has nothing to link. Every log I wanted connected had to be emitted between start_as_current_span and the end of that block.

What I would take from this

The correlation is the payoff. Once logs carry trace context, the tab switching and timestamp matching from the start of this post just goes away, and a warning becomes one click from the request that caused it. The setup that buys you this is small: point the OTLP exporters at the collector, install the logging handler, and keep your log calls inside spans.

If you want to try it, the two links worth having open are the SigNoz install docs and the OpenTelemetry Python docs. Start with the retry script above, watch one trace, then open its Logs tab. That single view is what sold me.