Day 27: Observability and Tracing for Agentic Systems — Telemetry, Logs and Metrics

Standard monitoring watches status codes, latency and memory. Those numbers can look perfect while an agent is failing. It can return HTTP 200 OK with a completely made-up answer, or burn through forty planning steps going in circles. To debug an agent you need to see its decisions, not only its servers.

In plain terms: A flight tracker that only shows "plane is in the air" tells you nothing about whether it is heading to the right city. You need the flight path.

Diagram: An agent can return HTTP 200 with normal latency while giving a hallucinated answer or looping on planning. Standard server health metrics cannot see that, so agent systems need traces that record their decisions. See the animated version.

The three pillars of agent observability

• Traces capture the end-to-end path of a run. Every model call, retrieval and tool call becomes a nested span with its time, input tokens and output tokens, plus enough prompt context to reproduce the step.
• Metrics track the cost and health of the system: token use, cache hit ratio, context-window utilisation and cost per transaction, by team or workflow.
• Logs record structured state changes in a multi-agent graph, so you can find the step where the reasoning diverged from what you expected.

Diagram: A trace records every step of an agent run as nested spans: planning, retrieval, tool calls and answer generation, each with its latency and its input and output token counts. The span names and numbers are illustrative. See the animated version.

Diagram: Agent observability uses three kinds of data. Traces show the path of one run, metrics track tokens, cache hits, context use and cost, and logs record state changes so you can find where reasoning diverged. See the animated version.

Standards: OpenTelemetry and correlation IDs

• Vendor-neutral instrumentation. Using OpenTelemetry (OTel) means your telemetry can flow into Prometheus, Grafana, Jaeger or a commercial backend without locking you to one vendor. OpenTelemetry also has emerging conventions for generative AI attributes, which are still evolving, so check the current names.
• Correlation IDs. Give each incoming request a unique trace ID that follows it through the gateway, application, vector database and agent workers. Search that ID and you see one user interaction end to end.

Diagram: A unique trace or correlation ID attached to each incoming request travels through the gateway, application, vector database and agent workers, so operators can follow one user interaction end to end. OpenTelemetry propagates it across services. See the animated version.

Here is a minimal Python example. It assumes you have installed opentelemetry-sdk and opentelemetry-exporter-otlp, and that a collector or a recent Jaeger is listening for OTLP on port 4317. The llm and run_tool functions stand in for your own.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="localhost:4317", insecure=True)))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("support-agent")

def handle(question: str, request_id: str):
    with tracer.start_as_current_span("agent.run") as run:
        run.set_attribute("app.request_id", request_id)

        with tracer.start_as_current_span("llm.plan") as span:
            plan = llm(question)                                  # your model call
            span.set_attribute("gen_ai.usage.input_tokens", plan.input_tokens)
            span.set_attribute("gen_ai.usage.output_tokens", plan.output_tokens)

        with tracer.start_as_current_span("tool.run_sql") as span:
            span.set_attribute("tool.name", "run_sql")
            rows = run_tool("run_sql", plan.sql)
            span.set_attribute("tool.rows_returned", len(rows))   # 0 rows is a useful signal
        return rows

One caution: prompts can contain customer or internal data, so redact sensitive fields before they go into spans or logs. With tracing in place, agent debugging moves from guesswork to something you can inspect, replay and audit.

Coming Up Next

Day 28: Human-in-the-loop architecture: machine second, human first in practice.

Observability #OpenTelemetry #DistributedTracing #MLOps #SRE

---

Originally published at https://sureshpallapothu.in/blog/day-27-agent-observability, where this post includes animated diagrams.