The "Logic Span": Using OpenTelemetry to Trace Hallucinations

The Signal: The 500 Error of the Mind
You’re staring at a "Task Failed" status in your dashboard. You check your logs and see a clean 200 OK from the LLM provider. The network was fast, the JSON parsed correctly, but the agent still decided that the best way to "Summarize an Invoice" was to delete the database entry for it.

Most developers use OpenTelemetry (OTel) to find slow database queries or network bottlenecks. But in 2026, the bottleneck isn't the network—it's the Reasoning. When an agent goes off the rails, a standard log is just noise. To find the "bug" in an LLM, you need to see exactly where the logic diverged from the plan.

Phase 1: The Architectural Bet
We are shifting from Flat Logging to Semantic Tracing.

The Vendor Trap is relying on the "History" tab in your LLM provider's cloud platform. It shows you the prompt and the completion, but it doesn't show you the Internal State of your application around that call.

The Ownership Path is wrapping every "Thought" or "Reasoning Step" in a dedicated OTel Span. We treat a hallucination like a stack trace. By nesting spans, we can visualize the parent "Plan" and identify exactly which child "Thought" introduced the error.

Phase 2: Implementation (Attribute-Heavy Spans)
We don't just log the output; we log the Hyperparameters. If an agent hallucinates at 3:00 AM, was it because of a bad prompt or because the temperature was set too high for a deterministic task?

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("agent.reasoning.monitor")

def execute_agent_step(step_name, prompt_version, params):
    # The Logic Span: Wrapping the thought, not just the network call
    with tracer.start_as_current_span(f"Agent Thought: {step_name}") as span:
        # Attribute Injection: Hardening the trace with metadata
        span.set_attribute("llm.temperature", params.get("temp", 0.7))
        span.set_attribute("llm.top_p", params.get("top_p", 1.0))
        span.set_attribute("config.prompt_version", prompt_version)

        try:
            # Simulated Agent Logic
            response = call_llm(params) 

            # Record the 'Thought' as a Span Event for granularity
            span.add_event("llm_completion_received", {"output.length": len(response)})

            if "error" in response:
                span.set_status(Status(StatusCode.ERROR))
                return None

            return response

        except Exception as e:
            span.record_exception(e)
            span.set_status(Status(StatusCode.ERROR, str(e)))
            raise

Phase 3: The Senior Security & Testing Audit
I put this tracing strategy through a professional Site Reliability and Security Audit. Here is why your telemetry might take down your own system.

1. The TSDB Cardinality Explosion (Infra Crash)
   The Fault: You want to track exactly what the agent was thinking, so you put the user's ID or the prompt snippet directly into the span name (e.g., tracer.start_as_current_span(f"Thought: {user_prompt}")).
   The Audit: Time Series Databases (like Prometheus or Datadog) index based on span names. If every span name is unique because it contains dynamic LLM text, you will cause a High Cardinality Explosion. Your TSDB will run out of memory and crash your observability stack.
   The Fix: Span names must be low-cardinality and static (e.g., "Agent Thought"). Dynamic data must always be stored as Span Attributes.

2. The Async Context Bleed (Logic Bug)
   The Fault: You are running a multi-agent swarm asynchronously. You initiate multiple execute_agent_step functions concurrently.
   The Audit: OpenTelemetry relies on context variables (like contextvars in Python or AsyncLocalStorage in Node.js) to link child spans to parent spans. If you don't explicitly pass or isolate the OTel context when spawning background tasks, Agent A's thought process will accidentally attach to Agent B's trace. You end up with a tangled, useless "Spaghetti Trace."
   The Fix: Manually propagate the OTel Context object into any thread pool or async queue you use for your agents.

3. The PII Compliance Breach (Security)
   The Fault: You log the raw LLM output into an event attribute so you can read it later to debug the hallucination.
   The Audit: LLMs routinely process PII (emails, names, API keys). By piping raw_output into OpenTelemetry, you are transmitting unencrypted PII to a third-party logging vendor. This violates GDPR, SOC2, and basic security hygiene.
   The Fix: Implement an OTel SpanProcessor at the global level. This acts as a middleware that regex-scrubs sensitive attributes (like SSNs, emails, or Auth tokens) before the telemetry data is exported from your server.

Phase 4: Checklist (The Architect’s Standard)
[ ] Nest Your Spans: The "Action" (Tool Call) should always be a child span of the "Thought" (Reasoning).

[ ] Log Hyperparameters: temperature, model_version, and seed are as important to debug as the code itself.

[ ] Redact Your Traces: Ensure no PII is flowing into your observability stack via a global SpanProcessor.

[ ] Monitor Token Truncation: Log the context_token_count and an is_truncated boolean. If an agent hallucinates, you need to know if your system silently deleted the correct answer from the context window before the LLM even saw it.

The Bottom Line: You can't fix what you can't see. Stop treating LLMs like black boxes and start treating them like distributed systems. Trace the logic. Find the hallucination.