What to Trace in an LLM App with OpenTelemetry: Signal vs Noise

TL;DR

• LLM tracing with OpenTelemetry standardizes visibility across model inference, tool execution, and vector retrieval using the gen_ai.* semantic conventions.
• Capturing prompt versions, token counts, time-to-first-token, provider errors, and tool parameters provides high-value debugging signal.
• Logging full prompt and completion payloads on every single production request creates expensive storage noise, compliance risks, and privacy leaks.
• Emitting separate spans for every streaming token chunk or dumping raw embedding vectors into span attributes severely degrades observability collector pipelines.
• Standardizing telemetry at the network layer with an AI gateway establishes consistent sampling, automated redaction, and unified cost attribution across providers.

Production LLM applications routinely generate gigabytes of distributed trace data that cloud bills and observability pipelines cannot sustain, even while engineers struggle to identify why an agent stalled or failed. LLM tracing with OpenTelemetry standardizes telemetry collection across generative AI operations, but uncurated instrumentation creates massive noise that obscures actionable failure signals. Bifrost, an open-source AI gateway created in Go by Maxim AI, captures standardized GenAI telemetry at the network layer to isolate critical signals before payload bloat degrades downstream collectors. This guide breaks down what to capture in your trace spans, what to drop, and how to structure a clean, cost-efficient telemetry pipeline.

Why Traditional APM Fails on LLM Workloads

Traditional Application Performance Monitoring (APM) tracks system health through HTTP response codes, process memory, and endpoint latency distributions. In non-AI microservices, an HTTP 200 response with sub-second latency indicates normal execution. In contrast, large language model applications often fail silently: a model can return a successful HTTP 200 payload that contains a severe hallucination, an ungrounded calculation, a refused prompt, or an empty tool call.

Standard request metrics cannot reveal whether an application retrieved appropriate knowledge chunks, how many tokens were spent on multi-step reasoning, or whether an internal agent entered an infinite retry loop. When engineering teams attempt to solve this visibility gap by turning on default auto-instrumentation packages, they frequently encounter the opposite problem: telemetry flood.

Instrumenting modern AI orchestration frameworks can generate dozens of nested spans for a single end-user prompt. If the application utilizes agents or iterative retrieval-augmented generation (RAG), each step multiplies the span volume. Unfiltered pipelines export megabytes of repetitive text, internal serialization routines, and redundant middleware hooks directly to storage backends. This telemetry volume expands cloud costs without helping engineers locate the root cause of an incident. Clean LLM observability requires deliberate signal isolation based on open industry standards.

The Core Signals of LLM Tracing with OpenTelemetry

OpenTelemetry GenAI semantic conventions define a vendor-neutral schema for instrumenting generative AI operations across spans, metrics, and events. Maintained under the Cloud Native Computing Foundation (CNCF), the gen_ai.* namespace establishes standard attribute names for model parameters, token counts, and operational types so telemetry works interchangeably across backends.

The OpenTelemetry specification models generative AI interactions into discrete spans:

• Client and Gateway Spans (gen_ai.client / gen_ai.gateway): Record the boundary call to the model provider, capturing the requested model, target server, token consumption, and response metadata.
• Agent and Orchestration Spans: Model higher-level reasoning sequences where an agent evaluates user goals, selects external tools, and reviews outcomes.
• Vector Retrieval Spans: Capture vector database queries, embedding model operations, similarity thresholds, and document chunk references.
• Tool and Execution Spans: Document external function calls, parameters passed, execution duration, and tool outputs.

The following Python snippet demonstrates how an application emits an idiomatic OpenTelemetry GenAI span with the standard attributes required for meaningful debugging:

import time
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("rag.pipeline", "1.4.0")

def execute_model_call(prompt_template_id: str, prompt_text: str, client):
    with tracer.start_as_current_span("chat completion") as span:
        # Standard GenAI metadata
        span.set_attribute("gen_ai.operation.name", "chat")
        span.set_attribute("gen_ai.provider.name", "anthropic")
        span.set_attribute("gen_ai.request.model", "claude-3-5-sonnet-20241022")
        span.set_attribute("gen_ai.request.temperature", 0.2)
        span.set_attribute("gen_ai.request.max_tokens", 1024)

        # Engineering and governance context
        span.set_attribute("app.prompt.template_id", prompt_template_id)
        span.set_attribute("app.prompt.version", "v2.4.1")
        span.set_attribute("client.tenant_id", "tenant-8912")

        start_time = time.perf_counter()
        try:
            response = client.messages.create(
                model="claude-3-5-sonnet-20241022",
                max_tokens=1024,
                messages=[{"role": "user", "content": prompt_text}]
            )

            # Post-call usage and response attribution
            span.set_attribute("gen_ai.response.model", response.model)
            span.set_attribute("gen_ai.response.finish_reasons", [response.stop_reason])
            span.set_attribute("gen_ai.usage.input_tokens", response.usage.input_tokens)
            span.set_attribute("gen_ai.usage.output_tokens", response.usage.output_tokens)

            span.set_status(Status(StatusCode.OK))
            return response.content[0].text

        except Exception as exc:
            span.record_exception(exc)
            span.set_status(Status(StatusCode.ERROR, str(exc)))
            span.set_attribute("error.type", exc.__class__.__name__)
            raise

Using consistent attribute naming ensures that traces recorded in one service link seamlessly to spans emitted by downstream proxies or data layers.

What Telemetry Is High-Value Signal

High-value telemetry represents data that directly accelerates root-cause analysis, tracks financial burn, or pinpoints operational bottlenecks. When troubleshooting production LLM workloads, on-call teams do not start by reading thousands of text interactions; they look for quantitative anomalies, structural breaks, and regression indicators.

The most valuable data points to collect across your spans include:

1. Exact Model Identifiers and Fallbacks: Always capture both gen_ai.request.model (what the application asked for) and gen_ai.response.model (the actual checkpoint that served the response). If a proxy or gateway transparently routed traffic to a secondary model due to rate limits or an outage, this difference explains unexpected response patterns.
2. Standardized Token Counts: Record gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and reasoning or cache-read tokens when available. These metrics drive unit-economic calculations and allow teams to identify token bloat caused by unchecked conversation histories.
3. Time to First Token (TTFT): For streaming responses, overall request duration tells an incomplete story. A prompt that takes ten seconds to return its final token might have emitted its first token in 350 milliseconds (an excellent user experience), or it might have stalled for eight seconds before emitting anything (a poor experience).
4. Prompt Template Versions: Tying the prompt version to the trace is critical. Code deployments are versioned via git commits, but prompt templates are often updated independently. Without a prompt.version attribute, isolating whether a regression was caused by a code push or a prompt rewrite requires tedious manual correlation.
5. Tool Invocations and Parameters: When an agent invokes external functions or Model Context Protocol tools, recording the tool identifier, argument schema, and HTTP return status isolates whether an error originated from model reasoning or an external API dependency.
6. Tenant, Virtual Key, and Cost Attribution: Attaching tenant identifiers or gateway virtual keys allows teams to calculate per-customer margins and enforce departmental rate limits.

The table below summarizes the core span types and the indispensable attributes that deliver actionable debugging signal:

Span Type	Target Operation	Required OTel Attributes	Primary Diagnostic Value
gen_ai.client	Direct Model Inference	gen_ai.request.model, gen_ai.response.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.response.finish_reasons	Tracks provider latency, token usage, cost attribution, and model routing.
retrieval	Vector DB / Knowledge Base	db.system, db.collection.name, retrieval.top_k, retrieval.returned_documents_count, retrieval.min_score	Reveals whether context retrieval underperformed or fetched irrelevant chunks.
tool.execute	MCP or Agent Tool Invocation	gen_ai.tool.name, gen_ai.tool.call_id, rpc.system, error.type	Isolates external tool failures from core model hallucinations.
agent.step	Iterative Agent Loop	agent.iteration_index, agent.max_iterations, agent.goal_id, agent.decision_type	Detects infinite reasoning loops, repetitive function calls, and agent stalling.

Tracing these operations creates an understandable trace tree that pinpoints failures within seconds.

What Telemetry Is Just Noise and How to Eliminate It

Telemetry noise refers to high-volume, low-utility data that consumes network bandwidth, inflates database bills, slows down query interfaces, and exposes organizations to security compliance violations. When configuring OpenTelemetry in AI applications, several common practices generate catastrophic amounts of noise.

                  ┌──────────────────────────────────────────────────┐
                  │                 Application Call                 │
                  └─────────────────────────┬────────────────────────┘
                                            │
                        Is it a structural execution step?
                                     /             \
                                   Yes              No
                                   /                 \
                 ┌───────────────────────┐      ┌─────────────────────────┐
                 │  Create Parent Span   │      │ Drop Span (Avoid Noise) │
                 │  (Model, Tool, RAG)   │      │ (No Per-Chunk SSE Spans)│
                 └──────────┬────────────┘      └─────────────────────────┘
                            │
               Contains large text or vectors?
                           /         \
                         Yes          No
                         /             \
        ┌─────────────────────────┐   ┌─────────────────────────┐
        │ Drop Raw Payloads       │   │ Record High-Value Meta  │
        │ Sample Rare Errors Only │   │ (Tokens, TTFT, Version) │
        └─────────────────────────┘   └─────────────────────────┘

Anti-Pattern 1: Span-Per-Chunk in Streaming Responses

When consuming Server-Sent Events (SSE) from streaming endpoints, naive auto-instrumentation or custom wrappers sometimes open an OpenTelemetry span for every returned token chunk. A model response generating 1,500 tokens can emit 1,500 individual spans for a single HTTP interaction. This practice overwhelms local memory buffers and crashes collectors. Instead, streaming operations must use a single parent span. Record chunk timing internally, set a span event or attribute for the initial chunk (TTFT), and finalize the span when the stream closes.

Anti-Pattern 2: Unfiltered Full-Payload Capture

Storing the full text of prompts and completions on every single production request is the fastest way to turn an observability pipeline into an unusable cost center. A high-throughput service handling tens of millions of tokens daily will generate gigabytes of unstructured string data. Furthermore, raw prompt capture routinely ingests personally identifiable information (PII), proprietary source code, or credential secrets, violating data privacy regulations like GDPR and HIPAA. Full content capture should be disabled by default in production and enabled only via dynamic sampling or in staging environments.

Anti-Pattern 3: Embedding Vectors in Span Attributes

Embedding models convert text chunks into numerical vectors containing 768, 1,536, or more floating-point dimensions. Storing an array of 1,536 floats directly inside span attributes yields zero human-readable value inside trace dashboards. It bloats trace payload sizes, increases network serialization overhead, and exhausts memory limits in downstream storage engines. Record the dimension size, the embedding model name, and the text hash, but never serialize raw float arrays into traces.

Anti-Pattern 4: Micro-Spans for Internal Framework Plumbing

Modern agent libraries construct deep call stacks involving input validation, schema parsing, JSON serialization, and internal hook triggers. Wrapping each dictionary transformation or regex validation in its own span clutters the trace view with dozens of microsecond spans. An engineer attempting to understand why an LLM call took six seconds will be forced to scroll past hundreds of irrelevant framework spans. Focus instrumentation exclusively on boundaries: network calls, database queries, and external tool invocations.

The following comparison clarifies how to separate actionable signal from noise across key telemetry dimensions:

Telemetry Element	High-Value Signal (Keep)	Low-Value Noise (Drop)	Reason to Prune
Streaming Output	Single span with duration, total tokens, and TTFT event.	Distinct spans or span events for every token chunk.	Prevents collector memory exhaustion and span explosion.
Payload Data	Prompt template ID, template hash, token usage, finish reason.	Full prompt and completion text for 100% of successful queries.	Avoids regulatory PII exposure and reduces storage bills.
Embeddings	Model name, token count, embedding dimensions, document chunk IDs.	Raw multi-dimensional floating point vector arrays.	Floats cannot be searched in trace UI and inflate serialization overhead.
Tool Execution	Tool name, argument keys, execution latency, error messages.	Intermediate internal framework serialization calls.	Eliminates trace clutter and focuses triage on real I/O boundaries.
Agent Loops	Iteration step count, decision state, tool selection intent.	Every internal state machine tick and variable assignment.	Prevents deep, unreadable trace trees during multi-turn execution.

Pruning these low-utility elements maintains trace fidelity while keeping data pipelines lean and cost-effective.

Managing the Data Plane: Sampling, Redaction, and Cost

Keeping an OpenTelemetry pipeline efficient requires smart data-plane management. The OpenTelemetry Collector functions as a proxy between application runtimes and storage backends, providing a centralized location to filter, sample, and redact telemetry before it crosses network boundaries.

Tail-Based Sampling for High-Impact Traces

Traditional head-based sampling makes a decision at the start of a request (for example, keeping 5% of all incoming requests uniformly). This approach fails for LLM applications because critical edge cases, such as timeout errors, rate-limiting events, and extreme latency outliers, occur in the long tail. Uniform 5% sampling discards 95% of the rare failures you need to debug.

Tail-based sampling evaluates the entire trace lifecycle before deciding whether to persist the data. A well-architected collector pipeline applies policies to:

• Retain 100% of traces containing an HTTP 4xx, 5xx, or span status error.
• Retain 100% of traces exhibiting p95 or p99 latency spikes.
• Retain 100% of traces where fallback routing was triggered.
• Retain a modest 1% to 2% sample of normal, fast, HTTP 200 requests for baseline comparison.

Centralized PII Redaction

Instead of relying on application developers to manually clean prompt strings, organizations can deploy redaction processors in the OpenTelemetry Collector. Using regular expression matching or entity recognition, the collector replaces credit card numbers, email addresses, and API secrets with masked tokens before data is written to disk. This guarantees that even when full-payload sampling is activated for incident diagnosis, compliance posture remains intact.

Standardizing Telemetry at the Gateway with Bifrost

Instrumenting every service, microservice, and agent script with local OpenTelemetry SDKs can lead to inconsistent naming conventions, divergent sampling rules, and maintenance friction. Moving standard telemetry capture to the network layer via an AI gateway solves this inconsistency.

Bifrost sits directly between applications and upstream model providers, operating as a centralized control plane for all inference traffic. Because Bifrost adds only 11 microseconds of overhead per request at 5,000 requests per second in sustained benchmarks, it captures comprehensive telemetry without degrading end-user response times.

Every request that passes through Bifrost is automatically normalized to OpenTelemetry GenAI semantic conventions. Applications simply point their OpenAI, Anthropic, or LangChain clients to the gateway, and Bifrost handles:

• Unified OpenTelemetry Export: Bifrost emits traces via the standard OpenTelemetry Protocol (OTLP) to any compatible backend, including Grafana, Honeycomb, or Jaeger.
• Native Prometheus Metrics: Built-in Prometheus metrics aggregate token usage, request latency, and cache efficiency without requiring separate scraper configurations.
• Datadog Connector: For enterprise environments, the Datadog connector streams APM traces, LLM observability data, and metrics directly into existing corporate dashboards.
• Resilience and Cache Visibility: When Bifrost executes automatic fallbacks across providers or serves responses from semantic caching, those events are recorded as first-class span attributes, giving engineers full visibility into operational cost savings and routing behavior.
• Automated Log Exports: Teams can stream structured audit logs directly to cloud storage buckets using native log exports for long-term analytical review.

Beyond routing, Bifrost applies governance and security controls (virtual keys, budgets, guardrails, audit logs) centrally, and Bifrost Edge extends that same governance and security to AI traffic on employee machines, with endpoint enforcement on each device. This endpoint integration guarantees that developer desktop tools, local coding agents, and terminal scripts adhere to the same app governance and MCP governance policies configured on the main infrastructure gateway, ensuring complete visibility across the entire development fleet.

Practical Architecture: Setting Up an OTel LLM Collector Pipeline

A robust telemetry pipeline combines an application runtime, an AI gateway, and an OpenTelemetry Collector to govern the flow of data.

Below is an annotated otel-collector-config.yaml file demonstrating how to configure tail-based sampling, attribute normalization, and secure batch exporting for LLM workloads:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  # Ensure collector does not exceed memory boundaries under load
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 15

  # Redact sensitive credentials and patterns from attributes
  redaction:
    allow_all_keys: false
    attributes:
      - action: replace
        key: gen_ai.prompt
        pattern: '(?i)(bearer\s+[a-z0-9_\-\.]+)|(api[_-]?key[_-]?=\s*[a-z0-9_\-\.]+)'
        replace: "[REDACTED_SECRET]"

  # Retain high-value anomalies while discarding routine noise
  tail_sampling:
    decision_wait: 10s
    num_traces: 10000
    expected_new_traces_per_sec: 2000
    policies:
      # Policy 1: Keep 100% of errors
      - name: drop_errors_policy
        type: status_code
        status_code: { status_codes: [ ERROR ] }
      # Policy 2: Keep latency outliers (greater than 5 seconds)
      - name: latency_outliers_policy
        type: latency
        latency: { threshold_ms: 5000 }
      # Policy 3: Keep traces where provider fallback occurred
      - name: fallback_routing_policy
        type: numeric_attribute
        numeric_attribute: { key: "bifrost.fallback_triggered", value_condition: { equal_to: 1 } }
      # Policy 4: Retain a 2% baseline sample of normal requests
      - name: probabilistic_baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 2.0 }

  batch:
    send_batch_size: 512
    timeout: 5s

exporters:
  otlp:
    endpoint: "trace-backend.internal:4317"
    tls:
      insecure: false

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, redaction, batch]
      exporters: [otlp]

This configuration ensures that operational anomalies and failovers are preserved with complete fidelity, while routine successful executions are sampled down before storage costs accrue. For teams planning long-term infrastructure investments, reviewing the LLM Gateway Buyer's Guide provides additional architectural patterns for deploying high-scale AI proxy layers.

Frequently Asked Questions

What are the official OpenTelemetry semantic conventions for LLMs?

The official OpenTelemetry GenAI semantic conventions define standardized naming rules for generative AI telemetry under the gen_ai.* attribute namespace. Maintained by the CNCF, they specify span formats, metric instruments, and event structures for model names, token consumption, inference parameters, and tool invocations across providers.

Should production LLM traces capture full prompt and response bodies?

Production traces should not capture full prompt and completion bodies by default. Storing complete conversational payloads across 100% of traffic inflates network and database storage expenses while exposing systems to data privacy and PII compliance violations. Payloads should be captured only in staging environments or on targeted error samples.

How does OpenTelemetry measure token usage across different providers?

OpenTelemetry standardizes token metrics using common span attributes, specifically gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. Providers report these numbers differently in raw HTTP responses, but compliant instrumentation libraries and gateways normalize them into standard fields to enable uniform cost tracking.

What is the difference between head-based and tail-based sampling for LLM apps?

Head-based sampling makes an ingestion decision at the immediate start of a request, discarding a fixed percentage of calls without knowing their final outcome. Tail-based sampling evaluates the entire trace after completion, allowing pipelines to retain 100% of errors, fallbacks, and slow requests while sampling away repetitive successful calls.

How do I trace streaming LLM responses without overloading the collector?

To trace streaming responses safely, maintain a single parent span for the complete HTTP streaming connection rather than creating spans for individual token chunks. Record the duration until the first token arrives as a span event or attribute to measure time to first token (TTFT), then close the span once the stream terminates.

How does an AI gateway simplify OpenTelemetry instrumentation?

An AI gateway like Bifrost sits on the network path and automatically emits standardized OpenTelemetry spans for every request. This eliminates the need to install, configure, and maintain separate auto-instrumentation libraries across diverse microservice codebases, ensuring uniform attribute naming and metrics collection across all model providers.

Getting Started with Scalable LLM Observability

Designing an effective LLM observability pipeline requires ruthless focus on actionable signal. By adopting OpenTelemetry GenAI semantic conventions, isolating boundary spans, and implementing intelligent tail-based sampling, engineering teams can maintain complete visibility into model behavior without drowning in payload noise. Standardizing this telemetry layer at the network level with Bifrost simplifies deployment, enforces consistent data governance, and keeps operational infrastructure performant under heavy production load.

Teams evaluating how to observe, route, and govern their AI infrastructure can explore the Bifrost open-source repository or schedule a technical demo to review enterprise gateway capabilities.

Sources

• OpenTelemetry Semantic Conventions for Generative AI - Official Cloud Native Computing Foundation specification for LLM span attributes, metrics, and events.
• W3C Trace Context Specification - Industry standard defining context propagation and distributed trace headers across cloud-native architectures.
• OpenTelemetry Architecture and Collector Guidelines - Reference architecture for building high-throughput telemetry pipelines with processors, sampling, and exporters.