LLM Observability: What to Measure and Where to Instrument It

TL;DR

• LLM observability captures execution traces, operational metrics, and contextual logs across non-deterministic model calls where traditional HTTP status codes fail to indicate output correctness.
• The five critical metrics for production AI workloads are token consumption, real-time cost, time to first token (TTFT), fallback rates, and tool execution success.
• Comprehensive instrumentation requires capturing telemetry at four distinct architecture layers: the AI gateway, application orchestration code, external tool servers, and employee client endpoints.
• Bifrost provides gateway-level OpenTelemetry traces and native Prometheus metrics with 11 microseconds of overhead, unifying request telemetry across more than 20 model providers.
• OpenTelemetry GenAI semantic conventions standardize model names, token counts, and invocation spans into vendor-neutral schemas that export directly to existing monitoring platforms.

Production AI applications fail in ways traditional application performance monitoring cannot detect, because a model request can return an HTTP 200 status code while delivering an incomplete completion, triggering a costly retry loop, or hallucinating a schema. Bifrost, an open-source AI gateway developed in Go by Maxim AI, routes, governs, and instruments model requests from a centralized control point before traffic reaches upstream providers. Effective LLM observability requires engineering teams to track granular operational signals (tokens, cost, streaming latency, failovers, and tool calls) while choosing the correct architectural layers for telemetry collection. This guide details what metrics engineering teams must collect, why each signal matters, and where in the modern AI stack instrumentation belongs.

What is LLM Observability

LLM observability is the practice of collecting, correlating, and analyzing telemetry across large language model interactions, agentic execution steps, and supporting infrastructure to understand system behavior, diagnose degradation, and control operational costs. While traditional monitoring evaluates deterministic code paths through CPU load, memory utilization, and network latency, model monitoring must evaluate probabilistic operations where prompt inputs directly dictate compute requirements and financial expense.

Unlike relational databases or microservices, large language models exhibit non-deterministic behavior: identical requests can yield variable token counts, variable execution times, and different tool invocations depending on sampling parameters. Furthermore, third-party model APIs operate as opaque services where upstream congestion or rate limits manifest as sudden latency spikes or provider-side 429 errors. Without structured telemetry that correlates prompt structure with downstream execution, platform teams cannot isolate whether a user-facing slowdown originates in the application logic, vector database retrieval, tool execution, or model inference.

Implementing comprehensive LLM observability provides platform engineers with three core operational capabilities:

• Attribution and accounting: Linking every prompt and completion to an authenticated team, virtual key, or end-user session for precise internal chargeback.
• Root-cause isolation: Dissecting multi-turn agent chains into discrete spans to identify whether latency stems from tool execution, prompt bloat, or slow upstream generation.
• Reliability management: Tracking upstream provider degradation, error frequency, and fallback activations before transient provider incidents cause widespread application outages.

The 5 Critical Signals to Measure in LLM Observability

To maintain reliable systems, platform engineers must collect specific operational telemetry. Generic request counters and round-trip HTTP timers fail to capture the economics and mechanics of large language models. Production systems require instrumentation centered on five foundational signals: token usage, real-time cost, streaming latency, fallback activations, and tool execution telemetry.

1. Token Usage (Input, Output, and Context Growth)

Token volume serves as the fundamental unit of compute, context consumption, and billing in generative AI systems. Teams must separate prompt tokens (input) from completion tokens (output), as providers price these streams at radically different rates.

Total Tokens = Prompt Tokens + Completion Tokens
Cost = (Prompt Tokens × Input Rate) + (Completion Tokens × Output Rate)

Tracking input tokens identifies prompt bloat, uncompressed conversational history, and oversized context injections from Retrieval-Augmented Generation (RAG) pipelines. A sudden increase in input tokens often signals that an application is repeatedly passing unbounded chat histories without pruning older turns. Conversely, tracking completion tokens monitors output length, catches run-away generation loops caused by missing stop sequences, and tracks model verbosity across releases.

Modern observability pipelines also record cached prompt tokens. As providers introduce prefix caching and prompt caching discounts, measuring cache hit ratios reveals whether prompts are structured with static prefixes to maximize provider-side KV-cache reuse. Through semantic caching, Bifrost enables teams to intercept semantically identical queries before transmission to the provider, recording local cache hits that reduce upstream token consumption to zero.

2. Real-Time Cost

Because model providers bill on a variable per-token structure, tracking infrastructure expenditure after the monthly invoice arrives creates substantial financial risk. Real-time cost monitoring computes the exact financial cost of each model interaction at the moment of completion.

Calculating request cost requires an up-to-date catalog of provider pricing models, factoring in prompt token prices, completion token prices, cached token discounts, and reasoning token surcharges. Correlating cost metrics with metadata such as user IDs, application environments, and tenant keys allows organizations to enforce cost boundaries. Bifrost maintains a dynamic model pricing catalog and records costs directly on every request span via virtual keys, enabling teams to implement automated budgets and rate limits that halt spend before runaway agent loops trigger billing overruns.

3. Latency and Time to First Token (TTFT)

Standard round-trip latency (the duration between sending an HTTP request and receiving the final byte of the response) is an inadequate metric for streaming interfaces. Users reading generated responses judge responsiveness by the delay before the interface begins outputting text.

Teams must break model latency into three distinct phases:

• Time to First Token (TTFT): The time elapsed between dispatching the request and receiving the initial streaming chunk from the provider. TTFT reflects model queue time, prompt processing time (prefill), and upstream network transit.
• Inter-Token Latency (ITL): The time between consecutive tokens during generation (often measured as tokens per second). ITL measures provider generation speed and GPU throughput.
• Gateway and Middleware Overhead: The processing duration added by local proxies, guardrail inspections, and routing logic. In published benchmarks, Bifrost adds only 11 microseconds of overhead per request at 5,000 requests per second, ensuring infrastructure layers do not contribute to perceived user delay.

Monitoring TTFT percentiles (P50, P95, P99) highlights provider congestion and prompt prefill degradation long before total generation time reflects an incident.

4. Fallbacks, Retries, and Error Classifications

Model providers regularly experience transient rate limits (HTTP 429), context length limits (HTTP 400), internal server faults (HTTP 500), and regional gateway timeouts (HTTP 504). Tracking retry counts and fallback execution paths is critical to maintaining operational reliability.

An observability system must measure:

• Attempt Trail: The sequence of attempts made for a single logical request, capturing which API key or provider was contacted first.
• Fallback Trigger Rate: The frequency with which primary models fail and traffic diverts to secondary models or backup providers.
• Error Breakdown: Categorization of failures into client errors (such as prompt formatting or token ceiling violations) versus provider infrastructure outages.

When an outage occurs, Bifrost executes automatic fallbacks across configured provider chains. The gateway records each attempt as an individual sub-span while preserving the root trace context, allowing engineers to identify provider degradation without experiencing client-side downtime.

5. Tool Calls and Agentic Executions

As AI applications evolve from basic conversational interfaces into autonomous agents, large language models spend significant time formulating structured tool calls. Observability must extend into the Model Context Protocol (MCP) servers and external APIs invoked by the model.

Key tool metrics include:

• Tool Invocation Frequency: Which functions or MCP tools are selected by the model for a given task.
• Argument Validation Failures: How often the model outputs malformed JSON or invalid parameter types that fail execution schemas.
• Tool Execution Duration: The latency of the underlying database query, API request, or code sandbox relative to total task duration.
• Agent Loop Depth: The number of reasoning steps, re-prompts, and tool calls executed before reaching a final answer or termination condition.

Capturing tool metrics prevents scenarios where engineers incorrectly blame slow model generation for delays caused by downstream external APIs.

---

Metric Dimension	Primary Signals	What It Reveals	Key Telemetry Type
Tokens	Prompt, completion, cached, reasoning	Context bloat, infinite generation loops, cache efficiency	Metric counter / Span attribute
Financial Cost	Cost per request, cumulative user spend	Margin erosion, project budget exhaustion, model cost efficiency	Metric gauge / Span attribute
Latency	TTFT, inter-token duration, gateway overhead	Upstream provider queue delays, streaming responsiveness, proxy overhead	Metric histogram
Reliability	429 rate, 5xx rate, fallback activation count	Upstream outages, quota exhaustion, circuit breaker trips	Metric counter / Span event
Agent / Tools	Tool duration, schema errors, loop count	Inefficient tool definitions, broken downstream APIs, runaway agent logic	Distributed trace span

---

Where to Measure: Gateway, Application, Tool, and Endpoint

Capturing these five signals requires placing instrumentation at the appropriate architectural boundaries. No single layer has access to all context: an API gateway possesses precise timing and multi-provider failover metadata, but lacks visibility into internal application state; conversely, client-side code understands user intent but cannot safely govern API credentials. A comprehensive strategy instruments four key layers.

+-------------------------------------------------------------------------+
|                              Client Layer                               |
|        Browser Interfaces / Developer Desktops / Bifrost Edge           |
+-------------------------------------------------------------------------+
                                    |
                                    v
+-------------------------------------------------------------------------+
|                            AI Gateway Layer                             |
|          Bifrost (Routing, Semantic Caching, Fallbacks, OTel)           |
+-------------------------------------------------------------------------+
                                    |
                  +-----------------+-----------------+
                  |                                   |
                  v                                   v
+-----------------------------------+ +-----------------------------------+
|     Application Orchestration     | |      MCP Tool Server / APIs       |
| Agent Logic, Prompt Construction  | | External Systems, Data Stores     |
+-----------------------------------+ +-----------------------------------+

1. At the AI Gateway Layer

The AI gateway sits directly between application services and external model providers, making it the most strategic location for capturing unified operational telemetry. Placing observability at the gateway captures every outbound model request regardless of programming language, SDK choice, or framework.

Gateway-level instrumentation measures:

• Exact upstream provider latency and TTFT before client application buffering.
• Provider failovers, retries, and rate limits across heterogeneous APIs.
• Centralized token counts and calculated financial costs based on standard model pricing.
• Request and response payload logging for regulatory audit compliance.

Because Bifrost acts as a high-performance proxy with a drop-in replacement API, teams capture complete model telemetry without modifying existing application logic or embedding vendor-specific SDKs into microservices.

2. In Application Code and Orchestration Pipelines

While the gateway records the mechanics of network transport and provider transactions, the application orchestration layer understands business context. Frameworks executing agentic logic or multi-step RAG pipelines must emit trace spans that envelop individual model calls.

Application-level instrumentation measures:

• User session IDs, prompt template versions, and conversational turn counts.
• Vector database retrieval latency, top-k similarity scores, and document chunk sizes.
• Business outcomes, such as user feedback (thumbs up/down) and task completion rates.
• Evaluation metrics generated by automated scoring models or guardrail checks.

By passing standard W3C trace context headers (traceparent) from application spans into requests sent through the AI gateway, engineers create end-to-end distributed traces that link a high-level user action to every subsequent model invocation and database query.

3. At the MCP Tool Server Layer

When models use the Model Context Protocol to interact with enterprise databases, file systems, and internal services, the tool servers themselves become critical instrumentation points. An unobserved tool server creates an architectural blind spot where agent latency cannot be explained.

Tool-level instrumentation measures:

• Internal execution latency of tools, shell commands, or database queries.
• Authorization decisions, credential validations, and access rejections.
• Data payload sizes returned from tools back into the model's working context window.

Monitoring tool servers ensures platform engineers detect slow database queries or failing microservices before the orchestrating model attempts repeated retries that consume excess tokens.

4. On Employee Client Endpoints

The newest operational blind spot in enterprise AI is shadow AI: developers and internal staff using desktop clients, command-line coding assistants, and browser extensions that connect directly to external model APIs without routing through centralized infrastructure.

Endpoint instrumentation captures:

• Tool usage across desktop environments, including Claude Desktop, Cursor, and terminal coding assistants.
• Outbound sensitive data (secrets, credentials, personal data) leaving developer machines.
• The inventory of MCP servers and plugins installed locally across enterprise workstations.

Beyond network routing, Bifrost applies governance and security controls centrally, and Bifrost Edge extends that same governance and security to AI traffic on employee machines, with endpoint enforcement on each device. By capturing client-side traffic and routing it through the centralized gateway, organizations ensure endpoint interactions emit identical telemetry to production microservices.

OpenTelemetry Semantic Conventions for Generative AI

To prevent vendor lock-in and allow organizations to export AI telemetry to existing monitoring platforms such as Grafana, Datadog, New Relic, or Honeycomb, the Cloud Native Computing Foundation maintains the OpenTelemetry GenAI Semantic Conventions. These standards define uniform attribute naming for spans, events, and metrics across all large language model operations.

Standardizing on OpenTelemetry ensures that whether an application calls OpenAI, Anthropic Claude, AWS Bedrock, or a local vLLM instance, telemetry attributes remain consistent across backends.

Key Span Attributes

When instrumenting model operations, traces should adhere to the official gen_ai.* attribute namespace:

{
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "name": "chat gpt-4o",
  "attributes": {
    "gen_ai.system": "openai",
    "gen_ai.request.model": "gpt-4o",
    "gen_ai.response.model": "gpt-4o-2024-08-06",
    "gen_ai.operation.name": "chat",
    "gen_ai.request.temperature": 0.7,
    "gen_ai.request.max_tokens": 2048,
    "gen_ai.usage.prompt_tokens": 482,
    "gen_ai.usage.completion_tokens": 156,
    "gen_ai.usage.cost": 0.00281,
    "gen_ai.server.address": "api.openai.com"
  }
}

Bifrost includes native OpenTelemetry export supporting both HTTP and gRPC transport protocols. Spans emitted by Bifrost automatically map provider-specific response metadata to standard OpenTelemetry attributes, calculating exact cost fields and attaching attempt indices for multi-provider fallback chains.

Prometheus Metrics for Real-Time Alerting

While distributed traces provide deep structural insight for post-incident debugging, high-throughput production operations require low-overhead aggregated metrics for dashboards and automated alerts. Prometheus metrics provide real-time visibility across latency distributions and error frequencies.

Standard metrics exposed by modern AI infrastructure include:

• gen_ai_requests_total: Counter tracking total requests partitioned by provider, model, virtual key, and HTTP status code.
• gen_ai_tokens_total: Counter tracking cumulative prompt and completion tokens partitioned by model and consumer.
• gen_ai_request_duration_seconds: Histogram tracking end-to-end request duration percentiles.
• gen_ai_time_to_first_token_seconds: Histogram tracking streaming latency percentiles.
• gen_ai_cost_dollars_total: Counter accumulating financial expenditure across teams and virtual keys.

Through its built-in Prometheus metrics integration, Bifrost exposes a high-performance /metrics scraping endpoint that Prometheus, VictoriaMetrics, or Datadog agents can poll without adding overhead to the runtime request pipeline.

# HELP gen_ai_requests_total Total number of model requests processed
# TYPE gen_ai_requests_total counter
gen_ai_requests_total{provider="anthropic",model="claude-3-5-sonnet",status="200",virtual_key="prod-search"} 14209
gen_ai_requests_total{provider="openai",model="gpt-4o",status="429",virtual_key="prod-search"} 18

# HELP gen_ai_time_to_first_token_seconds Time to first token for streaming responses
# TYPE gen_ai_time_to_first_token_seconds histogram
gen_ai_time_to_first_token_seconds_bucket{model="claude-3-5-sonnet",le="0.25"} 8412
gen_ai_time_to_first_token_seconds_bucket{model="claude-3-5-sonnet",le="0.5"} 13190
gen_ai_time_to_first_token_seconds_bucket{model="claude-3-5-sonnet",le="1.0"} 14102

Designing an Enterprise Observability Pipeline

Building a scalable telemetry architecture requires balancing visibility against data privacy, latency constraints, and storage costs. High-volume systems must ingest millions of daily events without degrading model response times or exposing customer data.

Asynchronous Telemetry Processing

Observability collection must never sit directly on the critical request path. If logging systems, external collectors, or analytical databases experience network latency, user-facing requests must not block.

Inside Bifrost, logging, metrics calculation, and trace generation run asynchronously via dedicated worker pools. When a completion terminates, the gateway immediately flushes the final response chunk to the caller while dispatching the generated telemetry span to internal queues for asynchronous batch export over OTLP. This architectural separation guarantees that comprehensive telemetry introduces zero user-perceived latency.

Payload Capture and Privacy Controls

Unlike microservice metadata, generative AI prompts and completions often contain unstructured customer information, proprietary source code, or personal data. Storing raw request bodies in centralized logging systems creates regulatory risk under SOC 2, HIPAA, and GDPR.

Platform teams must establish clear data governance rules:

• Header Enrichment: Injecting non-sensitive metadata (such as customer IDs, project tags, and prompt version hashes) via request headers to allow correlation without exposing prompt bodies.
• Redaction and Masking: Running regular-expression filters or localized redaction plugins at the gateway layer to sanitize API tokens, passwords, and personally identifiable information before telemetry export.
• Audit Trails: For regulated industries requiring full historical logging, Bifrost supports immutable, hash-chained audit logs that store encrypted interaction records in private cloud storage environments without transmitting data to external vendors.

Setting Up Proactive Production Alerts

Rather than monitoring static dashboards, engineering teams should establish automated alerts based on critical operational thresholds:

1. TTFT Degradation: Trigger a warning when P95 TTFT exceeds 1,200ms over a five-minute evaluation window, signaling upstream provider congestion.
2. Fallback Spikes: Alert on-call engineers when secondary provider routing activates on more than 2% of traffic, indicating primary provider instability.
3. Budget Runaway: Automatically throttle or halt traffic on specific virtual keys when spend crosses 80% of daily allocated limits.
4. Tool Error Rates: Notify developers when MCP tool execution errors exceed 5% within a single deployment version.

---

Frequently Asked Questions

What is the difference between LLM monitoring and LLM observability?

LLM monitoring tracks predetermined quantitative metrics over time, such as requests per second, error rates, token spend, and latency percentiles on dashboards. LLM observability is the broader capability of reconstructing and understanding complex, non-deterministic system states after the fact using detailed execution traces, contextual spans, tool invocation histories, and payload metadata. Monitoring alerts engineers that an anomaly occurred; observability provides the granular context required to diagnose why it occurred.

Why is Time to First Token (TTFT) more important than total response time for streaming LLM calls?

Time to First Token (TTFT) measures the perceived responsiveness of an AI application because users interact with conversational and generative interfaces as text streams incrementally. A request generating 800 tokens might take six seconds to complete, but if TTFT is 300 milliseconds, the user experiences immediate interaction. Conversely, a high TTFT creates noticeable latency and interface freezing, regardless of how fast the model generates subsequent tokens.

How does an AI gateway capture telemetry without adding latency?

An AI gateway like Bifrost captures telemetry asynchronously outside the critical request-response network path. Bifrost adds only 11 microseconds of overhead per request by processing request headers, matching routing rules, and streaming upstream model chunks in compiled Go routines. Telemetry metrics, OpenTelemetry spans, and audit logs are buffered into memory and dispatched via background workers, preventing downstream collector delays from impacting user response times.

Can OpenTelemetry trace agentic tool calls and multi-turn workflows?

Yes, OpenTelemetry traces agentic workflows by structuring operations into hierarchical parent and child spans using W3C Trace Context propagation. An agent run acts as the root span, while individual thought processes, vector database retrievals, tool invocations, and subsequent LLM generation steps execute as nested child spans. Standard GenAI semantic conventions record tool names, input parameters, execution durations, and exit statuses directly on each child span.

Where should LLM metrics be collected in a multi-provider architecture?

In a multi-provider architecture, operational metrics should be captured primarily at the AI gateway layer. Collecting metrics at the gateway ensures uniform token counting, standardized cost calculation, and consistent latency tracking across all model providers without requiring separate instrumentation for each vendor's SDK. Application-specific business metadata and evaluation scores can then be layered in via distributed trace headers.

How do virtual keys improve observability in multi-tenant environments?

Virtual keys serve as an attribution layer that isolates metrics, costs, and rate limits across distinct teams, applications, or end-users. By routing requests through virtual keys, platform administrators can segment usage dashboards, analyze token efficiency per business unit, track unit economics, and enforce granular budget limits without provisioning individual provider API keys for every consumer.

---

Next Steps

Establishing comprehensive LLM observability ensures production AI applications remain reliable, cost-effective, and performant as workloads scale across multiple models and autonomous agents. Engineering teams looking to centralize model telemetry, automate multi-provider fallbacks, and control token spend can evaluate Bifrost or examine the open-source implementation in the Bifrost GitHub repository. Teams can also request a Bifrost demo to discuss enterprise clustering, governance policies, and custom OpenTelemetry pipelines.

Sources

• OpenTelemetry Generative AI Semantic Conventions Specification: https://github.com/open-telemetry/semantic-conventions-genai
• Model Context Protocol (MCP) Architecture and Telemetry Standards: https://modelcontextprotocol.io/
• Cloud Native Computing Foundation (CNCF) Observability Overview: https://www.cncf.io/
• Bifrost AI Gateway Documentation and Benchmarks: https://docs.getbifrost.ai/