AI workloads generate 10–50× more telemetry data than traditional API calls, and the observability platforms charging by span volume are quietly becoming the most expensive line item in your AI infrastructure. You can cut your token spend in half by switching to cheaper models and caching aggressively, only to watch your monitoring bill triple because you're now tracing every intermediate agent call to debug the cost overruns. That's the hidden span tax, and it's the pattern I keep seeing across teams shipping agentic workloads in 2026.
LLM telemetry is the instrumentation layer that captures request rate, latency, token consumption, cost, cache behavior, and errors across your model traffic, broken down by model, provider, tenant, and status, per DVARA's observability guide. It's ordinary observability applied to a new kind of traffic — plus a few AI-specific signals that generic APM doesn't capture. The problem is that those AI-specific signals multiply fast, and the pricing models weren't designed for that multiplication.
If you're already thinking about the prompt observability hidden cost problem, this post goes one level deeper into the mechanics of why telemetry costs decouple from inference costs and what you can actually do about it.
Why Can't Traditional APM Handle LLM Traffic?
Traditional application performance monitoring assumes software is deterministic — requests succeed or fail, latency is numeric, errors throw exceptions. LLMs violate every one of those assumptions. Traditional APM tools are blind to LLM-specific failure modes like silent semantic failures, context window exhaustion, and runaway token costs because they assume deterministic software. An agent can return HTTP 200 with a hallucinated answer. Error rate stays at 0%. Accuracy drops to 40%. Your APM dashboard shows green across the board.
Here's what that looks like in practice. A RAG pipeline hits a vector database, retrieves context, calls an LLM, post-processes the response, and maybe retries on a timeout. Each of those steps generates telemetry — token counts, embedding vectors, chain-of-thought traces, evaluation metrics, retry logs. AI workloads generate 10–50× more telemetry data than traditional API calls because of this multiplicative structure. A single user request fans out into a dozen internal operations, each needing its own span for debugging.
Infrastructure metrics still work fine — CPU, GPU utilization, network latency to provider endpoints, container memory. The gap is entirely at the semantic layer. You need to know whether the model's answer was correct, not just whether the HTTP call succeeded. You need token-level cost attribution per request, per user, per feature. Traditional tools treat all of that as "custom metrics" and charge you accordingly.
How Does LLM Telemetry Pricing Actually Work?
Most observability platforms charge by volume — logs per GB, custom metrics per host, traces per span. That pricing model was designed for a world where telemetry scaled roughly linearly with traffic. AI workloads break that assumption because a single user request can generate dozens of spans, each carrying token counts, cost estimates, and full prompt/response bodies.
LLM cost monitoring attaches token counts and dollar estimates to every request, aggregating by model and team to enable pre-invoice alerting. But here's the catch: those estimates are allocation numbers based on published token prices, not authoritative financial data. Dashboard cost estimates are allocation estimates based on published token prices; the provider invoice is the only authoritative financial source of truth. Datadog, for instance, estimates LLM request cost from public provider pricing and token counts on spans, but these estimates are not authoritative provider invoices. Teams rely on these dashboards for real-time budget alerts and cost-per-success metrics, treating estimates as actionable financial data — but cached-input pricing and batch discounts mean the invoice usually lands below the dashboard number.
The real budget killer is span volume. Datadog's pricing model charges by LLM span volume, so tracing every intermediate agent call can increase observability spend even when token spend is reduced. Observability costs can increase even as token spend decreases because platforms charge by span volume and agentic workflows generate 10–50× more telemetry than traditional services. This is what I call the span-token divergence: the two cost axes pull in opposite directions, and most teams only optimize one of them.
| Tool | Pricing Model | Starting Price | Best For |
|---|---|---|---|
| Langfuse | Billable units (traces + observations + scores) at $8/100k | $29/mo Core | Open-source default with self-host option |
| LangSmith | Per-event managed cloud | ~$2,514/mo at 1M events | Teams already in the LangChain ecosystem |
| Datadog LLM Obs | Per LLM span volume | — | Teams with existing Datadog infrastructure |
| Arize Phoenix | Open-source, OTel-native | Free (OSS) | RAG evaluation and OTel-first monitoring |
What Is the Span-Token Divergence Pattern?
Here's the core tension. Output tokens cost 2–5× more than input tokens across every provider, creating asymmetric cost profiles that dominate enterprise cost modeling. LLM API pricing in 2026 ranges from $0.10 per million input tokens for budget models to $30 per million input tokens for frontier reasoning models. Teams obsess over that 300× spread between the cheapest and most expensive models, and they should — it's a real lever.
But here's what happens when you optimize only the token axis. You switch from a frontier model to a cheaper one. You add caching. You trim system prompts. Token spend drops 40%. Meanwhile, your agent now makes more internal calls to compensate for the cheaper model's lower per-call quality — more retries, more tool calls, more retrieval steps. Each of those generates a span. Your observability platform bills you for every span.
A multi-step agent trace can contain a dozen observations, meaning a single trace may generate 10+ billable units when scores are included. The lever you control is observations per trace. A chatty agent with deep tool trees multiplies your unit count. A simple single-call chatbot barely moves it. This is why LLM serving stack costs are only half the picture — the monitoring layer on top has its own cost dynamics that scale with architectural complexity, not just token volume.
Should You Self-Host or Use Managed LLM Observability?
Langfuse is the most widely deployed open-source LLM observability platform — MIT-licensed, 26M+ SDK installs/month, acquired by ClickHouse in January 2026. Self-hosting eliminates per-unit licensing. The tradeoff is operational: self-hosted open-source tools require Postgres, ClickHouse, Redis, and S3, and engineering hours for collector upgrades and config drift dominate total cost of ownership within 12 months.
The "free self-host" pitch is real but misleading. Langfuse's MIT build gives you tracing, evals, prompt management, human annotation, and RBAC at no licensing cost. Opik offers Apache-2.0 with full feature parity. The marketing emphasizes $0 licensing — and that's accurate. But the operational burden is invisible in vendor comparisons. You're running four services in production, managing schema migrations, handling ClickHouse capacity planning, and maintaining an on-call rotation for your observability stack. That's engineering time that isn't going to your product.
For most teams, the managed tier is cheaper than self-hosting once you factor in engineering TCO. The break-even point depends on your traffic volume, retention requirements, and whether you already have ClickHouse and Postgres expertise on the team. If you do, self-hosting wins on unit economics at any serious volume. If you don't, the $29/month Core tier sidesteps the operational tax entirely.
How Much Does LLM Telemetry Actually Cost at Scale?
Let's walk through the math. Langfuse bills billable units — traces plus observations plus scores — at $8 per 100k units, with the Core tier at $29/month including 100k units. A multi-step agent trace can contain a dozen observations, so a single trace may generate 10+ billable units when scores are included.
Here's the projection for a 50-developer team processing 500,000 LLM requests per month, averaging 10 billable units per request:
- 500,000 requests × 10 billable units per request = 5,000,000 total units
- Minus 100,000 included units = 4,900,000 overage units
- 4,900,000 ÷ 100,000 × $8 = $392 overage
- $29 base + $392 = $421 total
That aligns with Langfuse's published $400–$430 range for this volume. For comparison, at 1 million events per month, Langfuse Cloud costs approximately $101/month versus LangSmith at roughly $2,514/month — a ~25× gap. The difference is stark enough that it changes the build-vs-buy calculus entirely.
Is OpenTelemetry the Convergence Point for LLM Telemetry?
Major observability platforms — Langfuse, LangSmith, and Arize Phoenix — all added OTLP endpoints in early 2026, converging on OpenTelemetry as the shared protocol layer. This matters more than it sounds. If your telemetry is OTLP-native, you can switch backends without re-instrumenting your codebase. That's the only real hedge against vendor lock-in as this market consolidates.
But there's a contradiction. OpenTelemetry GenAI semantic conventions — the gen_ai.* attributes that standardize spans, metrics, events, and agent-specific data across vendors — all remain in Development (experimental) status as of May 2026. The protocol is converging, but the vocabulary is still in flux. You can standardize on OTLP transport today and still need to update your attribute schemas when the conventions stabilize. That's not a reason to wait — it's a reason to instrument now and treat the attribute layer as a migration you'll do once, not a re-architecture.
The EU AI Act adds regulatory pressure to this equation. The European Commission now has powers to fine AI model providers up to €15 million or 3% of annual turnover, making audit-ready telemetry a compliance requirement, not just a debugging convenience. Teams that standardize on OTel now will have a far easier time generating compliance evidence when enforcement lands.
How Do You Prevent Observability Bill Shock?
Agentic applications may need span sampling or trace policies because agent tools can create many spans per user task, risking observability bill shock. The data suggests this is already a production reality for teams running multi-step agents at scale. Here's a practical framework:
- Set span-volume caps. Treat observability as a FinOps line item with a monthly budget, not a debugging afterthought. Cap the number of spans you emit per user request and sample aggressively for non-error traces. 2. Instrument on OTLP from day one. Even if you start with a managed platform, emit OpenTelemetry-native traces. When the market consolidates further — and it will — you can redirect telemetry to a different backend without re-instrumenting. 3. Separate cost dashboards from financial truth. Use dashboard estimates for alerting and trend detection, but reconcile to the provider invoice monthly. The LLM evaluation pipelines maturity gap is real — cost monitoring without eval discipline means you're optimizing spend without measuring quality. 4. Calculate observations-per-trace before choosing a platform. A chatty agent with 15 tool calls per request will generate 15+ billable units per trace. A simple chatbot generates 1–2. Your platform choice should follow your architecture, not the other way around. 5. Budget for the span tax explicitly. If you're running agentic workflows, assume your observability cost scales with span count, not token count. Model it in your infrastructure budget the same way you model GPU spend.
The teams that win here are the ones who treat LLM telemetry as infrastructure — with budgets, caps, and a migration path — not as a debugging tool they'll "figure out later." The question isn't whether you need observability. It's whether your observability bill will stay proportional to the value it delivers, or quietly become the most expensive part of your AI stack.
Originally published at SaaS with Alex
Top comments (0)