Why Your LLM Observability Stack Is Blind to the Thing That Matters

App-level logging tells you what one service did. It never tells you what your whole AI stack is doing.

Picture this: your model provider bill jumps sharply in a week and nobody can say why. Every app has its own logger, its own dashboard, its own slice of the truth. None of them show you the full picture, because none of them was built to.

That's the real failure mode in LLM observability right now. It's not a missing tool. It's a structural blind spot. You're trying to understand a multi-model, multi-team system by looking at it one application at a time.

Why per-app logging stops working

Wrapping your OpenAI or Anthropic client and shipping logs to your APM tool is the natural first move. It works fine for one app talking to one model. It falls apart once you have more than that, for three concrete reasons.

You only see your own app. Comparing token spend or error rates across App A, App B, and App C means stitching together several different logging setups, assuming they even measure the same thing the same way.

You miss what happens after your app sends the request. Client-side logs show the prompt you sent and the response you got. They don't show which model actually served the request if you're using fallback routing, or what the provider returned before any response filtering ran.

It doesn't scale with your surface area. Every new app needs its own instrumentation. Every model swap means updating that instrumentation everywhere it lives. It's the same tech debt problem API gateways solved for REST traffic years ago, most teams just haven't applied the lesson to LLM traffic yet.

What actually makes LLMs hard to observe

Traditional APM assumes deterministic behavior. Same input, same output, same latency profile. LLMs break that assumption on every axis. The same prompt can return a different token count, get routed to a different model depending on load, and cost a different amount depending on which provider handled it. Bolting LLM calls onto a generic APM setup built for deterministic services gets you incomplete data by design.

The six numbers worth tracking

If you're instrumenting LLM traffic, track these at the aggregate level, not per app.

• Request latency at p50, p95, p99. Averages hide tail latency, and tail latency is what your angriest enterprise customer is experiencing right now.
• Token usage per request, input and output counted separately. This is your cost driver and your earliest signal that a prompt or an agent loop has gone sideways.
• Cost per call, attributed by team or app. Without this you can't answer "who's burning the budget" with anything better than a guess.
• Error rate, tracked per provider so you can tell a provider outage from a broken prompt template.
• Fallback rate. If a meaningful share of traffic is hitting your fallback chain, your primary model or provider has a reliability problem you haven't noticed yet.
• Anomaly alerts on volume, cost, or failure spikes. This is also where you catch prompt injection attempts or a misconfigured agent stuck in a retry loop, ideally before it shows up on the invoice.

A single app's error rate tells you almost nothing on its own. The same number aggregated across your whole stack tells you whether the problem is the provider, the prompt, or your own configuration.

Percentiles, not averages

A request that takes 200ms most of the time and several seconds occasionally has a perfectly respectable average and a genuinely bad user experience. p50 tells you the typical case, p95 the near-worst case, p99 the actual worst case. Track all three, broken down by model and by team, because "which model is slow" and "who is sending the queries that are slow" are two different questions with two different fixes.

Where OpenTelemetry fits

You don't want a separate observability stack just for AI. OpenTelemetry already gives you a shared format for traces, spans, and metrics across your infrastructure, and it now has semantic conventions specifically for generative AI: standard attribute names for provider, model, and token counts, so any OTel-compatible backend can ingest LLM traces without a custom pipeline. Note that these GenAI conventions are still marked as under active development, so expect some attribute names to shift as the spec stabilizes.

A GenAI span built on this convention looks roughly like this:

{
  "name": "chat gpt-4o",
  "attributes": {
    "gen_ai.operation.name": "chat",
    "gen_ai.provider.name": "openai",
    "gen_ai.request.model": "gpt-4o",
    "gen_ai.usage.input_tokens": 812,
    "gen_ai.usage.output_tokens": 194,
    "gen_ai.response.finish_reasons": ["stop"]
  }
}

If your instrumentation emits spans in this shape, LLM traces sit right next to your regular service traces, in the same backend, with no separate toolchain for AI.

Where the gateway layer comes in

The cleanest way to get all of this without touching every app's codebase is to put an AI gateway between your applications and your model providers. Every request passes through one layer, so that layer can capture latency, token counts, routing decisions, and cost per call automatically. No per-app instrumentation required. It's also the natural place to run anomaly detection, since it's the only component that sees traffic from every app and every team at once.

TrustGate is NeuralTrust's open-source take on this pattern, sitting in the request path and logging the full trace for every call. There's a deeper breakdown of the metrics and the architecture behind this approach in the original writeup on gateway-level LLM observability, worth a read if you want more detail than fits here.

If you're also thinking about the security side of this, not just cost and latency but what happens when an agent's tool access gets abused or a prompt injection campaign shows up in your traffic, Agent Security has a decent library of threat patterns and mitigations worth checking out.

The takeaway

App-level logging was never going to give you system-level answers. If you want to know what your AI stack is actually doing, across every model, every app, and every team, the observation point has to move upstream of all of them. That's an infrastructure decision, not a logging library decision.