Observability in the age of LLMs: tracing what your AI actually did

A feature built on an LLM can hit every familiar SLO and still be failing in the way that matters. The API is up. The median latency looks fine. Error rate is flat. Meanwhile the model is taking the scenic route through three tool calls, burning tokens, and returning answers you would not want a customer to trust. The thing you need to see is not just whether the service is healthy. It is what the system actually did.

That is why LLM observability is best thought of as an extension of ordinary tracing, not a fresh religion with its own priesthood. An LLM call is a span. An agent run is a span tree. The useful questions are still familiar: where did the time go, which step failed, what changed, and how much did this request cost? What changes is the shape of the work. Now you also care about token usage, model choices, tool execution, and whether the answer was any good.

If you already run OpenTelemetry, that is good news. You do not need to throw away your tracing model to observe AI features properly. You need to extend it.

Why classic observability stops short

Traditional service telemetry assumes that the interesting failures are mostly about infrastructure or request handling: timeouts, retries, saturation, bad downstreams, deployment regressions. Those problems still exist in LLM-backed systems, but they are no longer the whole story.

A generated answer can be wrong while every system metric stays green. A retrieval step can bring back unhelpful context without producing an obvious exception. An agent can choose a slow tool path, or call the same tool twice, or hand the model a bloated prompt that quietly doubles token spend. Keep the old graphs; just stop expecting them to catch this class of failure.

OpenTelemetry made the same point in its 2026 GenAI observability post: every application call to an LLM hides "a chain of model calls, tool invocations, and token exchanges" behind the scenes. That is the part standard web-service monitoring does not naturally reveal. When a request takes 45 seconds, the question is no longer just "which service was slow?" but "which decision inside the trace made it slow?"

This is the same reason fast feedback loops matter more when LLMs write your code. When generation gets cheaper, validation becomes the scarce resource. In production, observability is one of those validation loops.

The debugging primitive is still the trace

There is a mild industry temptation to treat LLM systems as too exotic for the tooling we already have. That is usually the wrong instinct.

A useful mental model looks more like this:

agent.run
├── llm.chat openai/gpt-5.6
├── tool.retrieval postgres/pgvector
├── llm.chat openai/gpt-5.6
├── tool.http billing-api
└── llm.chat openai/gpt-5.6

That tree is already enough to answer a lot of awkward production questions:

• which step dominated latency
• whether the model asked for tools more than once
• whether retrieval happened before the answer changed course
• whether one provider or model variant behaved differently
• where token usage and therefore spend started to climb

None of that needs a brand-new theory of observability. It just means treating model calls and tool calls as first-class trace events instead of leaving them hidden inside application logs or framework internals.

That framing matters because it keeps you portable. If the trace is the source of truth, you can ship it to a general-purpose backend, to an AI-focused product, or to both. If the observability story starts and ends inside one vendor's SDK, you have made your debugging model harder to move.

What the OpenTelemetry GenAI conventions give you

The concrete standard here is the OpenTelemetry semantic conventions for Generative AI. As of July 2026 the gen_ai work has moved into its own dedicated repository, semantic-conventions-genai, and every document in it is still marked Development rather than stable. The new repository does not even have a tagged release yet, so there is no version number to pin against; the honest options are pinning a commit, or recording the last main-repo release that carried the gen_ai conventions before the move. That does not make them useless. It means you should adopt them with your eyes open and expect some movement.

You do not need to memorise every field. The useful thing is the shape of the telemetry the conventions are trying to standardise.

For spans and attributes, the useful core includes fields such as:

• gen_ai.operation.name for the kind of model operation
• gen_ai.provider.name for the provider
• gen_ai.request.model and gen_ai.response.model
• gen_ai.usage.input_tokens and gen_ai.usage.output_tokens
• gen_ai.response.finish_reasons
• gen_ai.tool.name for tool execution
• gen_ai.agent.name and gen_ai.conversation.id when you need to group larger workflows

For metrics, the current GenAI docs define client-side measurements such as gen_ai.client.token.usage and gen_ai.client.operation.duration.

That is enough to capture the parts of an LLM request you usually end up caring about first: how many tokens went in, how many came out, how long the operation took, which model handled it, and where tool execution sat in the middle.

It is also worth being precise about one thing the conventions do not settle for you: cost. Cost is still best treated as a derived metric rather than a magical standard field. In practice that means joining token counts to your model pricing table and calculating spend from there. If someone promises a universal LLM cost number with no context, raise an eyebrow.

Instrument these first

You can spend a lot of time arguing about evals, guardrails, and which agent framework is fashionable this month. Instrumentation is less glamorous, but it is the bit that keeps the rest honest.

If you are tracing LLM-backed features today, start with these:

• One span per model call. Capture provider, model, operation, latency, finish reason, and token counts.
• One span per tool call. Retrieval, HTTP calls, database lookups, and internal helpers should sit in the same trace, not in a separate logging universe.
• Prompt and response references, not reckless raw dumps. Keep enough data to debug, but be deliberate about privacy, redaction, and storage costs.
• A derived cost metric. Token counts are useful; token counts joined to pricing are useful to finance and engineering at the same time.
• Quality signals beside the trace. User rating, evaluator score, moderation outcome, or task success should sit close enough to the trace that you can compare bad answers with the path that produced them.

Getting the first span is cheaper than most teams expect. If your stack is Python calling OpenAI, the contrib instrumentation does it with zero code changes:

pip install opentelemetry-distro opentelemetry-instrumentation-openai-v2
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true opentelemetry-instrument python app.py

Note the environment variable. Prompt and completion content is off by default and opt-in, which is exactly the right default: you get spans, models, token counts, and finish reasons without capturing user text, and you turn on content capture deliberately, with redaction and retention already thought through.

That is the practical core. You can add fancier workflow views later.

Experimental is fine. Invisible is not.

Some teams hesitate because the GenAI conventions are still moving. That is a real caveat, but it is not a good reason to keep model behaviour opaque.

The honest trade-off is this:

• if you wait for the conventions to feel perfectly settled, you stay blind longer
• if you adopt them now, you may need a migration later

I would take the migration.

A moving standard is still much better than bespoke telemetry that only one SDK understands. The conventions already give you a common vocabulary for spans, token usage, and model metadata. Even if a field changes later, the shape of the problem is now visible in your traces instead of trapped inside ad hoc logs.

The practical discipline is boring, which is usually how you know it is right:

• record the commit or snapshot of the conventions you instrumented against, since there is no tagged release to pin yet
• note that it is a Development-status schema
• keep your instrumentation behind a thin abstraction where you can
• re-check the docs before each major upgrade

That is manageable. Debugging a production agent system with no trace of its internal choices is much less manageable.

The AI observability tools are mostly OpenTelemetry underneath

Once you start looking around, the AI observability market can seem louder than the underlying technical differences justify.

There are good tools here. Langfuse's own docs position its OpenTelemetry integration around ingesting OTLP traces for LLM observability. MLflow describes its tracing product as OpenTelemetry-compatible and aimed at LLM and agent observability. OpenInference, from the Arize ecosystem, describes itself more plainly still: OpenTelemetry instrumentation for AI observability.

That is the important pattern.

The credible tools are increasingly building on OpenTelemetry rather than asking you to abandon it. What they compete on is everything around the wire format:

• workflow visualisation
• evaluator and feedback loops
• prompt inspection
• dataset and experiment tooling
• agent-specific views
• hosted versus self-managed trade-offs

Those are meaningful differences. They just are not a reason to skip portable traces.

If you already have an observability stack you trust, start there. Send the traces somewhere you can query them. Add a specialised tool when you have a concrete gap: better prompt inspection, easier annotation workflows, richer experiment tracking, or an agent timeline that your current backend does not give you.

What I would not do is buy an AI observability platform before I had first proved that we were actually capturing the right spans.

Agent observability is where this gets interesting

Plain LLM tracing is useful. Agent tracing is where the debugging payoff gets larger.

The difficult production questions are rarely about one completion in isolation. They are about execution paths:

• why did this agent choose that tool
• why did it call the same tool twice
• why did it stop early
• why did one branch of the workflow take ten times longer than the others
• which step turned a reasonable user request into an expensive one

That is why the recent push toward agent timelines is worth paying attention to. Honeycomb launched its Agent Observability features on 12 May 2026, with an Agent Timeline view that is basically a more opinionated way to walk the same trace tree. That kind of view earns its keep because agent systems are not just slow or fast. They are branching, recursive, and occasionally a bit absurd.

The frontier here is less about inventing new telemetry categories than about making traces easier for humans to read. Multi-agent systems create deeper trees, more spans, and more "why on earth did it do that?" moments. The backend that wins will be the one that makes that path legible under pressure.

What to do this week

If you are shipping LLM-backed features and do not yet have a plan, the sensible order of work is refreshingly unromantic:

1. instrument model calls as spans
2. instrument tool calls in the same trace
3. capture token counts and duration
4. derive cost from usage and pricing
5. attach one quality signal you trust
6. only then decide whether you need a specialised platform

That sequence gets you out of the worst failure mode, which is running a system that can think, call tools, and spend money without leaving behind an explanation.

LLM observability is not separate from good observability practice. It is what happens when the service starts making choices on your behalf. Start with traces. Make token usage and cost visible. Keep the data portable. Then, if you need a shinier lens on top, add one later.