LLM Observability: Tracing, Logging, Debugging Agent Runs

Why your LLM app will fail silently, and how to see it before your customers do.

Three weeks. That is how long a customer-support agent shipped confidently wrong answers for a logistics client I work with before anyone noticed. The agent's job was simple: look up a shipment's status and reply to the customer. It did this hundreds of times a day. Every request returned HTTP 200. Latency was fine. The API bill looked normal. And the agent was quietly hallucinating tracking numbers.

The first clue came from a phone call. A customer in Jeddah had been told her package was "delivered" — it was not. We pulled the logs. There were logs. They said: request received, model called, response returned, 200 OK. Nothing else. No record of what the retrieval step actually returned, no record of what the prompt looked like that day, no record of which model version answered, no record of how many tokens it burned to be wrong.

That is the moment I stopped thinking about LLM observability as a nice-to-have and started treating it as the difference between a working system and an expensive black box. In this article I am going to walk you through everything I now do — and you should too — to trace, log, and debug LLM and agent runs in production.

Why LLM Observability Is Not Regular Observability

Traditional observability assumes your code is deterministic. You log an error, you see the stack trace, you find the bug, you fix it. An LLM breaks that assumption in four specific ways, and each one changes what "observable" means:

1. The output is nondeterministic. The same prompt can produce a correct answer at 9:00 AM and a hallucination at 9:15 AM with no code change in between. The system did not crash; it drifted. Your logs have to capture state, not just errors.
2. The hidden state is the whole story. For a normal API you care about the request and response. For an LLM you also care about what was inside the prompt — the retrieved chunks, the tool outputs, the system instructions, the model version. That is where wrongness hides.
3. Cost is per-token, and it compounds. A retry loop, a bloated context, or a greedy agent can quietly multiply your bill 10x without throwing a single exception. You need token counts, not just request counts.
4. The failure is often fluent. LLMs fail with perfect grammar and total confidence. Nobody files a bug report for a plausible wrong answer. Your logs are the only witness.

This is why "we log everything with our regular logging library" is not enough. You need tracing that reconstructs the chain of decisions, not a flat list of calls.

The Three Pillars: Tracing, Logging, and Evals

When I design observability for an LLM system now, I build three layers, and they answer three different questions:

Tracing answers "what happened, in what order, and how long did each step take?" A trace is a tree of spans. The root span is the request; child spans are retrieval, prompt assembly, each LLM call, each tool call. Every span carries duration, token counts, cost, model name, and the input/output that passed through it. This is what reconstructs the Jeddah incident — you can see the retrieval returned an empty chunk, the prompt went out truncated, and the model answered anyway.

Logging answers "what does this record look like for later analysis?" I log the full prompt and completion for every run, the retrieved chunks with their scores and sources, tool arguments and results, and a stable run ID. This is the raw material for audits, for compliance, for reproducing a specific failure after the fact.

Evals and metrics answer "is this getting better or worse?" Metrics are counters and gauges: tokens per request, latency percentiles, cost per resolved task, tool-call rate, cache hit rate. Evals are the scored test cases you run against a regression set when you change the prompt or the model. Tracing tells you what broke; evals tell you whether your fix stuck.

The eval loop is where most teams quietly drop the ball, so let me be concrete about what it looks like. I keep a regression set of 50–100 real, de-identified interactions — a few per failure mode the team has hit. When I change a prompt, a retrieval strategy, or a model version, I run the set and score it with a mix of exact checks (the answer contains the correct tracking number) and rubric-based checks (did the agent escalate when the tool returned an error). The output is a pass rate and a diff against the previous run. If the change improves the reported incidents but drops the eval score by three points, I do not ship it. That eval set is the only reason I can move fast on prompts without being scared — the trace tells me what changed, and the eval tells me whether it is okay.

The Anatomy of a Trace

Let me show you what a real agent trace looks like. Take a question like "What is the status of order SH-991?" The root span splits into children, and each child carries its own numbers:

request (root span)                          duration 3.1s   cost $0.084
├── auth + routing                           0.4ms           -
├── retrieval (vector store)                 28ms            top_k=5
│     └── 1 chunk returned (score 0.71)                      731 tokens
├── prompt assembly                          2ms             prompt=2,104 tokens
├── llm call 1                               1.9s            input 2,835 / output 142
│     └── decision: call_tool(lookup_tracking)
├── tool: lookup_tracking("SH-991")          120ms           400 error: not found
├── llm call 2                               860ms           input 3,112 / output 89
└── final answer                                              3,412 tokens total

That one trace told me exactly what went wrong in the Jeddah case: retrieval returned a single low-confidence chunk, the tracking tool returned a hard error, and the model answered anyway instead of asking for a corrected reference number. Three spans, three failure points, one trace. Without it I would have spent days guessing.

The ecosystem has settled on this shape. OpenTelemetry defined semantic conventions for generative AI (the gen_ai.* attribute family — gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens) so traces from different providers and frameworks can be correlated in one tool. On top of the OTLP transport, you get purpose-built backends — I have used LangSmith and Langfuse, and both will happily ingest OTLP traces now. You do not need to pick between a tracing backend and an APM; they speak the same wire format.

What I Actually Capture Per Run

Here is the concrete capture contract I use. If a run produces these fields, I can debug any incident in under an hour:

• run_id — stable across retries, correlatable to the customer session
• model — provider, model name, and version
• temperature / top_p / max_tokens — the sampling config at the time
• system prompt — the full text, version-hashed
• retrieved chunks — text, source, and score for each
• tool calls — function name, serialized arguments, result, error if any
• token counts — prompt, completion, and total per call
• latency — per span and total
• cost — cents per call, from the token counts × current price
• feedback — thumbs up/down or a follow-up human rating when available

I write this to a Postgres table with a JSONB column for the trace plus a few indexed columns for querying, and I ship the same data to a tracing backend for the visual waterfall view. The table is the audit trail; the backend is the debugger.

A Minimal Instrumented Loop

Here is a minimal agent loop with tracing bolted on — no framework, just the OpenTelemetry SDK and a couple of spans. This is the smallest thing I would actually deploy:

import json
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider

trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer("agent.observability")

def run_agent(goal: str, retriever, llm, tools, max_steps: int = 5):
    with tracer.start_as_current_span("agent.run") as root:
        root.set_attribute("goal", goal)
        context = []
        for step in range(max_steps):
            with tracer.start_as_current_span(f"step.{step}") as step_span:
                with tracer.start_as_current_span("retrieve") as ret_span:
                    chunks = retriever(goal)
                    ret_span.set_attribute("chunks.count", len(chunks))
                    ret_span.set_attribute(
                        "chunks.scores", json.dumps([c.score for c in chunks]))
                with tracer.start_as_current_span("llm.call") as llm_span:
                    llm_span.set_attribute("gen_ai.request.model", "your-model")
                    decision = llm.decide(context, chunks, tools)
                    llm_span.set_attribute(
                        "gen_ai.usage.input_tokens", decision.input_tokens)
                    llm_span.set_attribute(
                        "gen_ai.usage.output_tokens", decision.output_tokens)
                if decision.is_final:
                    return decision.answer
                with tracer.start_as_current_span("tool.call") as tool_span:
                    tool_span.set_attribute("tool.name", decision.tool_name)
                    tool_span.set_attribute(
                        "tool.arguments", json.dumps(decision.arguments))
                    result = tools<a href="**decision.arguments">decision.tool_name</a>
                    tool_span.set_attribute("tool.result", json.dumps(result))
                context.append((decision, result))
        raise RuntimeError("step budget exhausted")

The key habit: every span carries the input and output that moved through it. A span that only records duration is a pretty waterfall with no forensic value. When the trace shows chunks.scores = [0.71] and the tool result is a hard error, the debugging is over before it starts.

What I Actually Learned Debugging Real Runs

Once tracing was live, the failure modes stopped being mysterious and became a checklist. These are the ones I hit, in order of frequency:

1. Retrieval returned nothing, and the model guessed. The most common silent killer. The fix is never in the prompt; it is in the retrieval quality — better chunking, higher top-k, a fallback query. The trace proves it in one line.
2. Context bloat. A run that should use 4,000 tokens was sending 28,000 because history was never summarized. Latency doubled, cost quadrupled, and quality went down. Token counts per span expose this instantly.
3. Tool arguments out of schema. The model invented a field or formatted a date wrong. With arguments logged, you can add a validation layer or tighten the tool description in one iteration.
4. Model drift after a provider update. A silent provider-side change degraded one customer segment. Because I had model version on every span, I could see the version flip exactly when accuracy dropped.
5. Cost spikes tied to loops. One agent was retrying the same failing tool call five times per conversation, doubling the bill. Step budgets are not a guardrail question; they are an observability question — the trace showed the loop.

I keep a running rule now: if a production incident takes more than an hour to explain, the tracing is insufficient. Not the model, not the prompt — the tracing.

When You Do NOT Need Full Observability

Honest section, because not everything needs a tracing stack. If your use case is a stateless, single-shot LLM call — a summarizer, a classifier, a translation step with no tools and no retrieval — then a plain log of prompt, completion, model, and token count is 90% of the value at 10% of the setup cost. Ship that first.

Similarly, if you are prototyping and have fewer than a few hundred calls a day, the tracing backend is overkill. Log to a JSON file. Add the full stack when you hit real users, real money, or a multi-step agent loop. The rule of thumb: one LLM call, no tools, no state → simple logging. A loop, tools, retrieval, or autonomy → tracing, non-negotiable.

There is also a spectrum between the two extremes, and most teams are on it. If you are at the "loop with tools" stage but not yet at "needs alerting," start with the logging contract and the eval set, and defer the backend until the trace viewer is actually going to save you time. The mistake I see is the reverse: teams buy the expensive backend first and never build the logging contract, so they have a beautiful waterfall view of runs that do not contain the one field that would explain the incident. The data comes first. The tool is the last mile.

Alerting: Making the Trace Do Work

The final layer is alerting, and it is the one that turns observability from a postmortem tool into a prevention tool. I set alerts on the metrics that predict incidents before customers feel them:

• Cost per resolved task over a rolling day — when it spikes 30% above baseline, a loop or a context bloat is likely eating money.
• Tool-call rate per run — when an agent that should call a tool in 80% of runs suddenly calls it in 30%, it has started guessing from training data. That is the "silent degradation" failure mode, caught in hours instead of weeks.
• Error rate on tool executions — a spike means a downstream API broke, and the agent is about to hallucinate around it.
• Average retrieval score — a creeping drop means the chunking or the embeddings drifted, and every answer will get worse.

Alert thresholds have to be tuned to each system, but the principle is universal: alert on drift from the system's own baseline, not on absolute numbers. A 500-millisecond latency spike means nothing for a batch summarizer and everything for a chat product. Your historical traces are the baseline; the alert just detects the divergence.

The Practitioner's Checklist

Before you call an LLM system production-ready, go through this list:

• [ ] Every run has a stable run_id correlated to the user session
• [ ] Prompt and completion are logged verbatim, with model name and version
• [ ] Retrieved chunks carry source and score
• [ ] Tool calls log arguments, result, and error
• [ ] Token counts (input/output) and cost are recorded per call
• [ ] Latency is measured per span, not just end-to-end
• [ ] A trace viewer can reconstruct any run in under five minutes
• [ ] A regression eval set exists, and it runs on every prompt change
• [ ] Alerts fire on cost-per-task, tool-call rate, and error-rate thresholds
• [ ] You can answer "what changed" for any incident — model, prompt, or data

The Aftermath

After the Jeddah incident we rebuilt the agent's tracing from scratch, added a guard that asks the customer to re-confirm the reference number when retrieval returns nothing, and put the whole thing behind the observability stack I described. Two months later, the client asked what the new dashboards cost. When I told him, he laughed and said the alternative — three more weeks of confident wrong answers — would have cost his dispatch team more than that in a single day.

Your LLM will fail silently. It is not a question of whether; it is a question of how long you do not know about it. Tracing, logging, and evals are the difference between finding out in hours and finding out from an angry customer.

If you are starting today, do not buy a tool. Add the run ID, log the prompt and completion, and put token counts on every call. That is the whole foundation. Everything else — the backends, the dashboards, the alerting — is polish on top of a habit you have to build first.

---

*Gulshan Yad