My LLM app was fully traced. During an incident the trace was still useless.

A regression came in for our German enterprise users on the support agent. Quality had dropped for that one cohort, and I opened the trace store expecting to find the problem in a couple of minutes. We had tracing. I had set it up myself.

What I got was a flat list of 28 spans. None of them carried the prompt version. The model-call spans were named three different things across the same service, because different libraries named them differently. The retrieval spans had the raw user query sitting in them as plain text. And one span had a four-kilobyte blob holding the entire prompt body.

Forty-five minutes in, I still had not found the regression. The app was traced. It was not traced in any way that helped.

That incident is why I rewrote how we trace. Here is what actually makes a trace useful when you are the one staring at it at 2 AM, no code, just the shape of it.

What "a good trace" actually means

Forget the schema for a second. A good trace is one that answers these questions in seconds:

• Which prompt version did this user see?
• Which retrieval was the slow one?
• Which tool call failed?
• Which step's quality score dropped?
• Which cohort is the regression hitting?
• Which model produced the answer?

Those are the exact questions you ask during an incident. Your trace either has the structure to answer them or it does not, and "it shows the model was called but not which version" is the same as no answer at all. Mine was full of those non-answers.

A good trace is a tree, not a flat list

This was my first real mistake. A trace should be a tree that mirrors what actually happened:

• The user request is the root.
• Each meaningful step is a child underneath it: the planner, each retrieval, each model call, each tool call, the guardrail, the evaluator.
• Tool calls nest under the step that triggered them, so you can see cause and effect.

A flat list of spans is not a trace. It is a log file with span ids stapled on, and it buries the one decision point you actually need to find. The moment I switched to a proper tree, "which step went wrong" went from a scavenger hunt to a glance.

If you fix one thing, put the prompt version on every model call

This is the single highest-value change, so do it first.

Every model-call span should carry three tags: which prompt it was, which version, and which A/B variant if you run those. Without them, you literally cannot tell whether a regression came from a prompt rollout, because there is nothing on the trace tying the bad output to a specific version. With them, you filter the trace store by version and the culprit rollout falls out immediately.

My German-cohort regression was a prompt change. It took me 45 minutes precisely because nothing on the trace said which version each user got.

Attach quality scores to the spans, not just latency

Here is the one most people skip. Most traces track latency and errors, which only catch infrastructure problems. They say nothing about whether the answer was any good.

So I now run lightweight quality checks on the output and attach the scores right onto the span: groundedness, faithfulness, whatever matters for that route. Then an alert watches the rolling average of those scores per route and per prompt version. Latency alerts catch the server falling over. Score alerts catch the model quietly getting worse while every latency graph stays green. That second kind is the one that had been slipping past me for a week at a time.

Split cost into reasoning and cache tokens

If you collapse all your token counts into one number, your cost dashboard lies to you.

Reasoning-model tokens and cached tokens behave completely differently, and blending them hides the thing that actually moves your bill. A reasoning-model upgrade can double your cost per query without changing a single visible answer. If those tokens are broken out on the span, you catch it the day it happens. If they are lumped together, you catch it on the invoice. Also compute the cost per call at the moment it happens, so a later price change does not scramble your old numbers.

Redact sensitive data at the collector, not at the client

Traces love to swallow personal data: the raw user question, the full tool arguments, the whole prompt. For anything regulated, that cannot land in your trace store as-is.

The pattern that works: strip it at the collector, the layer the spans pass through on their way to storage, not just in the app. Use a consistent replacement so the same email or name always becomes the same placeholder, which lets you still follow a user through a trace without ever storing who they are. And keep that redaction rule in the same repo as the tracing code, reviewed like any other code.

Keep your span names stable

Span names are how every dashboard and alert groups things. Rename a span from one release to the next and you silently break every chart that was counting on the old name.

So pick a naming convention early, something plain like component-dot-operation, keep it lowercase and stable, and treat a rename as a breaking change that comes with updating the dashboards. Watch out for framework libraries that quietly rename their spans on an upgrade. Pin the versions and check the names when you bump them.

Sample the tail, not the head

Keeping one percent of traces at random to save money sounds reasonable and quietly defeats the entire point. The failures you built tracing to catch are rare, so random sampling throws almost all of them away.

Keep the interesting ones instead:

• Every trace with an error.
• Every trace where a quality score came back low.
• Every trace that was unusually slow or unusually expensive.
• Everything from a canary or experiment cohort.
• A small random slice of the boring, healthy rest.

That way the traces you actually open during an incident are the ones you kept.

What a bad trace looks like, in one glance

If yours has any of these, it will fail you when it matters:

• One giant span with everything crammed inside it.
• Span names that change between versions.
• No prompt version anywhere.
• The raw user input pasted straight into an attribute.
• All token costs mashed into a single number.
• A flat list where an agent run should be a tree.

The nasty part is that a bad trace looks fine at a glance. It only lets you down three weeks later, at 2 AM, when you are the one who has to read it.

The lesson I keep coming back to is that "we have tracing" and "we can actually debug from our traces" are two completely different states, and I had confused them for months. Fixing the tree, the prompt version, and the score-on-span was most of the gap.

If you want the exact attribute names and span shapes to copy into your own setup, this writeup lays them all out.

If you have debugged an LLM incident from a trace, I would love to hear the one attribute you were most glad you had. For me it is the prompt version, and it is not close.