- Book: LLM Observability Pocket Guide: Picking the Right Tracing & Evals Tools for Your Team
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
You add tracing to your LLM service in an afternoon. You wrap the model call in a span, stuff the full prompt and the full response into attributes, ship it, and move on. Three weeks later two things have happened. Your tracing bill has a comma in it that was not there before, and a customer's social security number is sitting in plaintext in a span your whole company can read.
Both are span-design problems. A span is not a log line you can dump everything into. It is a fixed-shape record you query, alert on, and pay per byte to store. What you put in it decides whether your traces answer questions in six months or just sit there as expensive sludge.
This is the attribute set worth capturing, the payloads worth dropping, and the size and sampling rules that keep the whole thing affordable.
Start from the questions, not the data
The trap is capturing everything because you can. Storage is cheap until it is not, and a span crammed with fields nobody queries is pure cost. Work backward from the questions you will ask at 2am instead.
You will ask: which model served this, how many tokens, how long, how much did it cost, which prompt version, did it error, and which user session does it belong to. Every one of those is a single scalar. None of them is the prompt text.
So the spine of a good LLM span is a handful of cheap, high-cardinality-aware scalars. The full prompt and response are a separate decision, made later, under different rules.
The attributes that pay rent
These map onto the OpenTelemetry GenAI semantic conventions. The names have churned over the last year. gen_ai.system became gen_ai.provider.name, and the token usage fields moved from prompt_tokens/completion_tokens to input_tokens/output_tokens. So check the GenAI semconv registry rather than copying an old blog post.
gen_ai.request.model "gpt-4o-2024-11-20"
gen_ai.provider.name "openai", "anthropic", "aws.bedrock"
gen_ai.usage.input_tokens int
gen_ai.usage.output_tokens int
gen_ai.request.temperature float
gen_ai.response.finish_reasons array
gen_ai.conversation.id stable id across turns
Then the fields the spec does not cover yet. They go in a project-prefixed namespace so nobody confuses them with the standard:
app.llm.cost_usd float, computed at emit time
app.llm.prompt_version "support-v7", or a git sha
app.llm.judge.score float 0-1, if you run an eval step
app.rag.retrieved_count int
app.rag.top_score float
Two of these earn their place loudly. app.llm.cost_usd, computed from your own price table at emit time, because the provider's billing endpoint will never settle fast enough to drive an alert. And app.llm.prompt_version, because when judge scores drop you will want to slice by the exact prompt that shipped, and a free-text prompt blob will not let you do that. A version string will.
A minimal emitter
This assumes the OTel SDK is already initialized.
from opentelemetry import trace
tracer = trace.get_tracer("app.llm")
# USD per 1K tokens (in, out). Verify against the
# provider's pricing page; these rotate.
COSTS = {
"gpt-4o-2024-11-20": (0.0025, 0.0100),
"claude-sonnet-4-5": (0.003, 0.015),
}
def usd(model, in_tok, out_tok):
cin, cout = COSTS.get(model, (0.0, 0.0))
return (in_tok / 1000) * cin + (out_tok / 1000) * cout
def emit_llm_span(model, provider, usage,
conv_id, prompt_version):
with tracer.start_as_current_span("gen_ai.chat") as s:
s.set_attribute("gen_ai.request.model", model)
s.set_attribute("gen_ai.provider.name", provider)
s.set_attribute(
"gen_ai.usage.input_tokens", usage["in"]
)
s.set_attribute(
"gen_ai.usage.output_tokens", usage["out"]
)
s.set_attribute("gen_ai.conversation.id", conv_id)
s.set_attribute(
"app.llm.prompt_version", prompt_version
)
s.set_attribute(
"app.llm.cost_usd",
usd(model, usage["in"], usage["out"]),
)
Notice what is not here: the prompt text and the model output. Those follow a different set of rules.
What to redact before anything leaves the process
If you capture prompt and response content at all, assume both contain whatever your users typed, which means email addresses, names, card numbers, occasionally a password someone pasted by accident. That data lands in a tracing backend a lot of people in your company can read, often a third-party SaaS. Treat the span content like any other PII sink.
The rule that holds up: redact at emit time, in your own process, before the data hits the exporter. Do not rely on the backend to scrub it — by then it has already crossed a network boundary you do not control.
A scrubber that runs on the content before it becomes an attribute:
import re
PATTERNS = {
"email": re.compile(r"[\w.+-]+@[\w-]+\.[\w.-]+"),
"card": re.compile(r"\b(?:\d[ -]*?){13,16}\b"),
"ssn": re.compile(r"\b\d{3}-\d{2}-\d{4}\b"),
}
def redact(text):
for label, pat in PATTERNS.items():
text = pat.sub(f"[{label}]", text)
return text
Run this on the content, then truncate, then set the attribute. Regex scrubbing is a floor, not a ceiling (names and addresses slip past it), so for regulated data the safer default is to not capture raw content at all and instead store a hash or a structured summary. The OpenTelemetry GenAI conventions reflect this: capturing message content is opt-in for exactly this reason, and you turn it on deliberately, per environment.
Payload-size limits
Spans are not built for large blobs. A long agent conversation can carry tens of kilobytes of prompt history per turn, and OTel exporters have default attribute-value length limits that will silently truncate you anyway — often at 128 characters unless you raise the span limits. Discovering that truncation after an incident, when the field you needed got cut, is a bad time.
So decide truncation on purpose. Cap content attributes at a length you choose, and record that you did:
MAX_CHARS = 4096
def capped(text, span, key):
if len(text) > MAX_CHARS:
span.set_attribute(f"{key}.truncated", True)
span.set_attribute(f"{key}.original_len", len(text))
text = text[:MAX_CHARS]
span.set_attribute(key, text)
For anything genuinely large — full RAG context, multi-turn transcripts — keep it out of the span entirely. Write it to object storage keyed by trace id and store only the pointer in the span. Your traces stay small and queryable, while the bulky payload lives somewhere built for bulk and priced for it. The span answers what happened. The blob store answers show-me-everything, only when you ask.
Sampling hooks
Capturing every span at full fidelity is the line item that surprises people. LLM traffic is expensive to store because the payloads are big, and most of it is unremarkable — successful, cheap, fast requests you will never look at.
Head-based sampling decides at the start of a trace, before you know anything interesting, so it throws away failures at the same rate as successes. For LLM systems that is backwards. The 0.3% of traces that errored, cost a lot, or scored badly on the judge are exactly the ones worth keeping.
Tail-based sampling, run in the collector, decides after the trace finishes and can read its attributes. That is the hook that matters. Keep everything that is interesting, sample the boring remainder hard:
processors:
tail_sampling:
decision_wait: 10s
policies:
- name: keep-errors
type: status_code
status_code: {status_codes: [ERROR]}
- name: keep-expensive
type: numeric_attribute
numeric_attribute:
key: app.llm.cost_usd
min_value: 1
- name: sample-the-rest
type: probabilistic
probabilistic: {sampling_percentage: 5}
That config keeps every error, keeps every trace that cost more than a dollar, and keeps 5% of everything else. The cheap scalar attributes you set at emit time are what make this possible — the sampler reads app.llm.cost_usd to make the keep decision, which is one more reason to compute cost in-process rather than after the fact.
The shape that holds up
Cheap scalars for everything you query and alert on. Content captured deliberately, redacted in-process, truncated on purpose, with the big stuff pushed to object storage. Tail sampling that keeps the interesting traces and drops the boring majority. That is a span you can still afford and still query in six months, instead of one you regret.
The hard part was never wrapping the call. It is deciding what not to keep.
If you want the full attribute set, the redaction patterns by data class, and the sampling configs worked through against real backends, the LLM Observability Pocket Guide covers what a healthy span looks like and how to keep it that way as model versions and conventions rotate underneath you.
Top comments (0)