LLM Observability: Trace Cost, Sampling & Privacy

LLM observability: why traces, not just logs

You probably already log responses from your LLMs. But logs alone often miss the "why." Traces — structured spans that show prompt → tooling → response — expose the call flow, token burn, retries, and downstream tool behavior that explain model regressions. They also raise hard production trade-offs: full tracing burns tokens and storage, and can leak PII. Treating LLM observability as a set of practical trade-offs (not an all‑or‑nothing feature) keeps your team fast and compliant.

A 5‑step checklist for actionable LLM observability

Below is a pragmatic checklist I use to ship observability that answers questions fast without bankrupting or exposing the business.

1) Instrument spans — capture the call flow

Wrap each user operation in a root span and make every LLM call, retrieval, and tool invocation a child span. Use the OpenTelemetry GenAI semantic conventions where possible (gen_ai.request.model, gen_ai.usage.input_tokens, etc.). A single conversation ID propagated across spans turns per-call traces into a navigable conversation timeline.

Why: you need to answer "why did this output happen?" not just "what did the user receive?" With spans you can see which tool failed, which retrieval result was used, and which model version answered.

2) Attach token counts & cost — record lightweight metrics on spans

Store numeric attributes for input/output token counts and a computed cost (USD) per span. Compute cost in-process or in a span processor so the sampler/collector can use cost to keep expensive traces.

Example attributes (OTel gen_ai names):

• gen_ai.usage.input_tokens
• gen_ai.usage.output_tokens
• app.llm.cost_usd (computed)

Why: token counts—not request rate—drive your bill. Without token-level telemetry you can't detect runaway spend or bad prompt growth.

3) Redact sensitive content early — scrub before persistence

Treat prompts and completions as sensitive by default. Options:

• App-level redaction: remove or hash PII before emitting spans.
• Gateway redaction: chokepoint for traffic routing and sanitization.
• Collector redaction: last backstop (but data already left process).

Record a boolean flag (e.g., app.llm.redacted = true) and include non-reversible hashes or prompt template IDs so you retain investigatory signal without raw content.

4) Sample smartly — preserve signal, limit volume

Keep these rules simple and enforceable:

• 100% capture for errors and schema changes (always keep failures).
• 100% capture for large-cost or high-token traces (threshold-based).
• Reservoir/probabilistic sampling (e.g., 1%) for routine successful traffic.
• Increase sampling during releases or for targeted tenants.

Tail-based sampling in the OpenTelemetry Collector is ideal because it decides after spans finish and can inspect token/cost attributes.

5) Alert on quality & drift — watch the right signals

Don't just alert on latency. Add alerts for:

• sudden token or cost spikes (tokens per minute, cost rate)
• hallucination rates (evaluation events or judge step failures)
• model finish reasons shifts (more "length" or "tool_call")
• per-tenant spend anomalies

Quality alerts plus sampled content let you escalate to triage with a real trace attached.

Concrete engineering example

Here is a compact Python example using OpenTelemetry-style spans. It shows creating a gen_ai span, setting token and cost attributes, marking redaction, and an example redactor function. In production you would also wire an OTLP exporter and a Collector with tail sampling.

from opentelemetry import trace
import hashlib

tracer = trace.get_tracer(__name__)

def redact_text(text):
    # Naive example: detect emails/CCNs with regex, replace, or hash.
    # Use a robust PII redactor in production.
    if not text:
        return None, False
    # example: keep only a SHA256 fingerprint of the normalized text
    h = hashlib.sha256(text.encode("utf-8")).hexdigest()
    return h, True

def emit_llm_span(conversation_id, model, input_tokens, output_tokens, prompt_text):
    cost_usd = compute_cost_usd(model, input_tokens, output_tokens)
    prompt_hash, redacted = redact_text(prompt_text)

    with tracer.start_as_current_span("gen_ai.chat") as span:
        span.set_attribute("gen_ai.conversation.id", conversation_id)
        span.set_attribute("gen_ai.request.model", model)
        span.set_attribute("gen_ai.usage.input_tokens", input_tokens)
        span.set_attribute("gen_ai.usage.output_tokens", output_tokens)
        span.set_attribute("app.llm.cost_usd", cost_usd)
        span.set_attribute("app.llm.redacted", redacted)
        span.set_attribute("app.llm.prompt_hash", prompt_hash)
        # Optionally attach a short truncated excerpt as an event if allowed
        # span.add_event("prompt_excerpt", {"excerpt": prompt_text[:200]})

def compute_cost_usd(model, in_tok, out_tok):
    # Keep a centralized price table or compute in a span processor
    PRICE_PER_1K = {"gpt-4o": (0.003, 0.012)}
    in_p, out_p = PRICE_PER_1K.get(model, (0.0, 0.0))
    return (in_tok/1000.0)*in_p + (out_tok/1000.0)*out_p

OpenTelemetry Collector tail sampling example (YAML snippet):

processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: keep-expensive
        type: numeric_attribute
        numeric_attribute:
          key: app.llm.cost_usd
          min_value: 0.5
      - name: sample-rest
        type: probabilistic
        probabilistic:
          sampling_percentage: 1.0

This keeps all errors, all traces costing >= $0.50, and 1% of everything else.

Trade‑offs and practical policies

• Sampling: higher sampling detects regressions faster but increases storage and telemetry token costs. Use higher sampling during rollouts or for critical tenants.
• Redaction: aggressive redaction protects privacy but can remove root‑cause text that speeds debugging. Use hashes, template IDs, and per‑tenant opt‑in for full captures.
• Cost computation: compute cost early so the sampler can make decisions; keep the authoritative price table in one place (a span processor or gateway).
• Storage & retention tiers: keep full traces briefly (7–30 days), metadata longer, and aggregated metrics indefinitely.

A simple policy that works: 100% structural spans (attributes only), 100% errors and high-cost traces, 1% full-content reservoir sampling, and a collector redaction layer as a safety net.

Start small, iterate fast

Begin with three things: instrument spans, attach token counts & computed cost, and add a redaction flag. Wire those to your existing APM and confirm you can query cost by model and by tenant. Once you can answer "who burned the most tokens in the last hour" and "which span failed", add sampled content capture and alerts.

LLM observability is a production necessity. Scope it sensibly, adopt GenAI/OpenTelemetry conventions for portability, and trade visibility against privacy and budget with clear policies.

What part of LLM observability causes the most pain on your team — cost, privacy, or noisy alerts?

Comments

[Raju Dandigam]

Redact before persistence and sample after outcome is the right pairing. One extra field I’ve found valuable is the sampling-policy version plus a dropped-span counter. Without those, a quiet dashboard can mean healthy traffic, an aggressive policy, or exporter loss—and those are very different conclusions. Tail sampling can still make good decisions from cost, finish reason, validation failures, and tool metadata even when raw prompts never leave the process. Do you retain the sampler’s decision reason on kept traces for later audits?

[kironovlaziz-del]

Great breakdown, especially the redaction tiers. One thing I'd add from building a prompt firewall: doing redaction at the gateway (your option 2) has a nice side benefit beyond privacy — if you mask secrets before the prompt leaves for the provider, the provider never sees raw PII at all, so your observability layer and your provider both only ever handle scrubbed data. App-level redaction protects your traces; gateway-level redaction also protects you from the provider itself.

On your question — for us the hardest part was privacy vs. debuggability, exactly the trade-off you name. Prompt template IDs + non-reversible hashes turned out to be the sweet spot: you keep enough signal to correlate incidents without ever persisting raw content. Curious whether you've found tail-based sampling on cost interferes with that, since the expensive traces are often the ones you most want full content for.