Observability for LLM Apps: Tracing, Cost Tracking, and Eval Loops

If you've shipped a traditional backend service, you already know the observability checklist: logs, metrics, traces, alerts. LLM-powered apps need all of that — plus a few things that don't exist in a normal request/response world: token spend, prompt/response pairs, and quality drift that no HTTP status code will ever tell you about.

This post covers the three pillars that actually matter once an LLM app leaves the demo stage:

1. Tracing — following a request across retries, tool calls, and multi-provider fallbacks
2. Cost tracking — turning token counts into real-time dollar visibility per route, user, or feature
3. Eval loops — catching quality regressions before your users do

I'll use patterns from a multi-provider AI gateway (Rust, Axum, Prometheus) as the running example, but the ideas apply regardless of stack.

Why normal APM isn't enough

A standard APM trace tells you latency and status code. For an LLM call, that's barely half the picture. The same 200 OK can represent:

• A cheap 400-token completion from a fast model
• A $2 completion from a frontier model after two retries
• A technically successful but hallucinated response

None of that shows up in a duration histogram. You need traces that carry LLM-specific attributes, not just HTTP ones.

Pillar 1: Tracing

The unit of work in an LLM app usually isn't a single call — it's a chain: prompt construction → provider call → possible retry/fallback → tool invocation → final response. Each of those needs to be a span, not just the outer request.

A minimal span shape for an LLM call looks like this:

#[derive(Debug, Serialize)]
struct LlmSpan {
    trace_id: String,
    span_id: String,
    parent_span_id: Option<String>,
    provider: String,        // "anthropic", "openai", "ollama"
    model: String,
    prompt_tokens: u32,
    completion_tokens: u32,
    latency_ms: u64,
    status: SpanStatus,      // Success, Retried, Failed, RateLimited
    retry_count: u8,
    fallback_from: Option<String>, // which provider it fell back from
}

The two fields that matter most for debugging in production are retry_count and fallback_from. Without them, a "successful" trace hides the fact that your primary provider is degrading and your fallback logic is quietly absorbing the pain — until it can't.

Wire this into OpenTelemetry so it plays nicely with whatever trace backend you already run:

use opentelemetry::trace::{Tracer, Span};

let tracer = global::tracer("llm-gateway");
let mut span = tracer.start("llm.completion");
span.set_attribute(KeyValue::new("llm.provider", provider.to_string()));
span.set_attribute(KeyValue::new("llm.model", model.to_string()));
span.set_attribute(KeyValue::new("llm.prompt_tokens", prompt_tokens as i64));
span.set_attribute(KeyValue::new("llm.completion_tokens", completion_tokens as i64));
span.set_attribute(KeyValue::new("llm.retry_count", retry_count as i64));

This gives you a waterfall view in Jaeger/Tempo where a fallback chain is visually obvious instead of buried in logs.

Pillar 2: Cost tracking

Token counts alone don't tell a story anyone outside engineering cares about. The move is to convert tokens to cost at write-time and expose it as a first-class Prometheus metric, not something you reconstruct later from a token count times a price you looked up once and forgot to update.

use prometheus::{register_counter_vec, CounterVec};

static LLM_COST_USD: Lazy<CounterVec> = Lazy::new(|| {
    register_counter_vec!(
        "llm_cost_usd_total",
        "Cumulative LLM spend in USD",
        &["provider", "model", "route"]
    ).unwrap()
});

fn record_cost(provider: &str, model: &str, route: &str, prompt_tokens: u32, completion_tokens: u32) {
    let (input_rate, output_rate) = pricing_for(provider, model);
    let cost = (prompt_tokens as f64 * input_rate) + (completion_tokens as f64 * output_rate);
    LLM_COST_USD
        .with_label_values(&[provider, model, route])
        .inc_by(cost);
}

Keep pricing_for in a config file, not hardcoded — provider pricing changes more often than you'd like, and a stale rate table quietly lies to your finance dashboard.

With provider, model, and route as labels, one PromQL query answers the question every team eventually asks:

sum by (route) (
  rate(llm_cost_usd_total[1h])
)

That's "which feature is burning the budget," per hour, without a spreadsheet. Pair it with a budget-based alert:

- alert: LLMCostSpike
  expr: sum(rate(llm_cost_usd_total[15m])) * 3600 > 5
  for: 10m
  annotations:
    summary: "LLM spend exceeding $5/hr sustained rate"

This is the single highest-leverage dashboard panel you can add — it's usually the first thing a manager asks for once real usage starts.

Pillar 3: Eval loops

This is the part with no equivalent in traditional backend observability. A response can be fast, cheap, and a 200 — and still be wrong. Nothing in your metrics stack catches that unless you build for it.

The practical version of an eval loop, at gateway scale, isn't a big offline benchmark suite — it's a lightweight, continuous check on production traffic:

1. Golden set regression — a small, versioned set of prompt/expected-output pairs run against every model/provider change before it ships
2. Sampled production scoring — periodically re-score a random 1–2% of real completions using a cheaper judge model, and track the score as a metric over time
3. Structural checks — cheap, deterministic checks that don't need a judge model at all: did the JSON parse, is the response non-empty, is it within expected length bounds

struct EvalResult {
    trace_id: String,
    check_name: String,       // "json_valid", "length_bounds", "judge_score"
    passed: bool,
    score: Option<f32>,
}

async fn run_structural_checks(response: &str) -> Vec<EvalResult> {
    vec![
        EvalResult {
            trace_id: current_trace_id(),
            check_name: "json_valid".into(),
            passed: serde_json::from_str::<Value>(response).is_ok(),
            score: None,
        },
        EvalResult {
            trace_id: current_trace_id(),
            check_name: "length_bounds".into(),
            passed: (20..4000).contains(&response.len()),
            score: None,
        },
    ]
}

Feed the pass rate into the same Prometheus instance as everything else:

static EVAL_PASS_RATE: Lazy<GaugeVec> = Lazy::new(|| {
    register_gauge_vec!(
        "llm_eval_pass_rate",
        "Rolling eval pass rate by check",
        &["check_name"]
    ).unwrap()
});

Now a model swap, a prompt template change, or a silent provider-side model update all show up as a dip on the same dashboard where you already watch latency and cost — instead of as a support ticket three days later.

Putting it together

The dashboard that actually gets used day-to-day has three rows, matching the three pillars:

• Tracing row: p50/p95 latency by provider, retry rate, fallback rate
• Cost row: spend by route/model, hourly burn rate, budget alert status
• Quality row: structural check pass rate, sampled judge score trend

None of these pillars is optional once an LLM feature has real users. Tracing tells you what happened, cost tracking tells you what it's costing you, and eval loops tell you whether it's still working — and that third one is the piece most teams skip until it's already a production incident.

What's next

The next post in this series covers inference optimization — KV cache strategies and quantization tradeoffs for teams that aren't running hyperscaler-scale infrastructure but still need to control latency and cost at the model-serving layer.

---

This is part of a series on AI infrastructure patterns, drawing on production work building multi-provider AI gateways in Rust. Follow along for the rest of the series.