LLMOps for Production RAG: Observability & Cost Guide

LLMOps for production RAG: Observability, Evals & Cost Controls

If you want a single, practical playbook to take a RAG feature from prototype to production in one week, adopt Trace → Eval → Route. Treating models as operational surfaces—traceable, testable, and routable—separates experiments from dependable product.

This article expands a compact checklist into an actionable plan you can apply this week to cut RAG costs, stop silent regressions, and make AI features debuggable for product teams.

Why treat models as operational surfaces?

RAG systems are compound applications: retrieval, reranking, prompt assembly, model inference, and downstream parsing. Failures are rarely exceptions; they’re silent quality regressions (wrong sources, outdated chunks, hidden prompt changes) that return HTTP 200 with a confident-sounding wrong answer.

To catch those, you need three operational primitives:

• Trace: per-request lineage and metadata so you can answer “why did this change?” in minutes.
• Eval: continuous, focused checks that turn traces into quality signals and failing examples.
• Route: cost-aware model routing that enforces budget, SLAs, and safety.

Below is a concrete 3-step checklist with examples and a short Python snippet you can adapt.

1) Trace — per-request observability

What to record on every request:

• stable request_id and trace_id (propagate via headers)
• retrieval results: chunk IDs + similarity distances (not full text by default)
• the assembled prompt (redact PII where needed) and prompt template version
• model used, input/output token counts, and provider model hash
• latency breakdown (embedding, retrieval, rerank, generation) and upstream DB calls
• reranker scores, judge scores, and routing decision

Why this matters: with those fields you can replay and diff retrievals, pin prompt versions, and attribute cost to features.

Sampling rules and retention

• Always keep 100% of traces that produced an error, exceeded a cost threshold, or failed an evaluation.
• Sample 1–5% of happy-path traffic for online evals and drift detection.
• Store trace metadata and small artifacts (IDs, scores). Keep full prompts and responses behind gated access to control PII exposure.

Span schema example (attributes to include):

• trace.request_id, user_id, feature_name
• retriever.k, retrieved_ids, retrieved_scores
• reranker.version, reranker_scores
• model.name, model.version, tokens.input, tokens.output, cost
• prompt.template_version, prompt.hash

2) Eval — continuous, focused checks

Evals turn traces into signals you can act on. Build two parallel eval loops:

• Offline golden set: a curated set of hard queries that must pass on every deploy (CI quality gate).
• Online sampled evals: judge 1–5% of production traces asynchronously and attach scores to traces.

What to measure:

• context recall (does retrieved context contain the evidence?)
• context precision (how noisy is the context?)
• faithfulness (are claims supported by context?)
• cost per successful response (dollars per accurate answer)

Concrete routing example (confidence-based):

• After retrieval, compute a reranker confidence score in [0,1].
• If confidence >= 0.85, route to a cheaper 8B model.
• If confidence < 0.85, route to a 70B model or a human review queue.
• Log the routing decision, request_id, and expected cost delta.

Daily eval job:

• Re-run yesterday’s sampled traces through the latest reranker + judge.
• Measure accuracy delta and cost per successful response.
• Persist failing examples as bookmarks (with trace_id) for engineers and product owners.

Store regressions as first-class artifacts. A bookmarked failing trace should include retrieval IDs, prompt hash, model used, judge score, and owner.

3) Route — cost-aware, safety-first model routing

Routing rules you can implement quickly:

• confidence thresholds (from reranker or query classifier)
• query complexity: tokenized length or features-based classifier (contains code, tables, or long legal text)
• user tier: free users → cheaper model + cache, paid users → higher-quality path
• hard daily spend cap and emergency fallback (cached answer, reduced context, or human triage)

Examples of policies:

• "If reranker_conf >= 0.85 AND tokens_context <= 1500 → model=8B; else model=70B."
• "If user_tier == 'enterprise' AND reranker_conf<0.5 → route to human-review queue."
• "If daily_spend(feature) > soft_limit → degrade non-critical features to cached responses."

Routing is also where business rules live: safety checks, regulation requirements, and SLAs. Make routing decisions auditable by logging the rule and inputs that triggered them.

Quick code example (Python): tracing + routing decision

import uuid, time, json

# simplified trace emitter

def emit_trace(trace):
    # replace with OTLP/HTTP call to your observability backend
    print(json.dumps(trace))

def handle_request(user_id, query, retrieved):
    request_id = str(uuid.uuid4())
    start = time.time()

    # compute reranker confidence (placeholder)
    reranker_conf = rerank_and_score(query, retrieved)  # returns 0..1

    # routing rule
    if reranker_conf >= 0.85:
        model = "8b-cheaper"
    else:
        model = "70b-highquality"

    # assemble trace
    trace = {
        "request_id": request_id,
        "user_id": user_id,
        "retrieved_ids": [r['id'] for r in retrieved],
        "retrieved_scores": [r['score'] for r in retrieved],
        "reranker_conf": reranker_conf,
        "routing": model,
        "prompt_hash": "v1.3-abc123",
    }

    # call model (mock)
    response, tokens_in, tokens_out = call_model(model, query, retrieved)
    trace.update({
        "model": model,
        "tokens_in": tokens_in,
        "tokens_out": tokens_out,
        "latency_ms": int((time.time()-start)*1000),
    })

    emit_trace(trace)
    return response

Replace print with OpenTelemetry or your observability SDK. Keep prompt versions and reranker model pinned in traces.

Operational playbook: one-week plan

Day 1–2: Add request_id to a single endpoint and log retrieval IDs + scores, prompt hash, model name, and token counts to your observability sink. Keep full content gated.

Day 3: Add a simple reranker confidence and one routing rule (e.g., threshold to cheaper model). Log routing decisions.

Day 4–5: Implement a daily eval job that samples traces, runs judge checks, and bookmarks failing traces.

Day 6–7: Add spend tracking and a hard daily spend cap with a fallback (cache or human queue). Create a dashboard for retrieval quality, eval trend, and cost-per-success.

What you’ll get

Tracing gives you the "why" (why did the answer change?). Evals surface the "what changed" and provide regression detection. Routing gives you the levers to spend wisely and keep costs predictable. Combined, they turn RAG from a black box into an operational surface that teams can iterate on.

Which part of Trace → Eval → Route will give your team the most value this month? Pick one, ship in a week, and use the traces to build the next two.