AI Observability and Evaluation: Catch Silent LLM Failures

Your AI support assistant just told a customer they can get a 30-day refund. They're on the Free plan, and Free has no refunds.

The API returned 200 OK in 1.2 seconds. There were no exceptions, CPU was normal, and the dashboard stayed green. Nothing in your monitoring knows anything went wrong.

Your infrastructure is healthy. Your product is wrong. This is a silent failure: the system executes the request successfully but delivers the wrong outcome.

Traditional monitoring asks whether the service is up, fast and error-free. AI applications need more questions answered:

• Did the model answer correctly?
• Was the answer supported by the retrieved documents?
• Did the agent call the right tool?
• Did a prompt change break something that used to work?
• Which step caused the bad answer?

That takes two capabilities working together: evaluation and observability.

What you'll build

In this tutorial you'll build the loop that catches the refund bug above:

1. A small evaluation dataset based on the refund scenario
2. A Python evaluator that runs under pytest and in CI
3. OpenTelemetry traces that show which step caused a bad answer
4. A link between each failed evaluation and its trace ID
5. A workflow that turns every production failure into a regression test

All code runs against a deliberately flawed stub app, so you'll see a real failure first. Then swap in your own answer_question().

Prerequisites: Python 3.9+ and pip install opentelemetry-api opentelemetry-sdk pytest.

1. Monitoring, observability and evaluation aren't the same thing

These terms get used interchangeably, but each answers a different question.

Capability	Question it answers	Example
Monitoring	Is the system healthy?	API error rate increased
Observability	What happened during this request?	Retrieval returned an outdated document
Evaluation	Was the result right?	The answer contradicted the approved policy
Continuous evaluation	Did a change make things worse?	A new prompt fails previously passing cases

In our refund example, monitoring says the model responded. Observability shows retrieval returned the Pro plan policy. Evaluation says that policy was wrong for a Free user. You need all three.

OpenAI's evaluation best practices recommend task-specific success criteria, representative datasets, automated checks where possible, and human judgment. A trace becomes far more useful once it's connected to an evaluation result, which is what we'll do below.

2. The architecture: four components

Before choosing a vendor or SDK, decide what you need to capture. A useful setup has four parts:

1. Instrumentation records what happened: model calls, retrieval, tool calls, duration, token usage, errors.
2. Observability backend stores traces and events so developers can investigate.
3. Evaluation pipeline checks outputs against explicit requirements using deterministic assertions, reference answers, human reviewers or model-based judges.
4. Feedback loop turns real failures into repeatable test cases that run against every future change.

The piece most teams miss is the join between 2 and 3: every evaluation result should carry the trace_id of the request it judged. Without it, a failed test tells you that something broke, but not where.

You can use OpenTelemetry for instrumentation, and a platform such as MLflow or LangSmith for tracing and evaluation workflows. You don't need all of them. Start with the smallest setup that gives you useful evidence.

3. Build an evaluation dataset before building dashboards

A common mistake: read a handful of AI responses, decide they look fine, and ship. A few good examples tell you almost nothing about edge cases or regressions.

Instead, build a small dataset where you know what acceptable behavior looks like. Create eval_cases.json:

[
  {
    "id": "refund-free",
    "question": "Can I get a refund on the Free plan?",
    "required_facts": ["no refunds"],
    "prohibited_claims": ["30 days", "30-day"],
    "expected_behavior": "answer"
  },
  {
    "id": "refund-pro",
    "question": "What is the refund window on the Pro plan?",
    "required_facts": ["30 days"],
    "prohibited_claims": ["no refunds"],
    "expected_behavior": "answer"
  },
  {
    "id": "refund-unknown-plan",
    "question": "What is the refund policy for the Enterprise Plus plan?",
    "required_facts": [],
    "prohibited_claims": ["30 days", "no refunds"],
    "expected_behavior": "abstain"
  }
]

These policies are illustrative. In a real project, required facts and prohibited claims should come from approved documentation and domain-expert review.

Notice the two separate fields. required_facts is what a correct answer must contain. prohibited_claims is what it must never say. Defining both is better than comparing against one reference sentence, because many different answers can be correct.

A solid dataset covers five categories:

• Normal cases: common requests.
• Boundary cases: incomplete, ambiguous or very long inputs.
• Negative cases: questions the available evidence can't answer.
• Adversarial cases: prompt injection, conflicting instructions, attempts to bypass access controls.
• Regression cases: failures you've already seen in production.

Rare failures can be more damaging than common ones, so don't optimize only for the happy path.

4. Write your first automated evaluator

Start with deterministic checks. They're cheap, repeatable, and ideal for rules you can state exactly.

First, the flawed stub app. Save it as my_ai_app.py. Its retriever has a deliberate bug: it always returns the Pro plan policy, the way a real retriever might pick the wrong document.

# my_ai_app.py  (deliberately flawed stub)
POLICIES = {
    'free': 'The Free plan has no refunds.',
    'pro': 'The Pro plan offers refunds within 30 days.',
}


def retrieve(question):
    # Bug: ignores the plan named in the question
    return [{'id': 'policy-pro-v2', 'text': POLICIES['pro']}]


def generate(question, documents):
    return documents[0]['text']


def answer_question(question):
    docs = retrieve(question)
    answer = generate(question, docs)
    return {'answer': answer, 'abstained': False}

Your real answer_question() should return the same structure: an answer string and an abstained boolean.

Now the evaluator. Save it as evaluate.py:

import json
from pathlib import Path


def load_cases():
    return json.loads(Path('eval_cases.json').read_text(encoding='utf-8'))


def check(case, result):
    """Return None if the case passes, otherwise a failure reason."""
    answer = result.get('answer', '')
    text = answer.casefold() if isinstance(answer, str) else ''

    # Prohibited claims fail every case, including abstentions
    for claim in case['prohibited_claims']:
        if claim.casefold() in text:
            return f'prohibited claim present: {claim}'

    if case['expected_behavior'] == 'abstain':
        if result.get('abstained') is not True:
            return 'should have abstained'
        return None

    missing = [f for f in case['required_facts'] if f.casefold() not in text]
    return f'missing facts: {missing}' if missing else None


def evaluate_case(case, answer_question):
    result = answer_question(case['question'])
    reason = check(case, result)
    return {'case_id': case['id'], 'passed': reason is None, 'reason': reason}


def run_evaluation(answer_question):
    results = [evaluate_case(c, answer_question) for c in load_cases()]
    failures = [r for r in results if not r['passed']]
    passed = len(results) - len(failures)

    # Log IDs and reasons only, never raw user content
    for r in failures:
        print(f"FAIL {r['case_id']}: {r['reason']}")
    print(f'{passed}/{len(results)} passed')

    return {'passed': passed, 'total': len(results), 'failures': failures}

Add the regression test as test_quality_gate.py:

from evaluate import run_evaluation
from my_ai_app import answer_question


def test_ai_quality_gate():
    report = run_evaluation(answer_question)
    assert report['total'] > 0
    assert not report['failures']

Run it:

pytest -q

Output (abridged):

FAIL refund-free: missing facts: ['no refunds']
FAIL refund-unknown-plan: prohibited claim present: 30 days
1/3 passed

There's the bug from the opening: a Free-plan user gets the Pro policy, and your green dashboard would never have shown it.

Important limitation: substring matching isn't a general measure of factual correctness. A response can contain every required phrase and still mislead, and a correct paraphrase can fail a literal match. Use deterministic assertions for deterministic requirements, and add semantic or human evaluation where the task needs judgment.

In a real project, keep the evaluation report as a CI artifact and avoid printing sensitive user content to build logs.

5. Know which metrics actually matter

A single overall quality score can hide serious problems. Pick metrics based on the failure modes your application can't afford.

Metric	What it tells you	Limitation
Task success rate	Whether the task was completed correctly	Needs a reliable definition of success
Groundedness	Whether claims are supported by evidence	The evidence itself can be wrong
Retrieval precision and recall	Whether the right documents were retrieved	Doesn't prove the final answer is right
Tool-call accuracy	Whether the right tool and arguments were chosen	A correct call can still fail to execute
Abstention quality	Whether unsupported questions are declined	Both false refusals and unsafe answers matter
p95/p99 latency	How slow the slower requests get	Says nothing about answer quality
Cost per successful task	What a correct outcome costs	Depends on how you define success and cost

Pass rate (passed cases divided by evaluated cases) is the simplest of these, and also the easiest to misread. A high pass rate on a small, easy dataset can coexist with frequent real-world failures, so always report dataset size, coverage and failure categories next to the number.

Also evaluate at the right level. Failures happen at the single step (wrong tool), the full trace (retrieval plus generation), the conversation (lost context across turns), and the business outcome (the operation didn't actually succeed). A correct tool call is not a completed task, and a fluent answer is not a grounded one.

6. Add observability, and link it to your evaluations

Evaluation tells you a test failed. Observability tells you why.

Without a trace, you reproduce the request by hand and hope the failure happens again. With one, you can inspect each step:

Step	Evidence to capture	Example failure
Request	Correlation ID, route, safe metadata	Unexpected input format
Retrieval	Document IDs, relevance scores	Outdated policy document
Model call	Model ID, prompt version, duration, tokens	Changed prompt behavior
Tool execution	Tool name, validated arguments, status	Wrong tool selected
Validation	Rule IDs and pass/fail	Required fact missing
Outcome	Evaluation label, user feedback	Wrong answer reached the user

First, set up tracing. Save as tracing.py. This prints spans to the console so you can see them immediately; in production, swap the exporter for OTLP to your backend.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor

provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)

tracer = trace.get_tracer('support-assistant')

Next, instrument the stub app. Replace answer_question in my_ai_app.py with:

from tracing import tracer


def answer_question(question):
    with tracer.start_as_current_span('ai.request'):
        with tracer.start_as_current_span('ai.retrieval') as span:
            docs = retrieve(question)
            span.set_attribute('app.retrieval.document_ids', [d['id'] for d in docs])

        with tracer.start_as_current_span('ai.generation') as span:
            span.set_attribute('gen_ai.request.model', 'stub-model')
            answer = generate(question, docs)

        return {'answer': answer, 'abstained': False}

The gen_ai.* attribute follows OpenTelemetry's GenAI semantic conventions; app.* is a custom name. The conventions are still evolving, so check the current specification and your instrumentation library before depending on a specific attribute. The example deliberately records document IDs rather than raw prompts or document text.

Now the join. Update evaluate_case in evaluate.py so each result carries the trace ID:

from tracing import tracer


def evaluate_case(case, answer_question):
    with tracer.start_as_current_span('eval.case') as span:
        span.set_attribute('eval.case_id', case['id'])
        result = answer_question(case['question'])
        reason = check(case, result)
        span.set_attribute('eval.passed', reason is None)
        trace_id = format(span.get_span_context().trace_id, '032x')

    return {
        'case_id': case['id'],
        'passed': reason is None,
        'reason': reason,
        'trace_id': trace_id,
    }

And print it in the failure line: print(f"FAIL {r['case_id']}: {r['reason']} (trace {r['trace_id']})").

Because answer_question runs inside the eval.case span, its retrieval and generation spans share the same trace ID. Paste that ID into your trace backend and you'll see ai.retrieval returned policy-pro-v2 for a Free-plan question. You've gone from "test failed" to "retrieval ignored the plan" in one step.

Don't stop at the final response

An agent can retrieve the wrong documents, retry a failed tool, fall back to a second model, and still return a plausible answer. If you record only the final response, that history is gone. For multi-step applications, trace the whole workflow, use consistent correlation IDs, and make sure asynchronous work preserves trace context.

For more, see OpenTelemetry's GenAI observability walkthrough.

7. Use model-based judges carefully

Some requirements can't be checked with string matching: Is the answer relevant? Does it contradict the supplied documents? Is it complete enough to solve the user's problem?

An LLM judge can score outputs against an explicit rubric. Provider-agnostic sketch (pass in your own call_llm):

import json

JUDGE_PROMPT = """You are grading a support answer.
Question: {question}
Allowed evidence: {evidence}
Answer: {answer}
Return only JSON: {{"grounded": bool, "relevant": bool, "complete": bool, "failure_reason": str}}"""


def judge(question, answer, evidence, call_llm):
    raw = call_llm(JUDGE_PROMPT.format(question=question, answer=answer, evidence=evidence))
    return json.loads(raw)

A judge is itself a model. It can miss errors, favor verbose answers, or agree with a fluent but wrong one. Add safeguards:

1. Write rubrics with examples of passing and failing outputs.
2. Validate judge decisions against human-reviewed examples.
3. Use deterministic checks wherever a requirement can be checked exactly.
4. Track disagreement between the judge and human reviewers.
5. Recalibrate when models, prompts or criteria change.
6. Never rely on a single judge score for high-impact decisions.

Save human review for the decisions where a wrong answer is expensive.

8. Turn production failures into regression tests

Here's a scenario that's easy to picture. A SaaS company's support assistant starts telling Free-plan users they can claim a refund. Support tickets don't spike, because the answer sounds reasonable and most people just accept it. A few weeks later, finance notices a pile of refund requests that don't match the policy.

Someone finally pulls the trace for one of the bad answers. The generation step looks fine; the model summarized exactly what it was given. The retrieval step is the problem: it returned the Pro plan policy, because a recent re-indexing job had dropped the plan metadata the retriever filtered on. Every Free-plan refund question was quietly matching the Pro document.

Nothing in the logs flagged it, and no test covered it. The fix was a few lines in the indexer. The more useful change came after: the team added that question to the evaluation dataset as a regression case, so the next change to indexing or retrieval would fail in CI instead of reaching a customer.

After a failure like that, the loop is always the same:

1. Retrieve the failed trace by its correlation ID.
2. Find the earliest step where behavior diverged from expectations.
3. Redact sensitive information before turning it into a reusable example.
4. Add the case to the evaluation dataset.
5. Fix the underlying cause.
6. Run the full regression suite and compare against the previous baseline.
7. Monitor the deployment for similar failures.

Different failures need different responses:

Failure	Likely next step
Retrieval returned irrelevant documents	Inspect queries, filters and ranking
Correct evidence, incorrect answer	Inspect prompt, model output and eval cases
Correct answer, wrong tool action	Fix authorization, tool schemas or execution logic
Sudden latency increase	Inspect spans, queue time, model duration and retries
Higher cost per successful task	Inspect token usage, retries and model routing
Quality regressed after deployment	Compare eval results and trace distributions across versions

Fix the failure at its source. Adding another sentence to the prompt every time something goes wrong is not a fix.

9. Add an evaluation gate to CI/CD

Run your dataset whenever you change anything that can affect behavior: system prompts, model versions and parameters, retrieval and embedding models, document ingestion and chunking, tool schemas, and safety validators.

A minimal GitHub Actions workflow:

name: ai-eval
on: [pull_request]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install opentelemetry-api opentelemetry-sdk pytest
      - run: pytest -q

The pipeline order is: code or prompt change, then unit tests, then the golden evaluation dataset, then a baseline comparison. Critical regressions fail the build. Otherwise review quality and cost trade-offs, deploy gradually, monitor traces, and feed reviewed failures back into the dataset.

Don't use an arbitrary universal pass-rate threshold. A regression in a formatting task and a regression in financial actions or access control deserve different release policies. A sensible starting policy:

• No failures in mandatory deterministic security tests.
• No unacceptable regression in critical task-success metrics.
• Quality compared against a known baseline.
• Latency and cost within limits for the service.
• Human approval for changes to high-impact workflows.

A fixed test set also goes stale. Keep expanding it with production failures and reviewed feedback.

10. Privacy: observability can create a second data leak

AI traces can contain user messages, personal data, proprietary documents and API arguments. OpenTelemetry's own conventions warn that some GenAI attributes may hold sensitive information.

Before enabling content capture, ask:

• Do we need the full prompt or response, or are document IDs and redacted excerpts enough?
• Who can access production traces, and for how long are they retained?
• Are secrets and personal data removed before export?
• Are tenant boundaries enforced in the backend itself?
• Can users request deletion where applicable?

Default to metadata-only traces, and enable content capture narrowly for specific debugging workflows. Redact before data reaches the exporter, restrict access, and encrypt in transit and at rest.

Also, a system prompt is not a security boundary. Your application and tools must enforce authorization, not rely on the model choosing to follow instructions.

11. What to implement first

You don't need an AI platform on day one. Follow this order:

Phase 1: Baseline. Define success for your main workflow. Build a small reviewed dataset covering normal, boundary, negative and adversarial cases. Add deterministic checks.

Phase 2: Instrument. Trace retrieval, model calls, tools and validation. Capture duration, errors, token usage and safe correlation metadata.

Phase 3: Connect. Attach trace IDs to evaluation results, record failure categories instead of only a score, and turn reviewed failures into regression tests.

Phase 4: Automate. Run evals in CI, compare against a baseline, add sampled online evaluation where cost and latency allow, and feed production failures back into the dataset.

Phase 5: Harden. Add privacy controls, retention policies, alert thresholds and release criteria. Track quality next to latency, reliability and cost.

Order matters. Instrumenting everything before you know what you need to learn produces telemetry without improving your ability to diagnose failures.

Conclusion

Monitoring only tells you the service responded. To know whether the answer was right you need evaluation, and to know why it was wrong you need traces. Without both, that Free-plan customer keeps hearing about 30-day refunds.

You don't need a platform to start. Pick one workflow, write 20 cases you can defend, trace retrieval and generation, and add the eval to CI. Every real failure after that becomes a permanent test.

You won't prevent every bad answer. What you can do is catch the next one before a customer does, see why it happened, and make sure it only happens once.

Need help shipping reliable AI features?

We're TechEniac, and we build AI-powered SaaS products for startup founders. RAG pipelines, AI agents and LLM integrations are a big part of that work, and they only stay reliable when they're tested and traced the way this post describes. If you'd like a second opinion on how your AI features are evaluated and monitored, book a free strategy call. You can also read about our RAG pipeline development and AI agent development services.

Let's discuss: What's the hardest AI failure you've debugged in production: bad retrieval, hallucination, tool execution, latency, or a regression after a prompt or model change? Tell us how you caught it.

TechEniac is an AI SaaS development company for startup founders, with 100+ projects delivered and 8+ years of experience. We're ISO 9001 and ISO 27001 certified, with offices in New Jersey, USA and Ahmedabad, India. Learn more at techeniac.com.