Your AI Evals Are Lying to You

Your eval dashboard says 94% pass rate.

Your support agent just refunded a customer who never bought anything.

Both of these can be true at the same time. That's the problem with most AI evals: they measure what's easy to measure, not what actually breaks.

Why "accuracy" isn't enough for agents

Classic LLM evals ask one question: was the answer correct?

Agents need at least four:

Question	What it catches
Did it complete the goal?	Agent says "done" but the booking never happened
Was it accurate?	Wrong order number, invented policy, hallucinated price
Did it follow policy?	Skipped approval, exceeded a refund limit, shared private data
Did it take the right path?	Called 7 tools when 2 were enough, or called the wrong one

An agent can nail the first two and still fail the third. Policy failures are the ones that become incidents.

The 3 lies your eval suite might be telling

Lie #1: "It passed, so it's safe."
Your test set only contains questions your team thought of. Real users (and attackers) don't follow your script.

Lie #2: "The answer looked right."
For agents, the final message is the least important output. A polite, correct-sounding reply can sit on top of a forbidden tool call.

Lie #3: "Our score went up."
Swap the model or tweak a prompt, and the overall score can rise while one critical behavior quietly regresses. Averages hide disasters.

Build evals in two layers

Layer 1: code-based checks. Fast, cheap and deterministic. Use them for anything you can define as a rule.

def check_refund_policy(trace):
    refunds = [c for c in trace.tool_calls if c.name == "issue_refund"]
    for r in refunds:
        assert r.args["amount"] <= 500, "Refund over limit"
        assert trace.has_call("verify_order", before=r), "Refund without verification"

Layer 2: LLM evaluators. For fuzzy criteria like tone, grounding or "did it actually answer the question," use a judge model with a tight rubric.

RUBRIC = """
Score 1 if the response is fully grounded in the provided order data.
Score 0 if it states any fact not present in the data.
Return only the number.
"""

Rule of thumb: if you can write it as code, write it as code. Save LLM judges for what code can't express.

Where good eval cases come from

The best eval cases aren't brainstormed. They're discovered:

• Production traces. Every confused user conversation is a free test case.
• Bug reports. Every incident should leave behind a regression test.
• Red teaming. Adversarial testing finds the cases nobody would think to write.

That last one is the most powerful pipeline. Tools like BotGauge connect the two directly: behaviors found during red teaming become evaluations scored on goal completion, accuracy and policy adherence, so your eval set grows from real failures instead of guesses.

Score per behavior, not per suite

Stop reporting one number. Report a scorecard:

goal_completion      96%   ✅
factual_accuracy     93%   ✅
refund_policy        100%  ✅
approval_policy      88%   ❌  (regressed from 100%)
pii_handling         100%  ✅

One red row should block a release, no matter how green the rest looks.

Your eval checklist

• Each critical policy has its own eval, not just a share of an average
• Code checks for rules, LLM judges only for fuzzy criteria
• Evals inspect tool calls and traces, not just final answers
• Every incident and red-team finding becomes a test case
• Evals run on every model, prompt or tool change

Wrapping up

A high eval score is only as honest as the cases behind it. Measure behavior, not vibes, and let real failures shape your test set.

To see how red-team findings can feed straight into your eval suite, check out BotGauge.

What's one failure your evals missed that production caught? Drop it below. 👇