Stop Vibe-Checking Your AI: How I Built a Real Eval and Found Hidden Failures

I built an AI app. It worked beautifully—for me. I'd throw a few test queries at it, nod approvingly at the responses, and call it a day. Vibe check: passed. Demo: polished. I was ready to ship.

Then real users showed up. Within minutes, the cracks appeared: unhandled edge cases, prompt failures, and impressive, highly confident hallucinations.

My "vibe check" testing wasn't going to cut it; I needed a rigorous way to evaluate and iterate on my system. That led me to Agentic AI: Build Your First Agentic AI System by Aishwarya Naresh Reganti. While the course relied on standard OpenAI API calls, I adapted its evaluation methodology and agentic routing benchmark to run locally using qwen2.5:7b. Measuring every output under those constraints taught me six fundamental lessons about AI evaluation that completely changed how I build.

Lesson 1: Start with action autonomy. Earn your way up.

Before any code, the course hit me with a framing I hadn't internalized:

Agency and control are a trade-off, not a slider you crank up.

Every time you give an AI more autonomy, you give up a little direct human oversight. The goal isn't "as autonomous as possible." The goal is the right operating point for the risk you're actually taking on.

The course breaks autonomy into three levels:

Level	What the agent does	Risk	Example
Action autonomy	One discrete task (a tool call, a classification)	Low	Route a ticket to the right team
Planning autonomy	Multi-step reasoning with a plan, but humans still in the loop	Medium	Research an issue, propose a resolution, wait for approval
End-to-end autonomy	Owns the whole workflow	High	Process a refund from start to finish

The advice I kept underlining: start at action autonomy. Earn the right to climb. This is the opposite of what I would have done on my own, which is "let's build the cool end-to-end thing first."

Lesson 2: Write evals before you write a single prompt.

I used to think "evals" was something AI labs did with their giant internal benchmarks. Turns out, for agent work, your eval set can be just 20 to 100 labeled examples that you carry through every iteration.

The instructor picked a customer support routing problem: given a free-text message, send it to one of seven departments:

• BILLING — charges, refunds, fees
• RETURNS — returns, exchanges
• TECHNICAL_SUPPORT — login, checkout, bugs
• ORDER_STATUS — tracking, shipping
• PRODUCT_INQUIRY — specs, availability, pricing
• ACCOUNT_MANAGEMENT — profile, payment method, address
• ESCALATION — angry customer, supervisor request

The 30 hand-labeled messages came with the course material. Distribution:

BILLING               6
TECHNICAL_SUPPORT     6
RETURNS               5
PRODUCT_INQUIRY       5
ACCOUNT_MANAGEMENT    4
ORDER_STATUS          2
ESCALATION            2

The set covers all seven departments, mixes obvious keyword cases with ambiguous ones, and throws in a few "trick" messages (a frustrated customer, a message with two possible departments). The point was to have a target, provided up front, that we could honestly fail at.

Lesson 3: Trace every call, from day one. If you can't see what the model said, you're guessing.

My Setup

To follow along with the course without relying on paid API credits, I swapped out the default hosted stack for a local setup:

• Ollama running qwen2.5:7b-instruct-q4_K_M — an open-weight 7B parameter model, quantized (Q4_K_M) to run efficiently on local hardware.
• OpenAI Python SDK pointed at Ollama’s OpenAI-compatible endpoint — allowing me to run the course's evaluation code almost entirely unchanged.
• Arize Phoenix for tracing calls, inspecting prompts, and monitoring execution flows.
• Python running inside VS Code Jupyter Notebooks (managed clean and fast via uv).

Phoenix is not optional. When a call goes wrong, you need to see the full context: the input, the prompt, the model's raw output, the system prompt that shaped it. Without that visibility, you're guessing on a non-deterministic system — and guessing is a waste of your time.

Lesson 4: An improving aggregate number can hide a real regression.

The first prompt was the smallest thing one could write:

SYSTEM_PROMPT_1 = """Route customer messages to departments.

Available departments: BILLING, RETURNS, TECHNICAL_SUPPORT, ORDER_STATUS, PRODUCT_INQUIRY, ACCOUNT_MANAGEMENT, ESCALATION

Respond with JSON ONLY:
{
    "department": "DEPARTMENT_NAME",
    "reasoning": "Your reasoning"
}
"""

No definitions, examples or rules. Just a list of names and a request in JSON format.

I ran all 30 cases.

76.7%. Higher than I expected from a bare-bones prompt — ACCOUNT_MANAGEMENT, ESCALATION, ORDER_STATUS, and RETURNS all hit 100% with zero help. But TECHNICAL_SUPPORT sat at just 33.3%, and that one category was doing a lot of the damage.

Looking at the 7 errors in Phoenix, the pattern was specific rather than random: login and password-reset messages kept landing in ACCOUNT_MANAGEMENT instead of TECHNICAL_SUPPORT, a promo-code complaint went to PRODUCT_INQUIRY, a refund-status question went to RETURNS instead of BILLING. The model wasn't guessing randomly — it was making the same specific category confusions over and over.

The fix was obvious once you saw the failures. We added definitions and disambiguation rules:

SYSTEM_PROMPT_2 = """You are an expert customer service router...

Definitions:
- BILLING: Charges, refunds, fees, payment failures.
- TECHNICAL_SUPPORT: Login issues, checkout errors, website bugs.
...

Important:
- Login/password problems = TECHNICAL_SUPPORT (not ACCOUNT_MANAGEMENT)
- Refund status = BILLING (not RETURNS)
...
"""

I ran it again.

86.7%. A +10.0 point jump, and TECHNICAL_SUPPORT went from 33.3% to 83.3%. Five of my seven original errors got fixed outright.

But two new errors appeared that hadn't existed before: RETURNS dropped from a perfect 100% down to 60%, with two messages ("I received a damaged item," "the item I received is completely different from what I ordered") both getting routed to ORDER_STATUS instead. Neither of those cases had been a problem in Prompt 1.

  BILLING              ████████████████░░░░ 83%   ← was 67%
  PRODUCT_INQUIRY      ████████████████████ 100%  ← was 80%
  RETURNS              ████████████░░░░░░░░ 60%   ← was 100%
  TECHNICAL_SUPPORT    ████████████████░░░░ 83%   ← was 33%

This is exactly the trap: +10 points overall looks like a clean win, but it's really "+50 points in one category, -40 points in another." If I'd only watched the headline number, I'd have shipped Prompt 2 thinking it was a strict improvement, when it had actually traded one broken category for another.

Lesson 5: A fix for one category can become a new bug in another.

The course's walkthrough stops at Prompt 2. For the third pass, I went off-script: I kept the definitions and the rules, and added five examples of my own design:

I ran the same 30 cases.

90.0%. A further +3.3 points, and — the part I actually cared about — RETURNS bounced back from 60% to a full 100%. The few-shot examples gave the model enough of a pattern to stop pulling damaged/wrong-item complaints toward ORDER_STATUS.

But the fix wasn't free. PRODUCT_INQUIRY dropped from a perfect 100% back down to 80% — the "your prices are too high" message that Prompt 2 had gotten right flipped back to the wrong answer (BILLING) in Prompt 3.

Every "improvement" in this experiment fixed something and broke something else. Only three of the seven departments stayed perfectly stable through all three iterations. The other four all moved in both directions at some point. That's the CCCD loop earning its keep: without the per-department breakdown, I'd have never caught any of this.

Lesson 6: Know when 90% accuracy is good enough.

Across every prompt iteration, two specific edge cases consistently failed:

1. "The promo code isn't working at checkout" — expected TECHNICAL_SUPPORT, but frequently routed to PRODUCT_INQUIRY or BILLING.
2. "Why didn't I get my loyalty points" — expected BILLING, but kept landing in ACCOUNT_MANAGEMENT or PRODUCT_INQUIRY.

Could I have forced the model to handle these exact cases with hyper-specific tie-breaker rules in the prompt? Probably. But these queries genuinely straddle the line between categories—a checkout error is both a technical bug and a billing query.

Chasing 100% accuracy on ambiguous edge cases usually leads to bloated, fragile prompts that break performance on standard queries. I was sitting at a solid accuracy score that handled the overwhelming majority of clean requests reliably. Realizing that edge-case ambiguity is an inherent feature of real-world data—and being comfortable leaving that minor margin of error as-is—saved me from over-engineering the system for marginal gains.

The numbers, side by side

90.0% is a genuinely good result for a fully local, quantized model with no fine-tuning. But the headline number undersells the real lesson. The value isn't in the 90% — it's in knowing which cases are still wrong, why, and what it would actually take to fix them.

What's next for me (and the series)

V1 is the action-autonomy floor: one LLM call, one label out. V2 in the course moves to planning autonomy — the agent will plan, look up policies, and propose multi-step resolutions. That one introduces a new piece: AI-as-Judge, where a second LLM call scores the first one instead of relying on a hardcoded ground truth.

If you're working through the same course, or building something similar, I'd love to compare notes. Drop a comment, or find me on [your contact link here].

Resources

• The notebook: I've put the full implementation up as a Colab notebook — link here so you can read through it, fork it, and follow along. One caveat: this notebook talks to a local Ollama server (qwen2.5:7b-instruct-q4_K_M), not a hosted API — Colab's VM won't have that running by default, so cells that call the model will error out until you either run Ollama locally and point the notebook's Config at your own machine, or install and start Ollama inside the Colab session itself.
• The course: Agentic AI: Build Your First Agentic AI System on DeepLearning.AI
• Stack: Python 3.12 · Ollama (qwen2.5:7b-instruct-q4_K_M) · OpenAI Python SDK · Arize Phoenix · uv + VS Code

See you in V2.