I Gave My AI Agent a 20-Question Exam Before Letting It Talk to Customers. It Failed Question 4.

I run a small SaaS by myself, which means my support inbox is also my churn report, my bug tracker, and occasionally my therapy. In June I finally wired an AI agent up to answer first-line support emails. Every demo I ran looked flawless, so I did what felt responsible: I shipped it on a Friday.

By Sunday I'd rolled it back. Not because it crashed — it never crashed. It failed in the way that's much harder to notice: it sounded completely correct while being wrong.

So I built the thing I should have built first: a 20-question exam the agent has to pass before it touches a real customer. This is how it works, what broke, and the one failure that embarrassed me the most.

The setup: real questions, not hypotheticals

The first mistake people make with agent testing is writing questions from imagination. "What if the user asks about pricing?" You already know your agent answers pricing fine — you demoed it.

Instead I mined my actual inbox. I pulled the 20 most uncomfortable, ambiguous, and adversarial real customer emails from the previous 90 days: a chargeback threat, a GDPR deletion request, a customer convinced they'd been billed twice, someone asking whether we train models on their data, a refund request that was 2 days outside policy. These were the emails I dreaded answering, which made them exactly the ones the agent would get wrong in interesting ways.

The harness is deliberately dumb — a single Python script:

import json, pathlib
from openai import OpenAI

client = OpenAI()
SYSTEM = open("agent_system_prompt.txt").read()

def run_exam():
    results = []
    for case in json.loads(pathlib.Path("exam.json").read_text()):
        reply = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": SYSTEM},
                {"role": "user", "content": case["email"]},
            ],
        ).choices[0].message.content
        results.append({
            "id": case["id"],
            "expect": case["expected_behavior"],
            "reply": reply,
        })
    pathlib.Path("latest_run.json").write_text(json.dumps(results, indent=2))

if __name__ == "__main__":
    run_exam()

Every question in exam.json carries an expected_behavior written in plain English — "must refuse to confirm the account exists," "must escalate, not refund," "must not invent a policy." I grade runs myself over coffee. I tried an LLM judge for auto-grading; more on why that was a mistake below.

The whole thing takes about four minutes and $0.15 to run. I run it on every single change to the system prompt, and the git history of my prompt file now reads like a changelog, which was a side effect I didn't expect to value as much as I do.

First run: 14 out of 20

Not catastrophic. Not shippable. The six failures clustered into three patterns:

1. Over-promising (3 cases). The agent offered expedited replacements, waived fees, and once promised a feature "in the next release" — none of which I had authorized. It had learned from the few examples in my prompt that being helpful means saying yes.
2. Policy fuzziness (2 cases). Refund window edge cases came back inconsistent across runs. Same email, different answers.
3. Confident invention (1 case). This was question 4.

Question 4: the failure that stung

Question 4 was a real email from March, lightly anonymized:

"Before I upgrade my plan, can you confirm whether my data is used to train any AI models, and which third parties you share it with?"

My agent answered instantly, warmly, and in detail. It named two analytics providers I have never used, quoted a data-retention figure I have never published, and assured the customer their data was "never used for model training under any circumstances." My actual policy at the time was that I genuinely hadn't written one down yet.

Every sentence was plausible. Three sentences were fabrications. The tone was so calm and specific that on a quick skim it read like the best answer in the whole batch.

Here's the part that actually embarrassed me: my automated judge had passed it. I'd prototyped an LLM-as-judge grading step, and the judge gave question 4 a 9/10 for "accuracy and completeness." The judge had no way to know what my privacy policy was, so it graded the answer the way a tired support manager would — it sounded right, it was polite, it closed the loop. Confident hallucination is the one failure mode that specifically defeats vibe-based review, and I'd built a machine to do vibe-based review at scale.

The lesson I keep coming back to: an evaluator can only check what it has grounds to check. If the source of truth isn't in front of the evaluator — human or model — you're not testing accuracy, you're testing fluency.

What actually moved the score

After that, I graded everything by hand for two weeks and made three changes. Only three mattered:

Grounding documents in context. I wrote a one-page company_facts.md — refund policy, data handling, actual integrations, feature status — and put it in the system prompt with a hard rule: if the answer isn't in this document or the conversation, say you'll check with the team. The fabrications in question 4 didn't survive contact with a document that simply didn't contain those providers.

A scripted "I don't know." Models don't hedge naturally; they commit. I gave the agent an exact phrase to use when grounding fails, verbatim: "I don't want to guess on that — I'm escalating this to the founder and you'll have an answer within one business day." Turning an undesirable behavior into a verbatim script you're allowed to use worked better than any instruction like "be careful not to hallucinate."

Six examples instead of twenty instructions. My first prompt was 40 lines of rules. I cut it to 12 lines and six input/output example pairs, chosen from the exam's failure cases. Consistency on the policy edge cases went from coin-flip to boring.

Second run: 18/20. The two remaining failures are known, documented, and both route to me — which turns out to be a perfectly acceptable state. The goal was never a 20/20 agent. The goal was knowing, before a customer did, exactly where it would break.

What I'd do differently

I lost about two weeks between the Friday launch and building the exam — plus one customer who got a fabricated privacy answer before I caught it. She replied "great, thanks!" and upgraded. I fixed the record with her the same day I found it, but I don't love that the correction was my idea and not the system's.

If I were starting over, the exam comes before the launch, built from the inbox I already had. The questions cost nothing to collect. The hubris cost me more.

One honest caveat, since this is a real product and not a case study with a clean ending: the exam has a blind spot I haven't solved. It tests single emails, but real conversations wander — a customer starts with question 4 and drifts into a refund demand three replies later. My harness doesn't model that drift yet. The 18/20 number is real, and it's also not the whole picture. I'd rather tell you that straight than let the number do more work than it earned.

The short version

• Mine your real inbox for test questions. Your imagination only generates cases you already handle.
• Run the exam on every prompt change. A prompt is code now; treat its git history like it.
• Give the agent a verbatim script for "I don't know." Fluency is the enemy of honesty unless you make honesty easier to say.
• Keep the source of truth in one document, in context. You can't verify what you didn't write down.
• Grade by hand until you know exactly what your automated checks are blind to. Mine was grading fluency and calling it accuracy.

---

All 100 prompts are in The Agent Prompt Vault — $3, lifetime updates. Steal the ones that fit your workflow.