The single most valuable thing you'll build isn't the agent — it's the test set you evaluate it against. It's the ground truth every version is measured on, and it survives model swaps, framework changes, and rewrites.
Start with ten real cases, each paired with a verdict for what good looks like, and grow the set with every failure you find.
Why the test set is the asset
An evaluation is only as good as the cases it runs. A test set is a collection of scenarios — inputs paired with some notion of what a good response looks like — that represents the situations your agent must handle. It is the ground truth against which every version of the agent is measured, and it is the one asset that survives model changes, framework changes, and rewrites. Build it well and it pays off on every future decision.
Models change. Frameworks change.
The test set endures.
What a good test set contains
- Representative cases — the common situations your agent actually faces, so the score reflects real performance.
- Edge cases — the rare, tricky, and adversarial inputs where agents break, because these are what production surfaces.
- Known failures — every bug you've found, captured as a case, so it can never silently return.
- A verdict per case — an expected answer, a checklist, or a rubric. A case without a verdict can't evaluate anything.
Where test cases come from
The best test cases come from reality. Real user interactions — especially the ones that went wrong — are gold, because they represent situations that actually happen. Every production failure should become a test case. You can supplement with synthetic cases the model or your team generates to cover situations you haven't seen yet, but the core of a strong test set is drawn from real usage, curated over time.
Free Agent Evaluation QuickStart — the whole loop (define, measure, test, trust) on a few pages. Download it free.
A case is an input plus a verdict
Be precise about what a single case is: an input the agent will receive, and a way to decide whether the response was good. That verdict takes different forms — an exact expected answer, conditions the response must satisfy, a rubric a judge applies, or a reference to compare against. The discipline of writing the verdict for every case is what turns a pile of examples into an actual test set.
{
"input": "I was charged twice, I want a refund",
"expects": {
"task_success": "acknowledges double charge, checks policy",
"tool": "issue_refund only if within policy",
"faithfulness": "cites the refund policy",
},
}
Start small, grow deliberately
A common mistake is waiting to build a huge test set before evaluating anything. Start with ten cases that capture what matters, and grow the set as you learn where the agent fails. Twenty well-chosen scenarios that cover your real risks beat a thousand generic ones. The test set is a living asset that grows with every bug found and every new situation encountered.
Going deeper? AI Agent Evaluation & Testing: The Complete Guide is the full reference — 40 pages, 15 chapters, 5 appendices, with a worked support-agent example and a 30-day adoption path. Get the guide.
FAQ
How do I build a test set for an AI agent?
Collect scenarios — inputs paired with a verdict for what good looks like. Include representative cases, edge cases, and every known failure. Pull from real usage, write a verdict for each, and start with about ten rather than waiting for a huge set.
How many test cases do I need?
Start with about ten well-chosen cases that cover what matters, then grow as you find failures. Twenty scenarios covering your real risks beat a thousand generic ones. The set is meant to grow over time, not be complete on day one.
Where do good test cases come from?
From reality — real user interactions, especially the ones that went wrong. Every production failure should become a case. Synthetic cases can supplement coverage, but the core comes from actual usage.
What makes a test case complete?
An input plus a verdict: some way to decide whether the response was good. The verdict can be an exact answer, a checklist of conditions, or a rubric. Without a verdict, a case can exercise the agent but can't evaluate it.
Why is the test set so important?
It's the ground truth every version of the agent is measured against, and it outlives models, frameworks, and rewrites. It's the accumulated definition of what good means for your task — the asset that makes every future change safer.
Top comments (0)