Throughout the AI System Design series we follow Mailmind, a fictional AI email assistant that reads an inbox, triages messages, and drafts replies. It is our running example.
Before you read: The ReAct Loop: How Most AI Agents Actually Work and Evaluating Prompts: Beyond 'Looks Good to Me'.
A reliable agent evaluation pipeline rests on three pillars: a golden dataset of known-correct examples, a judge that can scale beyond human review, and a shadow mode that mirrors production traffic without user impact. Together they turn a nondeterministic system into something you can measure, compare, and improve with confidence.
Traditional software testing expects the same answer every time you press the button. AI agents don't work that way. The same inbox can produce different urgency flags on two consecutive runs. The interesting question is not whether the output changed. It's whether the change made things better or worse. And you can't answer that question with a single pass/fail test case.
Why standard test cases break with agents
A unit test for Mailmind's email classifier looks simple on paper: feed it an urgent thread and assert "urgent" is returned. In practice, the classifier might say "urgent" 90% of the time and "routine" 10% of the time because it took a different reasoning path. That 10% is not a crash to fix line by line. It is a rate to measure. Run each test case several times, report the pass rate, and decide what rate is acceptable. If you gate deployments on 100% pass rates, you never ship.
The core challenge is that agent quality is judgment-based, not binary. Two equally reasonable responses to "Is this email urgent?" might differ. A single pass/fail assertion cannot grade shades of quality across many runs. Evaluation for agents needs to capture precision, recall, consistency, and safety, all while accepting that some output variance is normal.
What exactly should you measure?
You need metrics across six axes: correctness, completeness, consistency, performance, reliability, and safety. No single number captures agent quality. Skipping one axis hides failures until users find them first.
Correctness measures whether the agent's decisions match the ground truth. For Mailmind, it answers: did the system identify the correct set of urgent emails? Correctness splits into precision (of the emails flagged as urgent, how many truly are) and recall (of the truly urgent emails, how many did we catch).
Completeness matters when the agent must extract structured information. Did Mailmind capture the meeting time, the refund amount, and the tracking number? A missed detail is as damaging as an incorrect one.
Consistency tracks how much the output changes across identical inputs. Some variation is normal, as above, so you measure it and set a limit. A jump in variance without a change in the underlying model or data is a red flag.
Performance covers latency and cost. For Mailmind, the p95 latency on triaging a batch of 20 emails (95% of batches finish faster than this) and the dollar cost per email are operational boundaries you need to monitor.
Reliability tracks failure modes: the rate at which tool calls fail, the model returns a malformed output, or the agent loop hangs. It's the uptime metric for an AI component.
Safety measures guardrail triggers: prompt injection detections, out-of-scope tool authorization denials, and personally identifiable information leaks. Each trigger is a signal you evaluate as part of the system's health.
How does a golden dataset anchor evaluation?
A golden dataset is a curated collection of inputs paired with expert-verified correct outputs and acceptable variance ranges. It is the source of truth that every other evaluation method calibrates against.
For Mailmind, a golden dataset starts with a hundred real inbox snapshots, anonymized. Each snapshot has a set of emails. A human annotator labels which emails are urgent, what the correct extracted details are (dates, amounts, entities), and what the ideal drafted reply should contain. The annotation process is iterative: two annotators label independently, then reconcile differences with a third reviewer. The final set becomes the ground truth.
The dataset must be diverse enough to represent the real distribution. If 90% of the original emails are routine newsletters, your golden set needs to reflect that. But you also need enough examples from each edge case: refund requests with missing order numbers, meeting invites with ambiguous times, and emails that look urgent but are spam. How many is enough depends on how small a change you need to see. With 100 examples in a category, a measured 90% could really be anywhere from about 84% to 96% (a 95% confidence interval). With 400, the range narrows to about 87% to 93%. So plan for a few hundred examples in each category that matters, and over-sample the failure modes you discover in production.
Once the golden dataset exists, you compute precision, recall, and F1 against it. Precision tells you how many of the emails Mailmind flagged as urgent were actually urgent. Recall tells you how many of the truly urgent emails were caught. F1 is a single score that balances the two. Together they give you an accuracy snapshot that respects the class imbalance between urgent and routine.
Keep an adversarial set beside it: a subset of attack emails from Guardrails: Input, Process, Output, malformed inputs, and huge threads. Pass means no forbidden tool call, no leaked data, and a graceful refusal or hand-off.
Updating the golden dataset is a continuous process. When a user reports that Mailmind missed an urgent email, that email gets anonymized, reviewed, and added to the set. When the system's job grows in a new direction (for example, after Mailmind starts handling travel confirmations), you add representative samples for the new category. A stale golden dataset is worse than no dataset because it gives a false sense of safety.
How does LLM-as-judge scale evaluation?
A second large language model (LLM), the judge, grades agent outputs against a rubric. This method is fast, reusable, and can run on every commit. It must be checked against human judgment on the same golden dataset before you trust it.
In Mailmind's pipeline, when the agent extracts a meeting time from an email, the judge model receives the original email, the extracted value, and a rubric that asks: "Is the extracted time exactly correct, partially correct, or wrong? Consider time zone ambiguity." The judge outputs a structured grade. For classification tasks, the judge can compare the agent's urgency label against the ground truth label and explain why it agrees or disagrees.
The judge is only as useful as its agreement with people. You check it by running both human reviewers and the judge on the same 200 golden examples. Then compare how often the judge agrees with humans to how often two humans agree with each other on the same items, and check that agreement separately for each label. Overall accuracy can hide a judge that is wrong on most of the rare urgent emails, and two humans rarely agree perfectly on subjective quality anyway. A judge that agrees with people about as often as people agree with each other is usable for regression gating. That is the comparison the best-known study of judges made (Zheng et al. 2023, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena): its strong judges matched human preferences more than 80% of the time, about the same level at which humans agreed with each other. If your judge falls short, you improve the rubric, switch to a more capable judge model, or keep human spot checks for that task.
One common failure mode: the judge rewards style over substance. An agent might produce a polite, confident but slightly incorrect draft reply, and the judge scores it highly because the rubric overweights tone. Tune the rubric to penalize factual errors heavily (for example, a point deduction for each misstated date). The same study documents other biases: position bias (favoring the answer shown in a particular position when comparing two), verbosity bias (favoring longer answers), and self-enhancement bias (favoring answers like its own). The countermeasures are simple: compare in both orders and keep only consistent verdicts, tell the rubric to ignore length, and use a judge from a different model family than the agent.
How do you grade the steps, not just the answer?
Everything so far grades the final output: a label, an extracted date, a draft. But an agent takes steps, and the steps can be wrong even when the answer looks right. A refund agent that gets the right amount by guessing, or that tries to send the reply itself before anyone approves it, has failed even if the final text reads well.
So a golden case for an agent stores the expected trajectory as well as the expected answer. For a Mailmind refund email, the case might say: look up the order, then check the refund policy, then draft a reply, and never call send_email. You then score four things:
- Tool choice: did the agent call the expected tools, and none of the forbidden ones?
- Arguments: did it pass the right order number and the right amount?
- Steps: did it finish within a sensible number of iterations, or wander?
- Completion: did it finish the task and stop, or hand off correctly when it could not?
Some of these are exact checks in code: a forbidden tool call is a fail, whatever the final answer says. Others need judgment, because a different order of steps can be just as valid. There, the judge reads the whole trace, not only the final answer. To keep runs repeatable, the read tools in an evaluation return recorded data, and the write tools only record what the agent tried to do. You need those traces in a form you can read and replay, which is what Debugging Agent Systems: Traces, Replays, Root Causes covers.
Where does human evaluation still fit?
Human evaluation is the gold standard that validates the golden dataset, calibrates the LLM judge, and audits production samples. It is too expensive to run on every change but is the only way to catch subtle judgment gaps.
In the Mailmind cycle, human reviewers perform three roles. First, they annotate the initial golden dataset, which is the foundation of everything downstream. Second, they spot-check a random sample of production outputs weekly, looking for issues that automated metrics miss, like a reply that is technically correct but socially off-key. Third, when the LLM judge flags an output as ambiguous, a human resolves the tie. This hybrid approach keeps quality high while controlling cost.
How does regression testing prevent silent breakage?
A regression suite re-runs the golden dataset and judge pipeline on every prompt, tool, or threshold change. Its goal is to detect the moment an improvement for one scenario degrades performance on another.
In Mailmind's CI/CD pipeline (continuous integration and delivery: the automated checks that run on every code change), when an engineer updates the urgency classification prompt to catch more refund-related emails, the regression suite runs 300 golden examples through the updated agent. The outputs are scored by the LLM judge. If recall for refunds improves from 88% to 94% but precision on routine newsletters drops from 97% to 91%, the suite surfaces that tradeoff. The team decides whether the recall gain justifies the precision loss before merging.
The suite must run fast enough to fit into a pull request workflow. Parallel execution spreads the load: 300 examples across 10 workers can finish in a few minutes. The key is that the suite catches regressions before a human reviewer ever sees the change.
What is shadow mode and why does it reduce deployment risk?
Shadow mode runs the new agent version in parallel with the existing one on live production traffic. Its outputs are logged and compared but never shown to users. This gives you a preview of real-data risk without any user-facing impact.
For Mailmind, deploying a new reply-drafting agent is high stakes. Before flipping the switch, the team puts the new version in shadow mode for about a week, so every weekday and a weekend appear at least once. Incoming email triggers both the old and new agents. The outputs are compared automatically: how often do they disagree? When they disagree, does the new version's output score higher under the judge? The team also monitors cost and latency under real load. If the new agent's p95 latency spikes under morning email volume, that's caught before users experience it.
An agent in shadow mode calls tools, so it needs care. The new agent's read tools hit real data, but its write tools (send, archive, calendar changes) are replaced by recorders that log what it would have done. Without that, shadow mode sends real emails. Shadow mode also roughly doubles your model bill for the period, so many teams shadow a sample of traffic rather than all of it.
The comparison logic must not rely on exact string matching. Two different drafts can both be correct. Instead, the judge scores each output against the email context, and the team looks at the distribution of score differences. If the new agent consistently scores higher, the confidence to promote it grows. If scores are tied but the new agent is cheaper, that's also a win.
How do you structure the evaluation pipeline end to end?
The pipeline progresses through five stages: unit, integration, regression, shadow, and production monitoring. Each stage adds realism and cost, and filters the change before it reaches the next.
graph TD
A[Unit tests] --> B[Integration baseline]
B --> C[Regression check on every change]
C --> D[Shadow mode]
D --> E[Production monitoring]- Unit stage: each tool and isolated agent function (extract date, classify urgency) runs against a small set of handcrafted examples. This catches obvious logic errors immediately.
- Integration stage: the full Mailmind agent runs once on the golden dataset, and the LLM judge scores its outputs and trajectories. This establishes the baseline score for each category and answers the question "Does the whole system work on known-good data?"
- Regression stage: on every pull request, the same suite runs again and is compared against that baseline. If any category drops, the change is blocked.
- Shadow stage: the new version runs live without user impact, with its write tools recorded rather than executed. Metrics are compared against the production baseline.
- Production monitoring: after promotion, live dashboards track accuracy, latency, cost, guardrail triggers, and human-review rate. Alerts fire when metrics drift.
Each stage is a gate. If the regression suite shows a drop in recall for an important category, the change does not advance to shadow. If shadow mode reveals a latency regression, promotion halts until it's fixed.
What tools do this for you?
You do not have to build all of this yourself. As of October 2026, several platforms record traces and run evaluations over them. LangSmith, from the LangChain team, traces runs from many frameworks and turns them into evaluation datasets. Langfuse is an open-source platform for tracing, evaluation and prompt management, part of ClickHouse since January 2026 (the announcement, ClickHouse welcomes Langfuse, says its core stays MIT-licensed). Arize Phoenix is a platform you can host yourself, with tracing built on OpenTelemetry and evaluations. Braintrust is a hosted platform for experiments, datasets and tracing. Promptfoo is an open-source tool for running test suites and attack simulations against prompts and agents. The ideas in this article carry over to whichever one you pick.
| Property | Mailmind's value |
|---|---|
| Golden dataset size per category | A few hundred examples, more where small differences matter |
| Judge validation | Agrees with humans about as often as humans agree with each other, checked per label |
| Trajectory checks | Expected tools, forbidden tools, arguments, step count, completion |
| Shadow mode window | About a week, so every weekday and a weekend appear |
| Key production metrics | Completion rate, p95 latency, cost per task, guardrail trigger rate |
| Per-category tracking | Precision, recall, F1, human-review rate per email intent |
| Regression suite runtime target | Under 5 minutes for pull request feedback |
Frequently Asked Questions
Q: How do you know when your golden dataset is large enough?
When the confidence interval on each category's score is narrower than the smallest change you need to detect. If you need to notice a 3-point drop in refund recall, a category with 100 examples (an interval of about 12 points around 90%) is too small; one with 400 (about 6 points wide) is closer, and comparing two runs needs more still. Check each category, not just the total.
Q: Can LLM-as-judge replace human evaluation entirely?
No. Even with high agreement, the judge can develop systemic blind spots (for example, over-prioritizing politeness). Humans must audit production samples and recalibrate the judge regularly. The judge handles scale; humans handle calibration.
Q: What is the biggest mistake teams make with shadow mode?
They assume identical outputs mean correct outputs. The new and old agent may agree on a wrong answer. Always compare outputs against ground truth scoring, not just each other. The second biggest is forgetting to stub the write tools.
Q: How do you prevent metric drift after deployment?
Track per-category precision, recall, and human-review rates on a dashboard. Set thresholds that alert when the review rate rises week over week, signaling a slow degradation that aggregate accuracy hides.
Q: When does it make sense to skip shadow mode?
Only for low-risk changes to non-decision-making components, like a caching layer update. Any change that alters the agent's reasoning, tool calls, or output surface should run through shadow mode. The cost of a bad output hitting a real inbox is higher than a week of shadow time.
Test yourself
After a prompt update designed to improve recall of refund requests, Mailmind users complain that their routine newsletters are now being flagged as urgent. Your dashboard shows overall accuracy at 96%. The golden dataset reports 97% precision and recall. No regression was caught. Walk through what likely happened and how you fix it.
Answer: The golden dataset likely under-represented routine newsletters, and the overall numbers hid the damage. A prompt change that boosted refund sensitivity probably made the classifier react to anything with a dollar amount, catching newsletters that mention "free shipping worth $10". The fix has two phases. First, roll back the prompt update to stop the user pain immediately. Second, expand the evaluation surface: add a false-positive rate per category, and build a set of annotated routine emails that include dollar amounts, large enough to measure that category's precision (a few hundred, as with any category that matters). Re-run the old and new versions against this expanded set to isolate the exact few-shot example or instruction that caused the over-sensitivity. Then tune the prompt to keep the refund recall gain while restoring routine precision. Before redeploying, confirm that the regression suite, now gated per category, shows both metrics within acceptable thresholds.
Next: Debugging Agent Systems: Traces, Replays, Root Causes. It covers how to find out why an agent failed once evaluation tells you that it did.
If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.
Originally published at Internals Decoded. AI internals, explained conversationally.
Top comments (0)