How to evaluate an LLM agent: evals, golden sets, and LLM-as-judge

Wibo — Sat, 01 Aug 2026 13:59:25 +0000

Short answer

You can't unit-test an LLM to correctness, because the same input can take a different path on the next run. Evals are the test suite for probabilistic systems: a scored, repeatable check of whether the system reached an acceptable outcome across runs, not whether it returned one exact string once.

The core toolkit is small: a curated golden set of input-to-expected-outcome examples, an offline run of that set in CI plus online checks against live traffic, and the right scorer for each task — assertion, golden set, LLM-as-judge, or human review. Wire it into CI as a regression gate and a quality drop fails the build.

Why evals, not unit tests

A unit test asserts that a deterministic function returns one exact value. An LLM is not deterministic: temperature, model updates, and the model's own sampling mean the same prompt can produce different — and differently worded — outputs across runs. Pinning to a single expected string makes a test that either flakes or asserts nothing useful.

Evals replace that assertion with a different question: did the system reach an acceptable outcome, across multiple runs, at an acceptable rate? That reframes testing as a measurement problem. You score outputs against a target, aggregate over a set, and track the score over time. It is closer to an SLO with an error budget than to a pass/fail unit test, and it is the discipline that lets you change a prompt or swap a model without flying blind.

Golden sets

A golden set (also called a reference or eval set) is a curated collection of input-to-expected-outcome examples. Each row pairs an input the system will actually see with the outcome you consider correct or acceptable — sometimes an exact answer, more often a rubric or a set of properties the answer must satisfy.

The most valuable golden sets are not invented up front; they are grown from real failures. Every production bug, every escaped edge case, every "the model did something weird here" becomes a new row. Over time the set encodes the actual shape of your traffic and your hardest cases, so a passing eval run means something concrete. Keep the set version-controlled, keep it representative rather than merely large, and treat adding a case after an incident as part of the fix.

Offline vs online

Offline evaluation runs your system against a fixed golden set, usually in CI, before anything ships. It is repeatable, comparable across changes, and cheap to run on every commit. Its limit is that it only measures the cases you thought to include.

Online evaluation measures behavior against live traffic and real outcomes after release — sampling production runs, scoring them, and watching real signals such as task completion, user corrections, or downstream success. It catches the distribution shift and the inputs your golden set never anticipated. Offline tells you a change is safe to ship; online tells you it actually worked. You need both, and online failures are the best source of new offline cases.

LLM-as-judge (and its pitfalls)

For subjective qualities — is this summary faithful, is this tone right, did the answer address the question — there is often no exact string to match against. LLM-as-judge uses a model to score the output against a rubric. It scales grading that would otherwise need a human on every row, and for many tasks it correlates well enough with human judgment to be useful.

It is also a component with real failure modes, and you should treat the judge as something you must itself evaluate:

Position bias: in pairwise comparisons the judge can favor whichever answer it sees first, regardless of quality.
Verbosity bias: longer, more confident-sounding answers tend to score higher even when they are not better.
Inconsistency: the judge is itself probabilistic, so the same pair can score differently across runs.
Gameable: outputs can be optimized to please the judge rather than the user, and a self-judging model can flatter its own style.

The mitigation is to validate the judge against a sample of human-labeled data before you trust it, measure how well its scores agree with those labels, and re-check that agreement when you change the judge model or its rubric. A judge you have not validated is an unscored assumption, not a measurement.

Evals as regression gates

An eval suite earns its keep when it runs automatically. Wire the offline golden-set run into CI so a change that regresses quality below a threshold fails the build, the same way a failing unit test or a dropped coverage number blocks a merge. That turns "we think this prompt change is better" into a measured claim, and it stops silent quality regressions from shipping when someone edits a prompt, upgrades a model, or refactors the harness.

One discipline matters more than the rest: pick the metric that reflects task success or outcome correctness, not just string similarity. Two answers can be worded completely differently and both be right; an exact-match score would fail a correct answer and a similarity score can pass a fluent wrong one. Score what you actually care about — did it do the job — and let the gate enforce that.

Choosing an eval approach

Eval approach	What it measures	Use when
Assertion / code checks	Hard, objective properties: valid JSON, required fields present, value in range, no banned content	The output has a checkable structure or invariant; cheapest and most reliable, so reach for it first
Golden set	Outcome correctness against curated input-to-expected examples, aggregated across the set	You have known-good answers or rubrics and want a repeatable score you can gate CI on
LLM-as-judge	Subjective quality — faithfulness, tone, relevance — scored by a model against a rubric	There is no exact answer to match and you need to scale grading; only after validating the judge against human labels
Human review	Ground truth and nuanced judgment; the reference everything else is calibrated against	Stakes are high, the task is ambiguous, or you are validating a judge or building the golden set

Evals before scale
Build the eval harness before you scale usage, not after. Without it you have no way to tell whether a prompt change, a model upgrade, or a new tool made things better or worse — you are shipping on vibes. The team that can answer "did quality go up or down, and by how much" is the team that can iterate safely; the one that can't will eventually regress in production and not know why. Eval design is also the single strongest signal in hiring that a person has actually shipped with LLMs.

As-built: how we evaluate the coach on aiarch.dev

This page has just recommended wiring evals into CI and validating your judge against human labels. We do neither. Here is what actually gates a coach change on aiarch.dev, including both places it falls short of the advice above.

Two tiers. npm run eval scores the golden set against a mocked model — cheap, local, run on every coach change. Above it sits a live gate: real calls, scored for the behaviours that matter here (fading hints instead of revealing answers, naming the misconception, escalating after two stalled turns), and it has to clear 80% before a coach prompt or a deploy ships. It runs up to three times and takes the majority, exiting early once two runs pass. The coach is probabilistic even where the scoring isn't, so a verdict read off a single sample is noise.

There is no CI. No .github/workflows/ in the repo, and git push deploys nothing — shipping is a deliberate wrangler deploy, with the gates as commands a human runs first. That is a gap, not a design choice. It is also why the offline tier has to stay fast: a gate you run beats a gate you describe.

The second gap is sharper. The live gate scores every reply against a rubric. No model reads it.

The scoring is deterministic assertions — eleven regexes over the coach's text, four that check which tools the turn actually called (did it fetch the lesson context, did it grade something the learner never attempted), two length floors, a stop_reason check. The harness is candid about the limits of that in its own comments: two of the checks are annotated as "necessary-but-not-sufficient evidence", naming a human or an LLM judge as the thing that would confirm real tutoring quality. So 80% measures observable adherence, against a threshold we chose rather than one calibrated against human labels. It catches regressions. It does not certify quality, and reading it as though it did is the mistake the section above warns about. See the eval-gate pattern for how it slots into the release pipeline.

Sources & provenance

Course material: aiArch Track B (eval design) — golden sets, offline vs online, judges, and CI gates.
Eval design guidance: Anthropic's docs on building and evaluating tool-using agents.
Anthropic — platform docs on evaluation (designing and running evals).
The limitations of LLM-as-judge (position bias, verbosity bias, inconsistency, gameability, and the need to validate against human labels) are widely documented in the evaluation literature.

This is a conceptual overview; no specific benchmarks or figures are claimed, and API shapes change — verify against current provider docs before implementing. Corrections: hello@aiarch.dev.

Originally published at aiarch.dev/llm-evaluation-guide, where it is kept up to date.

Want the skeleton instead of the essay? aiarch-templates has the src/lib/ seams, a threshold-gated eval stub and a cost-model skeleton. It is deliberately empty — it fixes the shape and you write the implementation. Apache-2.0.

DEV Community: Wibo