Without evals, every prompt change is a guess. You fix the example in front of you and quietly break three you aren't looking at. An eval set turns that guess into a number you can watch.
Start from real failures
The best eval cases come from real inputs, especially the ones that went wrong. Collect 50 to 200 of them, write down what a correct outcome looks like for each, and tag them by type: language, task, customer segment, whatever splits your traffic. An average score hides a feature that's broken for one slice.
Synthetic cases are useful for coverage, but they tend to be cleaner than real users. Keep real ones as the core.
Check with code first
A lot of quality is checkable without another model. Is the output valid JSON with the required fields? Did the agent call the right tool with the right arguments? Does every citation point to a document that was actually retrieved? Is the answer under the length limit? These checks are fast, free and give the same result every run. Write them before reaching for a judge.
LLM-as-a-judge, carefully
For open-ended qualities like helpfulness or tone, a model grading another model's output is often the only practical option. It has known biases, documented in the MT-Bench paper by Zheng et al. (2023):
- Position bias: when comparing two answers, judges tend to favour one position regardless of content.
- Verbosity bias: longer answers look better to judges even when they aren't.
- Self-enhancement bias: a judge can favour answers written by its own model.
What helps: a rubric with specific, checkable criteria instead of "rate quality from 1 to 10". Pass/fail or a three-point scale rather than fine-grained scores. Asking for the reasoning before the verdict. In pairwise comparisons, running both orders and only counting consistent wins.
Grade the answer against the reference.
Criteria, each PASS or FAIL:
1. States the refund window that the reference states.
2. Does not promise anything the reference does not support.
3. Tells the user the next step to take.
Give one line of reasoning per criterion, then the verdicts.
Then calibrate. Grade 50 to 100 cases by hand, run the judge on the same cases and measure how often it agrees with you. If agreement is poor, fix the rubric before you trust a single number it produces.
For agents, grade the trajectory too
An agent can reach the right answer the wrong way: ten tool calls where two would do, a write it shouldn't have made, a lucky guess after a failed lookup. Score the path as well as the result. Did it call the expected tools? How many steps did it take? Did it stop when it should? What did the run cost and how long did it take?
Run evals like tests
- Run the suite on every prompt change, model change and tool change, and compare against the last known good run.
- Model output varies between runs. Run each case several times and track the pass rate, not a single pass or fail.
- Watch the slices, not only the total.
- Every production failure becomes a new case. The set should grow every week.
An eval set doesn't need to be big or clever on day one. Twenty real cases with honest checks will catch more regressions than a polished benchmark nobody runs.
Originally published at anukumar.site.
Top comments (0)