If you have ever shipped an AI feature and then crossed your fingers hoping it works, this post is for you. An eval is a structured test that measures how well your AI system performs: its quality, reliability, and correctness across a range of scenarios. Think of it as the unit test of the LLM world, except the thing you are testing is probabilistic, moody, and changes every time a new model drops.
Why Bother with Evals?
Teams that invest in evals see four big payoffs. First, they cut development time because you get rapid iteration cycles and can test locally across multiple LLMs instead of eyeballing outputs one at a time. Second, they reduce costs because automated evals replace slow manual review and let you release faster. Third, they enhance quality because real-time monitoring and compliance checks reduce risk and improve the customer experience. Finally, they scale teams because a good eval setup lets non-technical collaborators contribute to building the best possible product experience.
Greg Brockman captured the idea in a memorable line:
The point is not that prompts, observability, or human judgment stop mattering. It is that once an AI system can produce plausible outputs, the hard part is often knowing whether your next change made it better.
The mental model has three pillars. Prompt and context engineering give you a place to prototype behavior. Evals answer the single most important question in AI development: did I just improve things, or did I regress? Depending on what you measure, a result might be a number, a pass or fail, a category, or written feedback. AI observability tells you what is actually happening in production, so when something breaks you can turn the failure into a reproducible test instead of relying on a guess.
A Useful Model for Most Evals
Most offline evals can be understood through the same three parts, even though frameworks give them different names.
The target is the code, prompt, model, or workflow you want to evaluate. Some tools call this a task, application, predict function, or system under test. It can be as small as one model call or as large as an entire agent workflow. The important part is that the same input can be run through different versions so you can compare their behavior.
The dataset is your collection of test cases. A case usually includes an input and may also include a reference answer, expected properties, a rubric, metadata, tags, or other context. The exact schema depends on the framework and the behavior you are testing. A reference is not always necessary: safety, style, latency, and many other criteria can be evaluated without one exact ideal output.
The evaluator is the logic that measures the target's behavior. Some frameworks call it a scorer, grader, metric, or judge. It can be deterministic code, a model-based judge, a human review, or a combination of them, and it can return a number, boolean, label, ranking, or structured feedback.
Offline vs. Online: Two Eval Mental Models
Two common modes are offline and online evaluation, and mature teams often use both.
Offline evals run on curated or historical test cases, usually during development, in CI, or before a release. The point is to catch regressions, compare versions, and understand tradeoffs under controlled conditions. You can run them with a local script, a test framework, a notebook, or an evaluation platform; the workflow is the same even when the interface changes.
Online evals measure selected production interactions as the system runs. They often operate on traces, logs, user feedback, or sampled traffic and focus on signals that do not require a known reference answer, such as safety, policy compliance, relevance, latency, or user satisfaction. Tracing and evaluation are related but different: tracing records what happened, while an evaluator judges some aspect of it. Interesting production cases should be reviewed and curated before they become offline test cases.
The Eval Framework: A Simple 2x2
Here is a mental framework worth memorizing. When you compare an output with an evaluator's result, there are four possibilities:
| Output quality | Evaluation result | What it means |
|---|---|---|
| Good output | Positive | Everything is working |
| Bad output | Negative | The eval caught a real problem, so improve the AI app |
| Good output | Negative | The eval is too strict or measures the wrong behavior |
| Bad output | Positive | The eval rewards a failure and needs improvement |
The interesting quadrants are the mismatches. A good output with a negative result, or a bad output with a positive result, means your evaluation criteria need improvement.
The practical takeaway is to create a baseline and start iterating immediately. Do not wait around for a perfect golden dataset that never arrives.
Target Patterns: From Model Calls to Agents
The simplest target is a single model call. Its prompt may combine instructions with variables such as the user's question, retrieved context, or conversation state. Template syntax varies across libraries, so the important contract is the data you pass in, not the braces used to represent it.
System: Answer using the supplied context.
User question: {question}
Context: {context}
Single-call targets are useful for testing instructions, examples, output formats, retrieval context, and model choices in isolation.
For multi-turn scenarios, evaluate a sequence of messages or an entire conversation. This is a good fit for measuring whether a chat experience remembers context, follows changing instructions, and stays coherent across turns.
Tool-using systems add another layer. The evaluator may need to inspect not only the final answer but also tool selection, arguments, permissions, intermediate results, cost, and latency. The tool can be implemented in any language or service; what matters is the observable contract between the model and the tool.
Agents and multi-step workflows combine model calls, tools, state, routing, and control logic. Evaluating them usually means scoring both the outcome and the trajectory: whether the system reached the right result in a safe, efficient, and reproducible way.
Datasets: Start Small, Never Stop
Datasets collect your test cases so you can run repeatable evaluations and track improvements over time. Three tips make the difference between a dataset that helps and one that rots:
- Start small and iterate. Focus on building a feedback loop rather than a perfect dataset.
- Never stop iterating. Use production logs to capture new edge cases and make your evals more holistic over time.
- Implement human review to establish ground truth, especially when you rely on an expected-output field.
Evaluators: The Spec for Your Project
There are two common families of automated evaluators.
Code-based evaluators handle anything deterministic: exact matches, numeric comparisons, structured checks, or factual checks. Something as simple as the following counts:
output === expected ? 1 : 0
They can also run in your CI pipeline. Techniques like schema validation, exact match, string-similarity measures such as Levenshtein distance, numeric tolerances, and binary checks fit here too.
Model-based judges handle work that needs semantic or subjective interpretation, such as relevance, tone, completeness, or improvement across drafts. A judge prompt can be as simple as: "Does this response contain an apology? Return PASS or FAIL and briefly explain why."
Two common judge patterns are:
- Direct assessment, where you design a rubric.
- Pairwise comparison, where the judge picks the better of two outputs and a ranking algorithm sorts overall quality. This is particularly useful for subjective tasks.
A few hard-won tips for evaluators:
- Use a judge capable enough for the criterion, and keep its configuration separate from the system being evaluated.
- Treat evaluators like real judges evaluating intent, style, and overall quality, not just correctness.
- Break evaluation into focused criteria such as accuracy, creativity, safety, and formatting so you can pinpoint exactly what broke.
- Calibrate judge prompts against representative examples with human labels before trusting them.
- Do not overload the judge with context. Keep it focused on the relevant input and output.
One of the most important ideas here is that evaluators are an executable part of the spec for your project. Writing criteria that reflect your users and failure modes is essential because generic metrics rarely capture everything your product actually needs.
Human in the Loop: When the Worst Case Matters
Not all AI mistakes are equal. In high-stakes industries like healthcare, finance, or legal tech, a single failure can mean regulatory violations or real user harm. That is why teams in these domains invest heavily in clear ground-truth definitions, use human subject-matter experts as annotators, and apply human-in-the-loop workflows to refine their scoring.
In these contexts, human review is not a nice-to-have. It is essential for catching hallucinations, establishing ground truth, and ensuring alignment with business, compliance, and user expectations. Automation misses nuance. Real people identify mistakes, label correct outputs, and make sure the final product actually meets human needs.
Human-in-the-loop work comes in two forms. Human review uses internal experts to manually label, score, or audit outputs. It is useful for building high-quality ground-truth datasets, auditing edge cases, and calibrating model-based judges. One example is legal subject-matter experts defining exactly how a legal AI assistant should respond.
User feedback is the implicit or explicit signal from end users during real usage. Thumbs up or down, flags, comments, corrections, and helpfulness ratings can trigger human review or flag production traces for dataset curation.
The formula is simple: automated testing for scale plus human judgment for nuance creates a north-star experience.
Spans: The Bridge Between Production and Testing
A trace records one end-to-end execution of your application. Within it, a span represents a unit of work such as a model call, retrieval step, tool invocation, or business-logic function. Spans can be nested to show how the smaller operations contributed to the overall result. The exact names and boundaries depend on how you instrument the application.
Why do traces matter for datasets? When your instrumentation records the relevant fields, traces can preserve the input, output, intermediate steps, and context behind real behavior. When something goes wrong, such as the AI rephrasing a user's question badly, you can save the relevant input and trace as the basis of a new test case. The failed production output is evidence of the problem, not the expected answer. Add a corrected reference, rubric, or explicit property that defines what success should look like, then use that curated case as a regression test.
The Flywheel Effect
This all comes together in a cycle worth internalizing:
- Instrument the app and run it in production.
- Collect traces, operational metrics, evaluator signals, and user feedback.
- Find important examples, such as a user thumbs-down, policy failure, or unusual trace.
- Curate those examples into test cases with a corrected reference, rubric, or expected property.
- Fix the code, redeploy, and repeat.
Every turn of the flywheel makes your evals more realistic and your product more robust.
Best Practices: Playing Offense with Evals
The biggest mindset shift is to use evals to play offense. Evals are not just a safety net for catching regressions. They are what lets you launch better products, faster and with confidence.
A few principles follow from that. Great evals need to be intentionally designed, not bolted on. Treat datasets as maintained engineering assets: record provenance, version meaningful changes, deduplicate near-copies, protect sensitive data, and keep labels consistent as the collection grows. Prompt engineering is also evolving into context engineering. System instructions are only one part of what a production model sees; conversation history, retrieved documents, tool definitions and results, memory, and policy context can dominate the input. Evaluate the whole context assembly process, not just the prompt text.
Finally, remember that everything changes when a new model comes out. Model-agnostic evals are your defense against the nondeterministic nature of LLMs. They let you swap models and release efficiently and intentionally. Optimize the entire evaluation system, not just the prompts. Once your evals are solid, you can even close the loop entirely and use them to automatically improve your prompts.
Wrapping Up
Evals turn AI development from vibes into engineering. Start with a target, representative test cases, and evaluators that express what good means for your users. Run offline evals while you build and monitor selected quality signals after you ship. Curate production failures into regression cases, keep humans in the loop where the stakes are high, and treat your evaluation criteria as a living part of the product spec. Do that, and "did I improve or regress?" stops being a guess and becomes a question you can answer with evidence.
Top comments (0)