Once the first prototype works, someone asks who is going to check all of this. The honest answer sounds like a lot of manual work: deciding what good means, pulling in domain experts, labeling outputs, reading failures, and doing it all again after every change. Many teams hope the AI will take that off their plate. Mostly it won't, and teams that skip the work end up shipping an agent nobody can vouch for.
Netflix is a useful reality check. Their team runs an LLM judge over hundreds of thousands of recommendation explanations a week, and it still started with people (Kong et al., 2026). Experts defined the criteria, wrote the labeling guidelines and handcrafted hard examples. Before launch, the benchmark held roughly 900 human-labeled explanations. After launch, about 300 freshly rated ones are added every week to catch the judge drifting.
The manual part doesn't go away. It moves from writing test cases to reviewing verdicts, and that is where you can take the volume off your experts.
The two approaches are good at different things. A domain expert can tell you an answer is wrong for your business. Automation can run thousands of conversations overnight. This post puts them side by side, shows where each one breaks, how they feed each other, and then walks through the loop in Rhesis.
The case for human evals
The context only lives in your organization. Generic metrics like helpfulness, toxicity or faithfulness tell you little about whether your travel agent got your fare rules right. Say it tells a customer they can cancel for free up to 24 hours before departure, and the policy says 48. The answer is fluent, polite and grounded in a fare document. It is also wrong, and the only people who know that sit in your operations team.
Experts are fast at spotting. Show someone from customer operations ten transcripts and they find the one with last season's baggage allowance within seconds. Hamel Husain recommends making a single principal domain expert the source of truth, with a binary pass or fail for every output and a written critique explaining why. The critique matters as much as the verdict, because it is what you later turn into a metric.
Criteria come from looking at outputs. You can't write the full rubric upfront. Shankar et al. call this criteria drift: people only work out their criteria while grading, and they keep refining them as they see more outputs. A rubric written before anyone read a transcript is a first guess.
Where it breaks: experts are slow, expensive and busy. Asking them to come up with 200 test cases by hand doesn't work, and asking them to re-read every conversation after each prompt change works even less.
Ye Yuan ran into exactly this building a customer service bot across travel, healthcare, education, legal services and more (Data Science x AI, 2026). Generic NLP metrics couldn't capture multi-turn quality, so the team built its own rubric, benchmarked against their best human sales and support conversations. The first bottleneck was annotation capacity. Training and calibrating annotators was slow and expensive, and the gold dataset stayed too small.
The case for automated evals
They can explore an agent without being told anything. Point an automated tester at an endpoint and it can probe the domain, map what the agent can do, and find where it refuses or redirects. No requirements document needed.
They run adversarial tests at scale. Jailbreaks, prompt injection and multi-turn pressure need volume and variation. Covering the OWASP Top 10 for LLMs and agentic applications by hand isn't realistic. Generating and running those attacks is exactly what automation is for.
They check requirements once you hand them over. Every run, every model swap, every prompt change, in CI. A human reviewer gets tired by the fortieth conversation. A metric doesn't.
Where it breaks: the devil lies in the details. LLM judges tend to be lenient, struggle with partially correct answers, and are sensitive to how the judging prompt is worded (Thakur et al., "Judging the Judges"). They miss the small, off-by-one-detail errors an expert catches immediately: 24 hours instead of 48. And agreement rates hide the problem. If only 5% of outputs fail, a judge that passes everything agrees with your expert 95% of the time and catches none of the failures.
Ye Yuan saw the same in production. LLM judges struggled with pragmatics, implied meaning and a user simply going quiet, with noticeable score variance between runs. Synthetic test conversations also missed the ambiguity and the emotional and topical shifts of real user traffic.
How the two feed each other
Experts are good at recognizing problems and bad at producing volume. Automation is the reverse. So the useful question isn't which one to pick, it's which part of the work goes where.
The expert's job shifts from writing test cases to reviewing and correcting verdicts. Reviewing a pass or fail with one sentence of explanation takes seconds. Writing a good test case from scratch takes minutes, and writing 200 of them doesn't happen.
Ye Yuan's team landed on a similar loop: automated evals at scale, human spot checks, manual review of every low-scoring dialogue, and failure patterns fed back into the next iteration. In practice it looks like this:
- Automation explores the agent and generates a first test set.
- An expert reviews a sample and leaves a pass or fail plus a comment.
- Where the expert and the judge disagree, those cases are used to tune the metric.
- The tuned metric runs automatically on every change.
- The expert only looks at conflicts and new kinds of failure.
You need less human input than it sounds. Anthropic suggests starting with 20 to 50 tasks drawn from real failures. Hamel Husain starts with around 30 examples and keeps going until no new failure modes show up, then aims for about 100 labeled examples per failure mode to check a judge against the expert. Netflix keeps adding around 300 ratings a week because live traffic keeps changing. That is a few hours of expert time, not a quarter of it.
Automation runs the tests, experts make the call
In the video, I point Rhesis at a travel agent with no requirements, let it explore and test the agent, and then review the results as the domain expert.
- The Architect explores the endpoint. In quick or comprehensive mode, it probes the domain, maps capabilities and finds where the agent refuses or redirects, starting from zero. The exploration docs cover both modes.
- The Architect plans and runs a suite. From what it found, it proposes requirements, metrics and test sets, and waits for approval before creating anything. I add one requirement myself: cancellation terms must match the fare rules. In the run, the cancellation test passes.
- The expert reviews the results. Annotations can sit on a whole test result, a single metric or a single turn of a conversation, each with a pass or fail and a comment. I fail the cancellation metric and write why: the fare rules say 48 hours.
- The human verdict wins. The expert's verdict now counts as the result. The automated one stays on record, and Rhesis flags the disagreement as a status conflict, so the team sees where the judge and the expert part ways. When they agree, the result shows Confirmed by with the reviewer's name, so over time you see which metrics people trust and which need work.
All annotations collect in the Annotations hub under Improve, where you can filter by conflicts and fix the metrics that keep getting it wrong. Engineers can also read and write annotations through the SDK and MCP, so expert verdicts end up in the same place as CI results.
Let experts judge, let automation do the volume
Neither approach is enough on its own. Automated evals without expert input pass fluent answers that are wrong for your business. Expert review without automation covers a few dozen conversations and stops. Put the expert where judgment is needed, and let automation handle the exploring, attacking and re-checking.
Rhesis is open source. Try it on app.rhesis.ai or self-host it from GitHub.
Top comments (0)