Your agents are productive. Pull request volume is up, the demos land, and the pipeline is green on every merge. Here is the uncomfortable part: green means the code compiled and the tests passed, and increasingly the same agent that wrote the code wrote the tests. Nothing in that pipeline measures whether the work is actually right. "Looks right" is carrying the weight "verified right" used to carry.
The decision in front of you is whether to build an acceptance layer for agent-produced work, the way CI became the acceptance layer for human-produced code, or to keep expanding agent autonomy on top of visual inspection. Agents do not fail loudly. A model update ships, a support agent starts missing escalation cues, no error is thrown, and you find out from churn metrics weeks later. The teams that scale autonomy safely will not be the ones with the best prompts. They will be the ones with the most disciplined evals.
First, the term itself, because it carries the whole argument. An eval is a repeatable, scored test of an agent's output against criteria you define: not "did the tests pass" but "was this change correct, scoped to the ticket, and safe to merge". Where a unit test returns pass or fail on deterministic code, an eval grades judgment across dimensions, using graders that range from simple scripts to models judging other models against a written rubric. A body of these, run on every change the way CI runs on every commit, is what I will call the eval estate. Earlier in the series I argued that the agent that writes must never be the agent that checks; that settles who judges. The harder problem, and the subject of this episode, is how the judge knows what good looks like.
The Acceptance Layer Is Being Rebuilt, Industry-Wide
Anthropic's engineering team published this reframe in January 2026 in "Demystifying evals for AI agents": evaluations are not a research afterthought but the CI/CD pipeline for agentic systems. Within weeks, Braintrust made the analogy precise with eval-driven development, the agentic answer to test-driven development, with one difference that matters: TDD is binary, EDD scores across dimensions, because an answer can be factually accurate but too long, or well-formatted but missing the key information. Several vendors landed on the same operational rule in the same quarter: if the agent's metrics miss threshold on your benchmark set, the deployment fails automatically. If you're reading this I don't think think need to convince you evals matter, instead, I'll try to show you how teams actually build them, and where they go wrong.
Braintrust's worked example is the best picture of the daily rhythm: a team swaps in a newer model, the first eval run shows tone dropping from 0.85 to 0.72, they adjust the prompt, rerun, and tone recovers to 0.88 with accuracy intact. Twenty minutes end to end. Without the estate, that regression surfaces as customer complaints two weeks later.
Why Your Test Suite Cannot Do This Job
Unit tests work because the same input returns the same output. Agents break that contract: the same prompt produces different outputs across runs and model versions, errors compound across multi-turn interactions, and agents choose solution paths you never anticipated. Your test suite checks the code. Nothing checks the judgment.
There is also a failure mode that did not exist when humans wrote the code: test-gaming. A developer rarely games their own test suite. For an agent, optimizing for test passage is the default unless you deliberately architect around it.
What to Grade: Correctness Is the Easy Third
Anthropic's taxonomy gives you the mechanics:
- code-based graders for deterministic checks
- model-based graders for rubric judgments
- human graders used sparingly, mainly to calibrate the model-based ones
But the most useful answer to "grade what?" came from FrontierCode, Cognition's own benchmark built with 36 external open-source maintainers: they scored patches across six dimensions:
- behavioural correctness
- regression safety
- mechanical cleanliness
- test correctness
- scope discipline
- code quality
How much of that list your pipeline measures today ?
The Playbook: Building the Estate From What You Already Have
The standard objection is that eval sets take a mountain of manual labeling. Your organization is already sitting on the raw material. Four steps, in the order that survives production.
1. Seed the set from your PR history and postmortems. SWE-bench established the template: mine merged PRs that fixed a linked issue and touched tests, reconstruct the repo state before the fix, keep only cases where tests fail before the patch and pass after while everything else stays green. They filtered about 90,000 PRs down to 2,294 clean cases, which tells you both how much material a real repo holds and how hard the filter should be. Your review comments are a second seam; Cursor Bugbot, Qodo, Greptile, and CodeRabbit already build learned rules from exactly this data. Your incidents are the third: a postmortem is not done until one thing changes, such as adding a regression eval; otherwise it was documentation, not engineering. Pitfall: writing eval cases from imagination; you will test what you feared, not what happens. Signal: every eval case traces to a real PR, incident, or production trace.
2. Store behavioral scenarios where the agent cannot read them. StrongDM's structural fix is the cleanest I have seen: behavioral scenarios kept outside the codebase, invisible to the agent while it works, functioning like a machine-learning holdout set. The agent cannot optimize for criteria it never sees. The same applies to the judge: it must not see the maker's reasoning, or it inherits the maker's assumptions. Pitfall: putting eval criteria in the repo "for transparency," where the agent reads them and optimizes for the letter of the check. Signal: the agent's pass rate on holdout scenarios sits visibly below its pass rate on in-repo tests.
3. Calibrate the judge before you let it gate. LLM-as-judge is the scaling mechanism for all of this and the place discipline usually collapses: industry figures circulating through 2026 put the share of teams that fail to implement judges well at around 93 percent. I treat the number as directional, but the root cause is missing calibration, not weak models. I learned this the direct way. The first Reviewer agent we ran inside the AI-DLC harness at Betsson approved almost everything it saw: it answered a binary question, pass or fail, and a capable model asked a binary question about plausible-looking code says pass. The gate only started failing work when we replaced the yes/no check with scored rubrics built from what our human reviewers had actually rejected: edge cases declared covered that were not, tests weakened until they passed, changes sprawling beyond the ticket. The workflow that came out of it: write scored rubrics with each band defined in plain terms (what a 0.2 looks like, what an 0.8 looks like); keep a fixed ground-truth set of known failures; measure the judge's agreement with your human reviewers on it; do not let it gate anything below roughly 75 to 90 percent agreement. Require reasons before the score, because a model that emits the score first defends it regardless. When agreement stalls, debug the rubric, not the agent prompt. Boring, repetitive work, and the highest-leverage work we did. Pitfall: trusting a judge because it agrees with you on ten samples. Signal: a standing agreement rate against fresh human-reviewed samples.
4. Wire the gate into promotion, then watch it drift. Eval scores become promotion criteria between environments: a change that drops a metric past threshold does not ship. Then treat the estate as a living artifact. A suite passing at 100 percent is not a healthy suite, it is a dead sensor. And every model change re-ages the estate: Anthropic engineers describe the same tool-use prompt drastically under-triggering on one model version and over-triggering on the next, so the eval that graded it gave a false signal across the upgrade. Pitfall: celebrating a saturated suite as success. Signal: the gate blocks something real most weeks, and new cases are added monthly.
What This Costs, and the Kill Metric
The honest cost is senior human time defining what "done" means for your own workflows; nobody has published a credible dollar figure, and I will not invent one. Fund first: a regression suite and one calibrated judge for your single most valuable agent workflow, seeded from its PR history and incidents. Fund later: fleet-wide coverage and drift monitoring on sampled production traffic. The kill metric within a quarter: the gate must catch real regressions before production, and the judge must hold its agreement rate against fresh human samples. A gate that has failed nothing in ninety days is decoration. Fix the rubrics or stop pretending you have a gate.
This spend compounds, the same way test effectiveness beat test coverage as the metric that matters: every eval mined from a real failure keeps paying on every future run, model swap, and vendor negotiation. And the trust gap is measurable: DORA's 2025 report found only 24 percent of developers trust AI output "a lot," and LinearB's 2026 benchmarks found AI-generated PRs merging at less than half the rate of manual ones, with leaders naming context and trust, not correctness, as the barrier. Evals are how trust gets a number.
Start, Stop, Continue
Executives
Start: funding the eval estate as acceptance infrastructure with a named owner, the same way CI had one; asking for judge-human agreement rates before approving any autonomy increase.
Stop: accepting "the pipeline is green" as evidence agent work is correct; approving agent deployments gated only on tests the agent itself wrote.
Continue: holding a human accountable for every merged change, with evals deciding how much of the checking they can delegate.
Engineers
Start: mining your last quarter of rejected PRs and incidents into a seed eval set this week; requiring reasons before scores from every judge.
Stop: gating anything on an uncalibrated judge; celebrating 100 percent pass rates; scoring quality with binary checks.
Continue: reading what the agents ship. The eval estate extends your judgment; it does not replace it.
Strategic Takeaway
An eval estate compounds: every mined failure and calibrated rubric makes the next agent, the next model, and the next autonomy decision cheaper and safer. Inspection plateaus: it scales exactly as fast as senior attention, and senior attention is already your bottleneck. Anthropic Institute's own account of running at over 80 percent AI-authored production code names human review as the new constraint, and their mandatory automated reviewer, retrospectively, would have caught roughly a third of the bugs behind past incidents. The acceptance layer is where the constraint moves next. Whoever industrializes it first sets their own pace.
So run this test on your organization: if a better model shipped tomorrow, could you tell by the end of the day whether your most important agent workflow got better or worse, with a number? If the answer is no, every autonomy increase you approve is a bet placed blind. Tell me where I am wrong. If your team ships agent work confidently without an eval estate, I want to know what you are doing instead, and if the argument holds, send this to whoever owns your CI pipeline and ask them who owns the evals.
Top comments (0)