What if your best-looking AI agent evaluation transcript is the warning sign your product is broken?
That was the uncomfortable point raised at VB Transform 2026, where Harrison Chase, CEO of LangChain; Hui Zhang, CTO and co-founder of Conviva; and Emmanuel Turlay, director of engineering at CoreWeave, argued that enterprises are measuring agents too narrowly, according to VentureBeat.
Their shared message was blunt: scoring one conversation in isolation can create false confidence. A chat can look clean. The agent can ask sensible questions. The user can complete the task. Yet the product can still be failing a category of users at scale.
For XOOMAR readers tracking how AI products are being constrained, approved and trusted in real deployments, this sits alongside our coverage of Approval Fail Sinks ChatGPT Work, Claude Cowork Wins and Beijing Forces Apple Intelligence China Into AI Trade-Off. The shared theme is not model hype. It's operational control.
Why can a flawless AI agent chat still signal a broken enterprise product?
A single transcript answers the wrong question.
It tells a team whether one AI agent interaction looked acceptable. It does not tell the team whether a class of users is being forced through extra friction, pushed out of the conversation, or handled differently from a baseline.
That is the shift Chase, Zhang and Turlay described: enterprise teams are moving from judging isolated traces toward comparing user cohorts against expected behavior. The method Zhang highlighted, contrastive analysis, looks for abnormal patterns across categories, users or tasks rather than grading one chat as pass or fail.
This matters because agents are no longer just demo interfaces. In the examples discussed, they touch shopping flows and higher-stakes domains such as legal, finance and healthcare. If the evaluation layer misses the real failure pattern, the company may ship a system that looks polished in review but performs poorly where it counts.
Zhang framed the trade-off sharply:
"You have scalable but ungrounded, whether it's agents as judge or LLMs as judge, you grade the outcome, you grade the work. It still is very difficult to ground it and then you use humans and that's just not scalable," Zhang said. "The whole industry is facing this, which poison you want to pick."
That is the core tension in AI agent evaluation: automation scales, but may miss context. Human review grounds the judgment, but cannot inspect everything.
How did AI agent evaluations become the new product requirements document?
Chase's argument was that evaluation criteria are becoming the product spec itself.
Not a checklist at the end. Not a one-time quality gate. A living statement of what the agent should and should not do.
"Evals are like the new PRD," Chase said. "They define what your agent should and shouldn't do."
That changes how teams should build. Chase warned against what he called eval paralysis, where teams keep expanding pre-launch evaluation sets because they do not feel ready to ship.
"We sometimes see teams that have almost eval paralysis," Chase said. "They're like, this is an eval set, I can't launch it. The best teams launch and then iterate."
Turlay described the same failure mode through conventional software testing. He said he had tried to reach 100% coverage for tests and still had bugs in production. The lesson is not that tests are useless. It's that coverage can measure the wrong thing.
The workflow Turlay described is more practical:
- Monitor broadly: Put wide online checks around live behavior.
- Find real failure classes: Let production patterns show where agents actually break.
- Build targeted offline evals: Turn observed failures into focused tests.
- Iterate: Treat evals as a product system, not a launch ceremony.
XOOMAR analysis: this is a more disciplined way to avoid fake certainty. A perfect pre-launch suite can become theater if it doesn't reflect what users actually do.
Why does scoring AI agent traces one by one miss the most useful failure signals?
Trace-level scoring asks: did this conversation look good?
Cohort-level analysis asks: is this category behaving abnormally compared with baseline?
That distinction is the heart of Zhang's critique. He said many teams sample traces, whether 50 or a full population, then score each one by itself. Even if every trace receives a decent grade, the product can still be generating a pattern of unnecessary work.
The missing layer is comparison. Zhang's contrastive analysis compares groups against a baseline to detect deviations that are invisible inside any single transcript.
| Evaluation approach | What it sees | What it can miss |
|---|---|---|
| Single trace scoring | Whether one conversation appears acceptable | Pattern-level friction across users |
| Full-population trace scoring | More individual judgments | Baseline deviations by category or task |
| Contrastive analysis | Cohort behavior versus baseline | Still needs broader context beyond the chat |
| Human review | Grounded judgment on specific cases | Scale |
Zhang also said the industry lacks a second data source: what happens before, between and after the conversation. The transcript is only the visible slice. Product failure may sit outside the chat window.
That makes AI agent evaluation less like grading an answer and more like diagnosing a workflow.
How would contrastive analysis expose a broken running shoe shopping agent?
Zhang's retail example is simple, which is why it lands.
A shopper asks an agent for a running shoe before a half marathon. The agent asks qualifying questions. The shopper buys a shoe.
If a reviewer scores that interaction alone, it looks successful. The agent responded. It narrowed the options. The purchase happened.
But across the full user population for that shoe category, the hidden signals changed the interpretation:
- Clarification ratio: The number of follow-up questions the agent asked before completing the task was three times higher than baseline.
- Off-conversation purchase completion: The rate at which shoppers finished buying outside the conversation was five times higher than baseline.
Neither signal appears in a single transcript.
Together, they point to a category-specific defect. The agent may be making shoppers work too hard, or failing to close the transaction inside the conversational flow. The important point is not the exact cause. The important point is that the cause becomes debuggable only after the team compares behavior against baseline.
That is the practical value of contrastive analysis: it converts vague dissatisfaction into a specific product question.
How should companies choose between LLM judges, agent judges, smaller models and regex guardrails?
The panel did not say agent-as-judge has replaced LLM-as-judge. Chase said LLM-as-judge remains the default.
The more useful question is sizing. What should judge which task?
The safest answer from the discussion is not a fixed recipe, but a validation sequence. Teams should first decide what failure they are trying to detect, then prove that the judge can reliably identify that failure against examples humans trust. Only after that should they optimize for speed, cost or simplicity.
For ambiguous judgments, a stronger automated judge or human review may still be needed. For narrower, repeatable checks, companies can consider lighter models or deterministic guardrails, but only if those checks are tested against real production failures rather than assumed to work because they are cheaper or easier to run.
That distinction matters because a judge is itself part of the product system. If it rewards the wrong behavior, the agent may learn to look better without actually serving users better. If it is too broad, it may be expensive and inconsistent. If it is too narrow, it may miss the failure class the business actually cares about.
A practical selection process would ask:
- Is the failure subjective or objective? Subjective issues need more grounded review.
- Is the judgment about one response or a pattern across users? Pattern failures need cohort analysis.
- Can the check be validated against labeled examples? If not, confidence should stay low.
- Does the control reduce risk without hiding new failures? Cheaper checks are useful only when they preserve signal.
That is a useful corrective. Companies should not pay for sophistication where a simpler validated control is enough, but they also should not confuse a simple control with a complete evaluation strategy.
When can enterprises trust automated AI judges without removing human accountability?
They can't fully remove humans yet, at least not in the scenarios the panel discussed.
The source material supports a cautious view: automated judges can expand what teams are able to monitor, but they do not automatically transfer responsibility away from people. They can help detect patterns, prioritize review and expose failures that would be hard to catch transcript by transcript. They are not, by themselves, a substitute for accountable decision-making.
That is especially true when the consequences of an agent's action are meaningful for the user or the business. In those cases, the evaluation system should make human review more focused, not pretend that human judgment has disappeared. Automation can narrow the search space. It can flag abnormal cohorts. It can turn messy production behavior into clearer product questions. But someone still has to decide what level of risk is acceptable and what standard the agent must meet before it is trusted.
This is also where the panel's broader argument about evals as product infrastructure becomes important. If evaluations define what the agent should and should not do, then humans remain responsible for writing, updating and challenging those definitions. The machine may score the behavior, but the organization owns the target.
The practical takeaway is narrow but important. Companies should not treat AI agent evaluation as a single score, a single judge, or a single pre-launch hurdle. They should treat it as an operating system for product learning: broad monitoring first, cohort comparison against baselines, task-appropriate automated checks where they work, simpler controls where they are validated, and human accountability where the decision still needs a person to own it.
Impact Analysis
- Enterprises may overtrust AI agents if they only review polished individual conversations.
- Cohort-level evaluation can reveal hidden failures affecting specific user groups or task types.
- Operational control is becoming as important as model performance in real AI deployments.
Originally published on XOOMAR. For more news and analysis, visit XOOMAR.
Top comments (0)