Your AI Agent Evaluation Harness Is Lying to You
Your eval suite is green and your agent is still doing something dumb in production. Both of those things can be true at the same time, and the reason is uncomfortable: AI agent evaluation that only scores the final answer is measuring the wrong thing. An agent can pass every check you have while accessing unauthorized resources, leaking private context, or triggering side effects nobody can undo. The final response looks fine, so the run gets marked successful.
Here is the part I think most teams get wrong. We ship agents to production with roughly the same evaluation rigor we would apply to a staging demo, then act surprised when the demo grade harness does not catch production grade failures. This one bit me. Below is what a harness has to look at instead, and what to start logging if you log nothing today.
Agents pass evals, then fail in production
That is not a contradiction. It is a trajectory problem.
Gartner expects over 40 percent of agentic AI projects to be canceled by the end of 2027, and 32 percent of organizations name quality as the number one deployment barrier. Those numbers are not about models being dumb. They are about teams not being able to tell a good run from a bad one.
Agent failure is usually trajectory level, not output level. A final answer can look completely acceptable while the intermediate steps show wasted cost, unsafe actions, or planning so brittle it only worked by luck. Your scorer never sees any of that, because your scorer only ever sees the last string.
Picture a support agent asked to summarize a customer's order history. It returns a correct summary. Green check. What the trace would have shown you is that it hit an expensive search endpoint eleven times because its first three queries were malformed, then pulled the record from an internal table it was never scoped to read. Correct answer. Terrible run. Your eval suite calls that a pass and moves on, and it will keep calling it a pass every night until a bill or an audit makes it someone's problem.
Why final answer only evals miss intermediate failures
Call it final answer bias. You grade one output string, so you can only ever detect defects that show up in that string.
Three categories slip straight through:
| Failure | What actually happened | Why the output looks fine |
|---|---|---|
| Unauthorized resource access | Agent queried a datastore outside its scope | The answer it produced was still correct |
| Private context leakage | Sensitive context ended up in a tool argument or downstream call | Leakage happened on the way, not in the reply |
| Irreversible side effects | Agent wrote, deleted, or dispatched something it cannot take back | The confirmation message reads perfectly |
None of those are detectable from a response string, no matter how good your judge prompt is. That is the whole point. A regression suite built only on final accuracy is not neutral, it is actively reassuring you about the exact class of failure it cannot observe. Green means "the last message looked right." It has never meant "nothing bad happened."
What a real metric framework actually covers
There is a published 12 metric evaluation framework for production agents drawn from over 100 deployments. I am not going to recite the twelve names here, because I would be reconstructing them from memory and getting one wrong helps nobody. What I can describe is the shape any serious llm agent evaluation metrics setup has to have.
Start with task outcome, since that is the one you already have. Did the agent do the thing. Keep it, just stop treating it as the whole score.
Then trajectory quality, which asks whether the path was sane. Two runs can land on identical answers and deserve wildly different grades.
Tool call correctness is the one I would add next if I could only add one. Right tool, right arguments, right order, correct handling when the call fails. Most bad trajectories are just a pile of bad tool calls wearing a trench coat.
After that: cost and token efficiency, because agents fail quietly by being expensive long before they fail loudly. Safety and permissions, which is where the three miss categories above finally become measurable. Latency, which nobody cares about until a reasoning loop goes from four steps to nineteen. And human judgment, because some qualities genuinely do not reduce to an automatic scorer, and pretending otherwise just moves the lie somewhere else.
Categories, not a checklist. Fill them in with metrics you can actually compute against your own system.
Trace based evals: what to capture
A trace is the honest version of a run. Every tool call, every argument passed, every intermediate step, every retry, every token spent, in order.
Once you have traces, a tool call audit becomes possible: replay the run and ask whether each call should have happened at all, whether the arguments were well formed, and whether anything in that call touched a resource outside the agent's scope. That is the audit your final answer scorer can never run, because it does not have the material.
This is also why the strongest harnesses stack four things rather than one. Traces tell you what happened. An eval dataset tells you what should have happened on cases you care about. Production monitoring tells you whether live behavior still matches either of those. Human feedback catches what all three miss. Accuracy alone gives you one number and no way to explain it.
If you log nothing today, here is Monday morning. Wrap your tool layer so every invocation writes a record before and after:
type ToolCallRecord = {
runId: string;
step: number;
tool: string;
args: unknown;
ok: boolean;
error?: string;
ms: number;
tokens?: number;
};
export function traced<A extends unknown[], R>(
name: string,
fn: (...args: A) => Promise<R>,
sink: (r: ToolCallRecord) => void,
) {
let step = 0;
return async (runId: string, ...args: A): Promise<R> => {
const started = Date.now();
const current = ++step;
try {
const result = await fn(...args);
sink({ runId, step: current, tool: name, args, ok: true, ms: Date.now() - started });
return result;
} catch (err) {
sink({
runId,
step: current,
tool: name,
args,
ok: false,
error: err instanceof Error ? err.message : String(err),
ms: Date.now() - started,
});
throw err;
}
};
}
That is it. One wrapper, one sink, and suddenly every run has a trajectory you can grade instead of a single string you can only trust.
The eval dataset is the bottleneck, not the harness
Teams spend weeks picking a harness and an afternoon writing the cases. It should be the other way around. Agent datasets almost never capture real production failure modes, which is exactly why a demo grade suite passes everything.
The fix is unglamorous: harvest. Every production run that went sideways, whether a user complained, a retry storm showed up in the logs, or a trace looked wrong on review, becomes a case. Freeze the inputs, record what the trajectory should have looked like, drop it into the regression suite. Do that for a month and you have an eval dataset your competitors cannot copy, because it is made of your own scar tissue.
If you want the fuller version of how these pieces fit together, I wrote up an AI agent evaluation framework with the layering in more detail, and a companion piece on LLM agent evaluation in production.
FAQ
How do you evaluate AI agents in production?
Score the trajectory, not just the reply. Combine traces of every tool call with an eval dataset built from real production failures, live monitoring of the running system, and periodic human review. Task outcome stays in the mix, it just stops being the only signal. The goal is being able to explain why a run passed, not only that it did.
What metrics should an agent eval harness measure?
Cover seven areas rather than one number: task outcome, trajectory quality, tool call correctness, cost and token efficiency, safety and permissions, latency, and human judgment. If you can only add one thing to an existing accuracy check, add tool call correctness. Most bad trajectories are a sequence of bad tool calls, and that metric surfaces them immediately.
Why do agents pass evals but fail in production?
Because evals grade the last message and production grades everything else. An agent can reach a correct answer through an expensive, unsafe, or barely working path, and a final answer scorer has no way to see any of it. Add the fact that eval datasets rarely contain real production failures, and a green suite tells you very little.
Three things to verify right now
- Open your eval config and check whether any assertion reads something other than the final output. If not, that is your gap.
- Pick yesterday's most expensive agent run and count its tool calls. If you cannot count them, you have no trace.
- Look at your last five production incidents. Count how many exist as cases in your regression suite.
If you want a deeper look at layering traces, datasets, and monitoring together, I cover it in more detail on my site.
If you want this wired up on your own site end to end, that is exactly the kind of work I take on.
Drop a comment if your setup looks different. Curious what variations people are running in production.
Top comments (1)
"Made of your own scar tissue" is the paragraph this post exists for, and I can confirm it from the other side - we've started punching runnable checks directly out of our incident history, and the harvested cases catch things invented cases never would. But practice taught us the discipline your harvest step is missing, and it's the same disease you diagnose one level up: a harvested case must prove it can fail. Before a case counts, it passes three gates - red on the untouched state, green on the fix, red again when the original mistake is deliberately re-planted. A regression case that can't go red is a green generator wearing evidence's clothes. Two additions to your "verify right now" list, from the same logic: fourth check - when did your harness last say no? A suite that hasn't refused anything in months is indistinguishable from one that stopped looking. And fifth, the one that bites later: case rot. The day the bug is fixed, the harvested case passes forever - does anything in your setup re-verify that old cases still discriminate, or does the scar tissue quietly heal into decoration? Also: separating harvested from invented cases in reporting matters more than it looks - harvested ones share a distribution with the system that failed, and if they're systematically easier, that's a finding, not a win.