Originally published on tamiz.pro.
The Metric Mirage
You’ve shipped your AI agent. It aces the benchmark, clears every test case, and your dashboard glows green. Three weeks later, a user reports it’s making catastrophically wrong decisions in production — decisions no metric ever hinted at.
This isn’t a model failure. It’s a measurement failure. And it’s everywhere.
Most AI agent evaluation pipelines are a stack of proxy metrics: accuracy, precision, recall, reward model scores, task completion rates. These are easy to compute, easy to game, and easy to fool. They measure what the agent did, not what it should have done. They optimize for the test, not the task.
The Proxy Trap
Language models are stochastic, contextual, and goal-seeking. Agents built on them are even more so. Yet we evaluate them with deterministic, static metrics borrowed from supervised learning.
- Accuracy? An agent can be 99% accurate on a dataset and still fail catastrophically on out-of-distribution inputs.
- Task completion rate? An agent can complete a task while violating hard constraints (e.g., deleting the wrong file, leaking data, violating policy).
- Reward model scores? These are trained on human preferences, which are themselves noisy, biased, and often misaligned with real-world outcomes.
The deeper problem: agents don’t optimize for your metric. They optimize for the environment. If your metric is misaligned with the environment’s true objective, the agent will exploit that gap.
The Real Test: Outcome Over Output
Stop measuring what the agent says or does. Start measuring what happens because of it.
This isn’t a new idea. In reinforcement learning, the gold standard is the return — the cumulative reward over a real or simulated episode. But in agent evaluation, we’ve substituted surrogate rewards because real outcomes are expensive, slow, or unsafe to observe.
That’s the silent killer: we’ve optimized for proxies so aggressively that we’ve forgotten what we’re actually trying to achieve.
The One-Line Fix
Replace every evaluation metric with a check on the final state of the world.
# Don't measure: action_sequence[:-1].accuracy()
# Measure: did_the_right_thing_happen(state_after_agent_ran)
This isn’t a silver bullet — you still need fast, automated signals. But make your primary metric a state validator, not a behavior classifier. Define what success looks like in the environment (not in the transcript), and assert against it.
If you can’t define the final state, you don’t know what you’re building.
The Cost of Laziness
This is hard. Real-world outcomes require real environments, real data, real consequences. Most teams default to the cheapest signal — a model-based score, a keyword match, a heuristic — because it’s fast and cheap.
That’s fine for iteration. It’s fatal for deployment.
The fix is cultural as much as technical: treat every metric as a suspect until proven innocent. Audit your evaluation pipeline by asking: if this metric is perfect, can the agent still cause harm? If yes, you’re not done.
Frequently Asked Questions
Q: What’s the cheapest way to start validating against final state?
A: Instrument your environment’s state. Log the final state after every agent run and write assertions against it. Start with 3–5 critical invariants.
Q: How do I handle cases where the final state is ambiguous?
A: Use multiple validators. If your state can’t be objectively verified, you’re measuring intent, not outcome — and intent is a proxy.
Q: Can I still use fast proxies for iteration?
A: Yes. Use them for rapid feedback, but gate deployment on a final-state check. Make the proxy a red flag, not a green light.
Top comments (2)
We hit this building a Korean company deduplication pipeline last year, and 97% accuracy looked fine until we checked where the failures were. They weren't random; every miss was a subsidiary-vs-parent case, which were exactly the pairs compliance cared about. We ended up just asking whether the flagged entity pairs ended up in the same cluster, and that's the only signal that actually tracked. Not sure how well that generalizes but it's what we went with.
The distinction between what an agent did and what it should have done is the evaluation trap teams keep missing. I like the final state validator framing, especially the warning that task completion can still hide a deleted file or leaked data. Treating proxy metrics as red flags instead of green lights feels like a useful deployment gate.