DEV Community

Aamer Mihaysi
Aamer Mihaysi

Posted on

Most Evals Measure the Wrong Thing

Reading through the awesome-evals list on GitHub yesterday, I realized something I've been circling for months: most of us are benchmarking wrong.

Not "wrong" as in bad numbers. Wrong as in measuring the wrong thing.

The list is a curated library of papers, tools, and benchmarks for evaluating AI agents — no fluff, no vendor pitches. What struck me is how many evals test whether a model can do something in isolation, and how few test whether it will do the right thing when the environment is messy, the instructions are ambiguous, and the tool call fails three times in a row.

That's the gap. Lab evals measure capability. Production evals measure reliability.

I've been running a small self-hosted agent stack for a few months now, and the bugs that actually bite me are never "the model couldn't answer this question." They're "the model tried to call a tool with a malformed argument because the previous step returned something unexpected." Or "the model got stuck in a loop because it didn't recognize the error message."

Standard benchmarks don't catch this stuff. They hand the model a clean prompt, a single turn, and a clear success criterion. Real agents get a context window full of history, a tool that sometimes returns 404, and a user who says "do the thing" without specifying which thing.

The awesome-evals repo points to a few resources that get this right — papers on agentic evaluation frameworks, benchmarks that inject noise and ambiguity, tools for tracing and replaying failures. Worth a read if you're shipping agents to anything resembling production.

What I'd really like to see: an eval that scores models on how gracefully they degrade. Not accuracy at the happy path, but how many retries before they give up, whether they ask for clarification when stuck, whether they notice when a tool returns garbage. That's the metric that matters when your agent runs unattended.

Until someone builds that, I'm adding a "failure replay" step to my own eval pipeline: capture real failures from prod, inject them into a test harness, and see if the next version handles them better. It's not a benchmark you can publish, but it's the one that saves you the 2 AM page.

Top comments (0)