Your agent's 99% eval score is measured under conditions that never occur in production

Taoheed — Mon, 03 Aug 2026 23:21:08 +0000

Ran the same agent through the same eval harness twice this week. Only variable: whether the network behaved like a demo or like production.

Clean conditions, no faults: 50/50 passed. 100%.

Same agent, same code, one change: 22% of calls now get hit with the kind of rate limit any LLM provider throws under real load. 36/50 passed. 72%.

That's not a bug in the agent. It's the eval never testing for it in the first place.

Most eval harnesses run against a provider that never drops a request, never times out, never sends back a malformed response. So the score is real, it's just measuring a condition your agent will never actually run in. Production has retries that don't fire right, timeouts that cascade, rate limits that arrive mid-conversation. None of that shows up in a single clean pass.

The fix isn't a better eval score. It's running the eval under the failure modes production actually has, and reporting the number that survives contact with them.

Building thaghr to do exactly this: inject the faults, run the trials, report what's left. More on that as it's ready.

DEV Community: Taoheed

Your agent's 99% eval score is measured under conditions that never occur in production