Two weeks ago, I published "Why Agent Evaluation Is Harder Than Model Evaluation." The core argument: with agents, you are not just judging an answ...
For further actions, you may consider blocking this person and/or reporting abuse
The field-test result is the useful warning here. If the 9 percent pass rate mostly measures adapter realism, the eval harness is already telling you something before it ranks agents: your test surface is still too polite. I would probably make "can this agent be invoked outside its author's environment?" a first-class score instead of treating it as setup noise.
Exactly right, and honestly, I was a bit naive and optimistic, the tool will score higher pass rate for higher starred agents and maybe successful in catching issues in lower popular ones - this seemed like a good hypothesis. And, at the same I had pivoted more towards putting this in CI pipeline so I can push the eval more to the left. With the lower pass rate, and adapter realism and the extra shims I had to build, I am considering exactly as you said - will agent eval be challenging if folks use outside my intended environment or do I consider making this part of a common agent development environment? I am definitely not aiming to be a SWE Bench analog to come baseline agents that you have already invested in eval-ing other ways. Anyway, I have to think through these and solidify my use cases as to what this should be and what it should not be. Setup noise turned out to be very high.
nice work
Hands down one of the most honest and insightful posts on AI Agent evaluation I've read on DEV! 👏 The 9% pass rate finding is a huge eye-opener—it completely shifts the conversation from 'model intelligence' to 'software environment and integration reality.'
To answer your question: currently, we are mostly relying on LLM-as-a-judge for final outputs, but trajectory/tool-call failures are definitely our biggest blind spot. Sticking strictly to the 'Safety > Correctness > Efficiency' hierarchy in your PRD is such a solid architectural choice. Looking forward to part 4 of the series!
The field-testing discovery is the part that matters most. Searching 150+ repos and landing on 19 testable agents, with most being "too heavy, too coupled, or no clear entry point" — that is not a sampling problem, it is an ecosystem problem. Most agents are not eval-ready because they were not built to be observed.
The adapter contract you described — scenario input, allowed tools, disallowed tools, budget, never seeing the expected answer or scoring thresholds — is the right isolation boundary. No ground-truth leakage is the minimum bar and most eval frameworks do not meet it. The five adapter surfaces (subprocess, Python import, HTTP, LangGraph, PydanticAI) also tell you something about the fragmentation in the agent space: there is no standard invocation contract, so the harness has to bridge every surface.
The 17 deterministic scorers vs 11 LLM-as-judge split is interesting. In my own work on verification gates, the deterministic checks caught the structural failures (missing tool calls, wrong artifact paths) while the LLM judge caught the semantic failures (right tool, wrong intent). The ones that broke the clean version of the story were usually the cases where the deterministic scorer said pass but the LLM judge said the agent solved a different problem than the one posed.
Curious about the adversarial generator — does it produce perturbations of existing scenarios or generate entirely new failure modes? The distinction matters because generated adversarials tend to cluster around known failure types while real-world agents fail in ways nobody thought to test for.
A robust evaluation framework should measure more than task completion. Observing how an agent responds to failures, ambiguous inputs, and changing conditions provides a far better indication of production readiness than success rates on controlled examples alone.
The 9/95 result makes adapter realism look like a first-class evaluation dimension, not setup noise. I would split the CI result into at least three gates: invocation compatibility (import, environment, tool contract), policy/safety compliance, and task quality. Otherwise a blank completion or import-time side effect can collapse into the same score as a genuinely incorrect trajectory.
For the field layer, a useful regression fixture might be a deliberately awkward agent: nested project root, delayed tool response, unavailable optional dependency, and a restart after partial output. If the adapter can resume and preserve the failure classification, the score becomes actionable. The current repo caveat that compatibility is not yet ranking quality is exactly the distinction I would keep visible in CI.
Are all 19 agents tested on identical hardware/OS? You mention local MLX on Apple Silicon, but what about CI runners (Linux/Docker)? Do you publish detailed reports per agent (environment vars, dependency versions, adapter logs)? Can readers reproduce or rerun evals on different machines/clouds to verify the 9% baseline? Raw test reports are linked, but how granular is the failure data—can someone tell if their agent fails due to adapter quirk vs. real incompatibility?
Hello there,
I hope you're doing well.
I have a good business idea that I'd love to discuss with you in more detail.
To give you some background, a friend of mine started this business with a U.S.-based partner three years ago. Since then, he's been paying his partner between $8,000 and $10,000 per month, and the business has been working well.
If you're interested in learning more, I'd be happy to share the details.
This is my whatsapp number: +81 70-9427-3751
Best regards,
Ono
The 9% pass rate measuring adapter realism instead of agent quality is the key finding here, it would've been easy to publish that number as "most agents are weak," but correctly diagnosing a blank completion as a compatibility failure, not a quality failure, is the harder and more honest read.
The cheap-vs-better tier result reframes eval work generally: if the expensive judge produces identical outcomes to the cheap one, the bottleneck was never the judge, it was whether the harness faithfully exercised what it was judging.
"Testing inside the author's own environment against the author's own examples is the worst possible way to evaluate whether something actually works" is the line that generalizes furthest past this project.
Curious about the blank-completion PydanticAI wrappers, is the current guess that the adapter's invoking correctly but the agent's own output path silently short-circuits, or is it still unclear where the failure originate
The honesty in this write-up is refreshing. Hitting a 9% pass rate because the field test mostly measured "adapter realism and environment setup" rather than agent intelligence is the exact wall everyone runs into when trying to evaluate third-party repos.
The point about config chaos—module-scope API calls, hardcoded model imports, and transitive dependencies like ormsgpack—highlights why standard CI/CD paradigms break when applied to AI agents. Making "Safety > Correctness > Efficiency" a non-negotiable hierarchy in the PRD was a great architectural choice. Until we standardize agent interfaces and execution boundaries, harness builders are going to spend 80% of their time writing compatibility shims rather than scoring logic. Congrats on getting agent-eval-forge onto PyPI!
To answer your question at the end: what breaks first when trying to evaluate third-party agents is almost always side-effects and environment assumptions. The moment an agent assumes a specific directory structure, a local database connection, or module-scoped model initialization, standard test runners fall apart.
The decision to build dynamic adapters for LangGraph and PydanticAI while using [unk] fallback patterns and local MLX models for rapid sweeps makes agent-eval-forge super pragmatic. Looking forward to part 4 of the series to see how you evolve trajectory scoring!