Two weeks ago, I published "Why Agent Evaluation Is Harder Than Model Evaluation." The core argument: with agents, you are not just judging an answer. You are judging a run. The path matters. The tools matter. The safety boundaries matter. I ended it by saying I would share the repo when it was ready.
It is ready now. AgentEval Forge is public, on PyPI, and this launch is really a report on what I learned while trying to build a believable way to validate agents.
I thought I was mostly building a scoring system. Real agents turned into the tsunami I was not prepared for, and the project became an integration reality check much faster than I expected.
What I Actually Built
I did not want to build a thin wrapper around an existing eval framework and call it a launch. So I went deep.
The PRD defines 20 Critical User Journeys across launch scenarios, regression workflows, adversarial case generation, and CI gating. The spec details an architecture with five core components: a scenario pack engine, a runner, a scoring layer with 17 deterministic checks and 11 LLM-as-judge metrics, a regression engine, and an adversarial generator. The WBS breaks into 118 tasks across 12 milestones: M0 scaffold through M12 launch.
I built adapters for five agent surfaces: subprocess, Python import, HTTP, LangGraph, and PydanticAI. Each adapter implements a thin contract: the agent receives a restricted invocation payload — scenario input, allowed tools, disallowed tools, and a budget. It never sees the expected answer or the scoring thresholds. No ground-truth leakage.
I built a security model: sandbox mode, trust policies, audit trail, API key sanitization. I built CI integration: GitHub Actions, GitLab CI, Docker sandbox. I built the docs: a user guide, a scoring guide, a scenario authoring guide, and the full field test reports with raw data.
The full capability list is in the repo README. If you want the deeper product/design docs, they are in the PRD, spec, scoring guide, and scenario guide. But the headline is: 20 launch scenarios across 10 families, 8 security scenarios, 17 deterministic scorers, 11 LLM-as-judge metrics, five framework adapters, and a product hierarchy where safety failures trump everything else.
Field Testing Changed The Project
I did not want to ship a test harness that only works against examples I wrote myself. But field testing was not part of the original plan. It got added ad hoc, late in the build, because I started getting nervous that unit tests and mock agents were hiding real integration problems. They were.
Sourcing real agents from GitHub sounded straightforward. Search for "langgraph agent" and "pydantic-ai agent," pick a few, run them. It was not straightforward.
I searched 150+ repos. Most were not viable. Large frameworks and platforms were too heavy. Database-bound agents required infrastructure I could not provision per-agent. Agents with no clear entry point, no run() method, no message surface, those just sat there silently. Agents behind an API gateway needed keys I did not have. I landed on 19 that looked testable: 11 LangGraph, 8 PydanticAI. Even those needed work.
Why I Bucketed Agents By Stars
I also had to create field-test buckets so the results would mean something. High-star agents were the reliability bucket: widely used repos that should mostly test my tool more than I test them. If EvalForge cannot even integrate with those, that is my problem first. Medium-star agents were the mutual-friction bucket: mature enough to be real, messy enough that the test cuts both ways. Sometimes I would find a weakness in their packaging or bootstrap; sometimes I would find a bad assumption in my harness. Low-star agents were the stretch bucket: experimental repos where I expected more chaos, but also where EvalForge could show obvious value by making a rough agent easier to evaluate, compare, and improve.
Each agent got its own configuration file, its own scenario pack, its own virtual environment. I tested across three tiers: local, cheap, and better. Because I was worried about token costs from the beginning, I put real effort into a local tier instead of treating it like a fake demo mode. On my machine that meant MLX serving Qwen3.5-9B-MLX-4bit, which fits comfortably on my Apple Silicon setup and was good enough to make local sweeps worth doing when the endpoint was healthy. The cloud tiers were gpt-4o-mini for cheap and gpt-4o for better.
That was the moment the clean version of the story broke. I thought I was mostly building a scoring system. Real agents forced the project to become an integration reality check.
What I Focused On — And What I Learned From The PRD
The PRD was not an afterthought. It is the thing that kept me honest.
The Product DNA section forced me to answer uncomfortable questions before I wrote a line of code. Who is this for, in order? Solo OSS builders first, small teams second, platform teams third. What is the smallest meaningful adoption outcome? Catch one regression before merge. What is the evaluation hierarchy? Safety > Correctness > Efficiency.
That last one was a forcing function. It meant I could not design a scoring system where a clean answer at the end washes away a policy violation on the way. Safety failures always fail the run. Correctness regressions warn by default but do not block unless you configure them to. Efficiency regressions are informational. That hierarchy is now baked into every scorecard the system produces.
The 20 CUJs were the other forcing function. A CUJ is not a feature. It is a specific moment where the product either earns its keep or does not. "An individual developer wants to evaluate a candidate agent version against a baseline." "A team lead wants to add a new scenario pack for a domain-specific tool." "A CI pipeline needs to block a PR when safety scores drop below threshold." Every milestone had to justify itself against at least one CUJ.
That discipline slowed me down in ways I am now grateful for. It stopped me from building a general-purpose eval library that does everything adequately and nothing well.
That was one of the earliest real learnings in the project: if the product cannot help someone catch one meaningful regression before merge, then the rest of the architecture is mostly decoration.
What the field test taught me
Here is what I expected, what I did not, and what surprised me most.
What I expected
I expected some agents to struggle with specific scenario families. I expected the better-tier judge to be more discriminating than the cheap-tier judge. I expected most agents to pass most scenarios, with the interesting signal being which ones failed and why.
None of those turned out to be true.
What I did not expect
The pass rate was 9%. Across all 19 agents and 95 scenarios, only 9 passes. That does not mean the roster is full of weak agents. It means the field test is currently measuring adapter realism more than agent quality. When a wrapper keeps the harness alive but returns blank completions, the judge scores it zero. That is a compatibility failure, not a quality failure. But it still dominates the headline number.
The most expensive model added nothing. I ran both cloud tiers against all 19 agents. Cheap and better produced identical outcomes: 9/95 passes on both. The better tier did not surface a single regression or improvement that the cheap tier missed. The bottleneck is not the judge model. The bottleneck is how faithfully the harness exercises real agent logic. When the adapter quality is low, the judge sees weak output and scores it low — regardless of which model is behind the judge.
Config chaos was the real blocker. One agent hardcoded ChatOpenAI() at module scope with a model name my local MLX endpoint does not serve. Patching required a monkeypatch on __init__ that runs before import. Another wrote to /root at import time. Three agents brought ormsgpack whose C extension ABI did not match my Python runtime — they were quarantined. One repo had pyproject.toml in a subdirectory; uv sync at the root was a silent no-op. I ended up writing eight compatibility wrapper modules, creating three pyproject.toml files for repos with broken packaging, and removing one agent entirely because it was stuck on an outdated PydanticAI API version.
This is just what happens when you run software you did not write, on a machine you did not set up. It is also why most agent evaluation stays inside the author's environment, against the author's examples, with the author's model. That is comfortable. It is the worst possible way to evaluate whether an agent actually works.
The one that worked
lg-mcp-agents, a LangGraph multi-agent repo that was previously non-runnable from outside its original Streamlit app, achieved 5/5 passes on both cloud tiers after targeted adapter work.
That result matters because it tells you the adapter is exercising real agent behavior. When you see a 5/5 from an agent you could not even import two days earlier, the integration is real.
Why field testing was a late discovery — and should not have been
The field harness was not in the WBS. I added it because unit tests and mock agents were passing cleanly and that felt wrong. The mock agents were too well-behaved. They did exactly what the adapter contract said. Real agents do not.
Real agents transitively import ffmpeg. They create venvs with a different Python version than yours. They write to absolute paths. They hardcode API keys at module scope. They nest their project files in subdirectories. Mock coverage caught none of this because mock coverage does not run real code.
The lesson is not "add field testing." The lesson is that unit and mock coverage is necessary but not sufficient for an evaluation harness. Plan for a real-agent field layer from the start. It will be the only test layer that catches integration reality.
That is probably the biggest learning from the launch: agent evaluation gets discussed like a scoring problem, but in practice it becomes an environment, adapter, and realism problem much faster than most teams expect.
I am not done
This is v0.1.0. I want to be clear about that.
The field test proves that AgentEval Forge can import, configure, invoke, and score real third-party agents from GitHub at scale. But it does not yet prove that the scorer can rank agents meaningfully across a large, heterogeneous roster. Some of the PydanticAI wrappers showed blank completions across all scenarios — the harness ran, but the agent did not produce real output. That is a compatibility achievement, not an evaluation one. I am keeping it honest because I think this is the kind of thing that gets hand-waved away in launch announcements and then discovered painfully by users.
The blank-completion behavior in some PydanticAI wrappers needs investigation. The ormsgpack C extension mismatch quarantined three otherwise viable agents and needs a containerized workaround. The field test roster needs to grow: more agents, more diversity, more corner cases. The adapter quality needs to improve until field-level compatibility crosses over into evaluation-level usefulness. The SWE-bench and WebArena connectors are built and verified end-to-end, but they are not yet integrated into the default CI path.
And the article series is only 1/5 done. I wrote about why agent evaluation is harder than model evaluation. Now the launch has given me better material for the next pieces: what scenario packs should actually look like, how to think about trajectory scoring, where adversarial testing becomes useful, and why cost-quality tradeoffs are often less important than integration realism.
What comes next
The repo is at github.com/deghosal-2026/agent-eval-forge. It installs with pip install agent-eval-forge. The README has the full capability list, and the repo docs include the user guide, field test report, and hard-won lessons.
If you are building agents, I have a question for you: what is your current evaluation workflow? Are you still eyeballing a few demos and calling it done? Have you tried scoring trajectories yet, or are you still just checking the final answer?
And if you have tried to evaluate agents at scale — especially third-party agents you did not write yourself — what broke first?
Top comments (11)
The field-test result is the useful warning here. If the 9 percent pass rate mostly measures adapter realism, the eval harness is already telling you something before it ranks agents: your test surface is still too polite. I would probably make "can this agent be invoked outside its author's environment?" a first-class score instead of treating it as setup noise.
Exactly right, and honestly, I was a bit naive and optimistic, the tool will score higher pass rate for higher starred agents and maybe successful in catching issues in lower popular ones - this seemed like a good hypothesis. And, at the same I had pivoted more towards putting this in CI pipeline so I can push the eval more to the left. With the lower pass rate, and adapter realism and the extra shims I had to build, I am considering exactly as you said - will agent eval be challenging if folks use outside my intended environment or do I consider making this part of a common agent development environment? I am definitely not aiming to be a SWE Bench analog to come baseline agents that you have already invested in eval-ing other ways. Anyway, I have to think through these and solidify my use cases as to what this should be and what it should not be. Setup noise turned out to be very high.
nice work
Hands down one of the most honest and insightful posts on AI Agent evaluation I've read on DEV! 👏 The 9% pass rate finding is a huge eye-opener—it completely shifts the conversation from 'model intelligence' to 'software environment and integration reality.'
To answer your question: currently, we are mostly relying on LLM-as-a-judge for final outputs, but trajectory/tool-call failures are definitely our biggest blind spot. Sticking strictly to the 'Safety > Correctness > Efficiency' hierarchy in your PRD is such a solid architectural choice. Looking forward to part 4 of the series!
The field-testing discovery is the part that matters most. Searching 150+ repos and landing on 19 testable agents, with most being "too heavy, too coupled, or no clear entry point" — that is not a sampling problem, it is an ecosystem problem. Most agents are not eval-ready because they were not built to be observed.
The adapter contract you described — scenario input, allowed tools, disallowed tools, budget, never seeing the expected answer or scoring thresholds — is the right isolation boundary. No ground-truth leakage is the minimum bar and most eval frameworks do not meet it. The five adapter surfaces (subprocess, Python import, HTTP, LangGraph, PydanticAI) also tell you something about the fragmentation in the agent space: there is no standard invocation contract, so the harness has to bridge every surface.
The 17 deterministic scorers vs 11 LLM-as-judge split is interesting. In my own work on verification gates, the deterministic checks caught the structural failures (missing tool calls, wrong artifact paths) while the LLM judge caught the semantic failures (right tool, wrong intent). The ones that broke the clean version of the story were usually the cases where the deterministic scorer said pass but the LLM judge said the agent solved a different problem than the one posed.
Curious about the adversarial generator — does it produce perturbations of existing scenarios or generate entirely new failure modes? The distinction matters because generated adversarials tend to cluster around known failure types while real-world agents fail in ways nobody thought to test for.
A robust evaluation framework should measure more than task completion. Observing how an agent responds to failures, ambiguous inputs, and changing conditions provides a far better indication of production readiness than success rates on controlled examples alone.
The 9/95 result makes adapter realism look like a first-class evaluation dimension, not setup noise. I would split the CI result into at least three gates: invocation compatibility (import, environment, tool contract), policy/safety compliance, and task quality. Otherwise a blank completion or import-time side effect can collapse into the same score as a genuinely incorrect trajectory.
For the field layer, a useful regression fixture might be a deliberately awkward agent: nested project root, delayed tool response, unavailable optional dependency, and a restart after partial output. If the adapter can resume and preserve the failure classification, the score becomes actionable. The current repo caveat that compatibility is not yet ranking quality is exactly the distinction I would keep visible in CI.
Are all 19 agents tested on identical hardware/OS? You mention local MLX on Apple Silicon, but what about CI runners (Linux/Docker)? Do you publish detailed reports per agent (environment vars, dependency versions, adapter logs)? Can readers reproduce or rerun evals on different machines/clouds to verify the 9% baseline? Raw test reports are linked, but how granular is the failure data—can someone tell if their agent fails due to adapter quirk vs. real incompatibility?
Hello there,
I hope you're doing well.
I have a good business idea that I'd love to discuss with you in more detail.
To give you some background, a friend of mine started this business with a U.S.-based partner three years ago. Since then, he's been paying his partner between $8,000 and $10,000 per month, and the business has been working well.
If you're interested in learning more, I'd be happy to share the details.
This is my whatsapp number: +81 70-9427-3751
Best regards,
Ono
The 9% pass rate measuring adapter realism instead of agent quality is the key finding here, it would've been easy to publish that number as "most agents are weak," but correctly diagnosing a blank completion as a compatibility failure, not a quality failure, is the harder and more honest read.
The cheap-vs-better tier result reframes eval work generally: if the expensive judge produces identical outcomes to the cheap one, the bottleneck was never the judge, it was whether the harness faithfully exercised what it was judging.
"Testing inside the author's own environment against the author's own examples is the worst possible way to evaluate whether something actually works" is the line that generalizes furthest past this project.
Curious about the blank-completion PydanticAI wrappers, is the current guess that the adapter's invoking correctly but the agent's own output path silently short-circuits, or is it still unclear where the failure originate
The honesty in this write-up is refreshing. Hitting a 9% pass rate because the field test mostly measured "adapter realism and environment setup" rather than agent intelligence is the exact wall everyone runs into when trying to evaluate third-party repos.
The point about config chaos—module-scope API calls, hardcoded model imports, and transitive dependencies like ormsgpack—highlights why standard CI/CD paradigms break when applied to AI agents. Making "Safety > Correctness > Efficiency" a non-negotiable hierarchy in the PRD was a great architectural choice. Until we standardize agent interfaces and execution boundaries, harness builders are going to spend 80% of their time writing compatibility shims rather than scoring logic. Congrats on getting agent-eval-forge onto PyPI!
To answer your question at the end: what breaks first when trying to evaluate third-party agents is almost always side-effects and environment assumptions. The moment an agent assumes a specific directory structure, a local database connection, or module-scoped model initialization, standard test runners fall apart.
The decision to build dynamic adapters for LangGraph and PydanticAI while using [unk] fallback patterns and local MLX models for rapid sweeps makes agent-eval-forge super pragmatic. Looking forward to part 4 of the series to see how you evolve trajectory scoring!