Threshold testing is blind to agent regression; we built the statistical alternative, validated against 29 PRs

Co-authored by Rudrendu Paul and Sourav Nandy.

Repo: github.com/RudrenduPaul/agent-eval, a statistical regression-testing library for LLM agents. pip install agent-regress-cli.

You change one line in a system prompt. The next day's eval run shows accuracy down three points. Is that a real regression, or just normal LLM sampling noise? Most eval suites can't tell you, because they check whether a single response cleared a threshold instead of measuring how the whole distribution of behavior shifted. You might tweak a prompt, swap from GPT-4o to a cheaper tier, or quietly bump a dependency in LangGraph or CrewAI, and the eval suite still shows green regardless.

agent-eval answers that question directly. It runs an agent 50 times on version A and 50 times on version B, then reports a p-value and an effect size so you know if anything actually changed:

from agent_regress import compare

report = compare(
    version_a=agent_v1,
    version_b=agent_v2,
    test_suite=test_suite,
    n_runs=50,
    metric="tool_accuracy",
)

print(report)
report.assert_stable()  # raises AssertionError if behavior regressed

Here's what a caught regression looks like, straight from the repo's own example running live:

This is the core value in one screenshot: agent-eval runs 500 real trials per version and reports a statistically confident REGRESSED verdict instead of just a lower score.

We didn't want to ship that and call it done. We wanted to know if it actually held up against real, merged, contested code changes from the frameworks people run agents on today. So we pulled 239 real merged PRs across six repos: LangGraph, CrewAI, and the OpenAI Agents SDK, plus EleutherAI's lm-evaluation-harness, DeepEval, and Hugging Face's evaluate (three adjacent eval and benchmark tools we sanity-checked the same pitch against before focusing on the framework integrations). For each one, we asked a simple question: if we pointed agent-eval at exactly this kind of change, would it deliver what a cold pitch about it claims? The rest of this piece covers what that campaign found, including the part where our own tool crashed on the first call.

Gartner expects more than 40% of agentic AI projects to be scrapped by the end of 2027, and the reason cited most often is "escalating costs, unclear business value, or inadequate risk controls" (Gartner, June 2025). "Inadequate risk controls" is often just this problem in disguise: nobody can tell whether the agent's behavior actually changed after the last deploy.

Why threshold testing misses this

Most eval tools answer one question: did this response pass? DeepEval, Promptfoo, and Braintrust are all built around thresholds like a rubric score, an exact match, or a judge-model verdict. None of them tell you whether an agent's behavior meaningfully shifted between two versions of the same system. A 3-point accuracy drop between v1 and v2 might be real, or it might just be ordinary LLM sampling variance. Without a distributional test, you can't tell the difference. Teams usually end up ignoring small drops and eating the risk, or escalating everything and drowning their on-call rotation in false alarms.

The eval space isn't short on tools, and it's consolidating. OpenAI acquired Promptfoo in March 2026, keeping it open-source but folding its team in-house. That's a sign the eval layer is becoming infrastructure people actually build on top of. DeepEval and Braintrust remain strong at answering whether a response cleared a rubric, an exact match, or a judge-model score. None of the three report a p-value, an effect size, or a bootstrap confidence interval on whether behavior shifted between two versions. That's a different statistical question than the one they were built to answer.

This distinction matters more as the surrounding ecosystem keeps growing. LangGraph alone is at 37,447 GitHub stars and roughly 66.7 million PyPI downloads a month as of this writing. CrewAI sits at 55,648 stars and about 11.3 million downloads a month. A lot of production agents change versions on a normal release cadence, and threshold evals simply cannot tell you on their own whether any given release changed real behavior or just changed the score.

How the statistics actually work

Three statistical tools do the actual work. Mann-Whitney U compares the two score distributions without assuming they're normal, which matters because LLM output scores usually aren't. A bootstrap confidence interval (1,000 resamples) adds magnitude to that significance call, measuring how large the shift actually was. Cohen's d separates statistically significant from operationally significant: a shift at p = 0.001 with d = 0.04 is real but irrelevant, while a shift at p = 0.06 with d = 0.5 is large and needs more samples to confirm. The default CI gate only fires when both p < 0.05 and d ≥ 0.2, which keeps a 50-run comparison from failing your build over noise.

Here's what those numbers look like on the tool's own demo comparison, swapping GPT-4o for GPT-4o-mini to cut cost:

• 50 runs on version A: mean tool accuracy 0.840, SD 0.060
• 50 runs on version B: mean tool accuracy 0.700, SD 0.090
• p = 0.0031, Cohen's d = -0.610, 95% CI [-0.221, -0.067]
• Verdict: REGRESSED, block the deploy

We measured statistical overhead on an M3 Pro at about 27ms per comparison at n=50. The agent calls themselves remain the real bottleneck; the math barely registers.

None of this is exotic. The core call is one line of scipy, and we say that plainly because it matters for where this tool actually earns its keep. Any eval platform could bolt those statistics on in an afternoon. The hard part to copy is the version-specific regression history and the framework integrations, because those are what make the statistics usable from a real agent in CI without hand-rolling a harness every time. That's what turns a leap of faith, like swapping to a cheaper model, into a testable claim. A REGRESSED verdict with an attached p-value and Cohen's d either backs that decision or kills it before it ships.

The bug that would have crashed every single comparison

This is where the story stops being a features list. Before we trusted the framework integrations enough to write outreach copy about them, we ran a validation pass, the first one, before we'd shown the tool to anyone outside our own repos. We took a real, merged PR from LangGraph, CrewAI, or the OpenAI Agents SDK and checked whether agent-eval could actually produce the comparison a pitch about that PR would claim.

The very first thing we found was a P0. openai_agents_runner() called agent.run(query). That method doesn't exist on the real SDK's Agent class; only Runner.run(agent, input) does. We confirmed it directly: hasattr(agents.Agent(...), 'run') returns False, and hasattr(agents.Runner, 'run') returns True. The integration raised AttributeError on its first call, for any test suite, any agent, any claim. It was a hard crash that took down the whole integration. This one wrong method name accounted for 15 of the 29 code-fixable gaps the campaign found, or 51.7% of all of them.

We reproduced that exact crash live against the real, currently installed OpenAI Agents SDK, running the actual pre-fix code straight from the commit history:

This isn't a mockup. It's the real buggy call site from the commit before the fix, run against the real SDK, producing the actual AttributeError that broke every comparison.

We fixed the call site and kept going. A crash that big rarely travels alone, and this one didn't either. langgraph_runner() built a fresh, stateless dict on every call, with no thread_id and no checkpoint continuity, so it couldn't represent multi-turn conversations at all. Meanwhile crewai_runner() only wrapped a full Crew.kickoff(), with no way to isolate one tool's behavior. run_suite() had no cache-busting nonce, which meant any framework with its own persistent result cache (LangGraph's cache_policy is a real example) would silently collapse the repeated-sampling variance the whole statistical approach depends on. That last issue turned into a second bug once we looked closer: a piece of state-narrowing logic downstream was dropping the cache-bust key we'd added. Re-review caught it; the first pass missed it.

Two commits closed most of it. 8411eb5 fixed the P0 and 16 other gaps, taking the test suite from 121 tests to 264 passing. f752c11 closed 6 of the 7 residual gaps that a second, independent re-validation pass surfaced, including an async cancellation liveness bug in the store layer. The 7th gap tied to a LangGraph interrupt/resume report and needed one more manual revalidation pass before we could confirm it closed.

Both passes independently caught the same mistake on the first attempt: a proposed fix credited code that was technically correct but still sitting uncommitted in the working tree. You only find that kind of thing by actually running an adversarial check instead of trusting your own fix, and we had to learn that lesson twice. This is a specific, repeatable failure mode of self-reported "this is fixed now" claims, which is why the validation methodology below runs every verdict through a propose-then-refute step instead of taking the first answer.

29 real merged PRs, and what it took to actually test them

We anchored every fix above on a real, merged pull request. The full validation set covered 239 rows across six repos. We excluded 209 of those because the underlying PR never carried the kind of behavioral-drift risk the pitch assumed: docs-only changes, trivial fixes, or PRs that plain didn't exist. Of the 29 rows where the code gap was real and fixable, all 29 now pass. One additional row (@sibblegp) has its code gap fully closed too, but it stays below PASS for a reason no amount of engineering can fix: there's no genuine merged PR to anchor the claim on.

A representative slice of those 29 spans all three frameworks and most of the fix categories above:

PR	Repo	What it changed	What agent-eval needed to test it
langgraph #5243	LangGraph	New typed context= API, replacing untyped config['configurable']	langgraph_runner() gained config, context, and thread_aware kwargs so both invocation styles compare directly
openai-agents-python #2463	OpenAI Agents SDK	Fixed agent-as-tool silently dropping the parent run's RunConfig	Fixed the P0 crash, then added capture_trace= telemetry so RunConfig inheritance is observable per nested call
crewAI #6236	CrewAI	Optional Pydantic output_schema on tools, structured JSON instead of str()	New crewai_tool_runner() isolates a single tool, plus a schema_conformance_scorer()
langgraph #7746	LangGraph	Reworked checkpoint snapshot cadence to key on supersteps, not just updates	thread_aware=True for per-case thread continuity across repeated runs
crewAI #6134	CrewAI	Security fix: file tools were leaking absolute filesystem paths in responses	New no_path_leak_scorer() flags raw paths in tool-response text
openai-agents-python #2214	OpenAI Agents SDK	Stopped image/audio/file tool outputs from being silently dropped to text-only	capture_trace= intercepts the SDK's real conversion function directly, so regressions of this exact class are now detectable
langgraph #3126	LangGraph	Reworked ToolNode dispatch, risking duplicate tool calls on interrupt/resume	tool_call_trace_scorer() plus a real langgraph_interrupt_resume_runner() that drives an actual interrupt-then-resume cycle
openai-agents-python #2902	OpenAI Agents SDK	Added a persistent MongoDB session backend for multi-turn continuity	session_aware=True reuses one session per logical test case instead of building a fresh one every call
langgraph #4486	LangGraph	Added node/task-level result caching, which can mask repeated-sampling variance	cache_bust_key_fn() plus graph_has_cache(), and the dropped-nonce bug described above
langgraph #6701	LangGraph	Fixed a cancelled-future re-queue bug in the async batching store layer	New arun_suite(), langgraph_async_runner(), and concurrent_cancellation_probe() to reach async liveness bugs at all
openai-agents-python #2328	OpenAI Agents SDK	Fixed a hard crash using DeepSeek's thinking mode through LiteLLM	Fixed the P0 crash; verified end-to-end against a live install
crewAI #6079	CrewAI	Four pluggable storage backends for memory, knowledge, RAG, and flow persistence	crew_factory= builds a fresh Crew per invocation, avoiding shared-mutable-state corruption under concurrent comparison
crewAI #4446	CrewAI	Substantial refactor of the Brave Search tool integration	subprocess_runner() compares crewai-tools versions pre- and post-PR
openai-agents-python #1744	OpenAI Agents SDK	Added support for Anthropic's extended/interleaved thinking via LiteLLM	Fixed the P0 crash, closing the runner-level failure that blocked exercising this feature at all

Every link above resolves to a real, merged PR. That covers 14 of the 29. The complete list, with a real PR link (or an honest note where one isn't on record) and the exact fixing commit for every row, is in the repo at docs/validation.md. The pattern across all 29 is the same: the plumbing connecting agent-eval's statistics to a specific framework's real API surface was incomplete, and that only became visible by testing against what maintainers actually shipped.

Supported frameworks

Four framework integrations are shipped as of v0.1, each installable as its own extra:

• LangGraph: pip install agent-regress-cli[langgraph]
• OpenAI Agents SDK: pip install agent-regress-cli[openai-agents]
• CrewAI: pip install agent-regress-cli[crewai]
• LangChain LCEL: pip install agent-regress-cli[langchain]

AutoGen support is planned for v0.3. A Vercel AI SDK integration for TypeScript agents is planned for v0.4. Until then, use subprocess_runner() as a workaround, which compares two installed package versions by running them as subprocesses.

If you're on a framework not listed here, the fastest path is opening an issue with the framework name and a link to one representative merged PR. The validation methodology above is documented and repeatable: point it at your integration and we can work through where it breaks.

Standard benchmarks

Beyond version-to-version comparisons, agent-eval ships harnesses for the three standard benchmarks in the field, so you don't have to write a separate scaffold for each. Tau-bench pass^k runs an agent across k independent attempts (k=1, 4, 8), because single-run scores hide reliability degradation that a repeated-attempt curve catches. The GAIA harness splits results by Level 1-3 difficulty instead of collapsing everything into one accuracy number, since a change that helps easy tasks can quietly hurt hard ones. The SWE-bench harness reports a scaffold pass rate, isolating what the agent framework itself contributes from what the underlying model contributes. All three are version-controlled in the repo's leaderboard/ directory, and results are independently reproduced before a submission gets merged.

Using agent-eval in CI

agent-eval shipped as a pure Python import for its first several months. As of this week, it's also a real CLI, which is how most people will actually run this in a pipeline rather than as a Python import.

One naming note before you install anything: the repo is agent-eval, the PyPI package is agent-regress-cli, and the Python import is agent_regress. Three names for one project, because agent-regress on PyPI was already blocked as too similar to an existing, unrelated project by the time we tried to publish, so the distribution got renamed and the import path stayed put.

pip install agent-regress-cli
# or
uv add agent-regress-cli
# or, via npx if you'd rather not touch Python directly
npx agent-regress-npm-cli --help

The Python API's compare() takes two callables and runs them itself. That shape has no clean command-line equivalent, so the CLI starts one step later: it takes two JSON files containing flat arrays of per-run scores your own harness already computed, then runs them through the same Mann-Whitney U, bootstrap CI, and Cohen's d pipeline the Python API uses internally:

agent-regress compare \
  --version-a-results v1_scores.json \
  --version-b-results v2_scores.json \
  --metric tool_accuracy \
  --json \
  --fail-on-regression

--json prints a single, parseable JSON object to stdout and routes warnings to stderr so stdout stays clean. The CLI's own source calls this "the agent-native machine-readable surface," meant to describe a report that an orchestration script, a CI job, or another agent can consume directly instead of scraping human-formatted text. --fail-on-regression exits 1 on a REGRESSED verdict, so the same command drops straight into a CI gate.

You pass in two pre-computed score files and get a full statistical verdict back, without needing any Python imports to run it.

Here's the shape of that gate as an actual GitHub Actions workflow:

# .github/workflows/agent-regression.yml
name: Agent regression check
on: [push, pull_request]

jobs:
  regression:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install agent-regress-cli[langgraph]
      - name: Run agent comparison
        run: |
          python scripts/run_agent_suite.py \
            --version-a prod \
            --version-b staging \
            --output-a v1_scores.json \
            --output-b v2_scores.json
      - name: Statistical regression check
        run: |
          agent-regress compare \
            --version-a-results v1_scores.json \
            --version-b-results v2_scores.json \
            --metric tool_accuracy \
            --json \
            --fail-on-regression

scripts/run_agent_suite.py is your own harness, whatever runs your agent 50 times per version and writes out the score arrays. The gate itself, the last step, exits 1 on REGRESSED and 0 on STABLE. Drop it into any workflow that already runs your agent.

Honest limitations

We'd rather state the limits plainly than let the 29-for-29 number imply more than it does. Of the 239 rows in the full validation set, we excluded 209 (87.4%) simply because the pitch itself didn't apply to those PRs. That's a very different claim from "29 out of 239 attempts succeeded." This campaign says nothing about agent-eval's coverage of frameworks like AutoGen, which is still on the roadmap. It also doesn't touch correctness questions the stats module was never meant to answer, like whether a scorer function itself is well-designed. The framework-specific plumbing is the real work here. This campaign spent two commits building and re-testing it, and the version-specific regression history only accumulates from real use. The statistical core itself is just one scipy call anyone could copy in an afternoon.

Two more limits are worth stating outright. First, sample size: both assert_stable() and RegressionGate warn but don't fail when either version has fewer than 50 runs. That's the sample size needed to reliably detect a moderate effect (Cohen's d of 0.2) at 80% power. Below 10 runs per version, agent-eval returns INSUFFICIENT_DATA instead of a REGRESSED/STABLE/IMPROVED verdict, because the sample is simply too small to trust any statistical conclusion. Second, agent-eval never makes a model call itself. compare() runs whatever callables you hand it, version_a and version_b, against your own test suite. There are no API keys to manage and no hidden model cost inside the tool. AutoGen and the TypeScript-native Vercel AI SDK integration remain unshipped; we're tracking them above rather than quietly dropping them.

Try it against your own agent

from agent_regress import compare, RegressionGate

gate = RegressionGate(p_threshold=0.05, min_effect=0.2)

def test_tool_accuracy():
    report = compare(version_a=prod_agent, version_b=staging_agent, test_suite=suite, n_runs=50)
    gate.check(report)  # raises AssertionError on regression, warns if n < 30

Apache 2.0, self-hostable, no SaaS account required. docker compose up gets you a working example and the leaderboard UI in one command if you want to see it running before wiring it into anything. The stats module (src/agent_regress/stats/) carries a hard constraint in the codebase itself: it has to stay pure Python and scipy, with no LLM calls, ever. That way the statistical core never becomes a moving target with its own model dependency.

If you maintain or contribute to LangGraph, CrewAI, or the OpenAI Agents SDK, there's a decent chance one of your own merged PRs is hiding in that 239-row dataset. The good-first-issues list is open if you want to help close the remaining gaps. The leaderboard takes PRs too: you can run one of the benchmark harnesses above against a framework or model we haven't covered, then drop the resulting JSON into leaderboard/ matching leaderboard/schema.json. It gets independently reproduced before merging.

Two things are on the roadmap for the next release. One is AutoGen support. The other is deeper CI ergonomics for the three frameworks already shipped: a native GitHub Actions annotation format and a richer --json schema for dashboards. We'd rather ship the one people actually hit first, so if you have a strong preference, or if you maintain a framework we haven't covered yet, open an issue. The validation methodology is documented and the pattern is repeatable.

Repo: github.com/RudrenduPaul/agent-eval. If this caught a regression you would have shipped, or if the validation methodology gave you a pattern worth repeating on your own framework, a star helps other teams find it before they need it.

---

References

• Gartner, "Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027", June 25, 2025
• OpenAI, "OpenAI to acquire Promptfoo", March 9, 2026
• GitHub and PyPI star/download counts self-reported via public API as of July 16, 2026

Co-authored by Rudrendu Paul and Sourav Nandy.

Rudrendu Paul and Sourav Nandy build open-source developer tools for the AI agent ecosystem. They are the co-authors of agent-eval (agent-regress), a statistical regression-testing library for LLM agents, and spent the past several months validating it against real merged PRs in LangGraph, CrewAI, and the OpenAI Agents SDK.