DEV Community: benchclawio

We ran 160 agent tasks across two frameworks. The frameworks tied. Then we changed the model.

benchclawio — Sat, 29 Aug 2026 13:27:52 +0000

We ran 160 agent tasks across two frameworks. The frameworks tied. Then we changed the model — and one task failed 20 times in a row.

The LangChain vs LangGraph question comes up every time someone starts a new agent project. We had a slightly different version of it: once you've chosen LangGraph as your runtime, does picking the library around it actually change production outcomes?

So we ran a controlled benchmark to find out.

The setup

LangGraph 1.2.9 against Pydantic AI 2.13.0. Four tasks, designed to test realistic agent work: order processing with tool calls, shipping quote calculation, refund eligibility with date logic, and inventory reorder decisions. All tasks shared the same tool implementations and the same scorer — the only variable was which library orchestrated the calls.

We used gpt-4o, temperature 0, no parallel tool calls. The harness is public at github.com/benchclawio/harness. Every run was logged, nothing was cherry-picked, the raw JSONL is published.

160 task executions total. Here's what happened.

The frameworks tied

LangGraph: 100% correctness across all four tasks.
Pydantic AI: 100% correctness across all four tasks.

Not close — exact tie. Zero correctness gap on any task.

There was a performance gap: LangGraph was roughly 1.4–1.8 seconds faster per task on median wall time. The cause is Pydantic AI's async-to-sync bridge — when you call it synchronously, it spins an event loop internally, and that overhead is real and consistent. If you're building a latency SLO, that matters. If you're evaluating correctness first, the frameworks are identical with gpt-4o.

This result is unsatisfying in a useful way. It tells you that the framework abstraction is not where correctness variation lives — at least on these tasks, at this model, at this scale.

So we ran the same tasks again. Same harness, same task suite. Different model.

The model didn't tie

We substituted gpt-4o-mini for gpt-4o. Everything else held constant.

Overall correctness dropped to 75%. The framework split: same.

One task failed completely, 20 for 20 runs: refund-policy-minimal-tools. The task requires computing whether a return is within the 18-day return window. gpt-4o-mini calculated 19 days inclusive where the answer is 18 days exclusive — a date-arithmetic edge case. It failed this identically on every single run, regardless of which framework was handling the tool calls.

The other three tasks: 100%. The failure wasn't variance — it was systematic. And it was entirely a model property, not a framework property.

What this means for the framework debate

We had to decide what to compare LangGraph against before we started. Our dependency analysis (benchclaw.io/langchain-vs-langgraph/) found that LangChain 1.3.14 now declares LangGraph as an unconditional dependency — installing LangChain installs LangGraph. The reverse isn't true. So "LangChain vs LangGraph" is less of a choice than it used to be; the more real decision is whether to write in Pydantic AI or in LangGraph primitives once you're in the LangGraph runtime.

Even reframed that way, our benchmark says the framework abstraction doesn't move correctness outcomes on structured tasks at gpt-4o.

What did move outcomes was the model. By a lot, on a task that sounds easy.

The practical implication: if your task suite involves any temporal reasoning — eligibility windows, expiry dates, SLA calculations — test both your intended model and a cheaper fallback explicitly before you ship. "It works in testing" is not enough if testing only used gpt-4o and production routes some traffic to gpt-4o-mini or a smaller model.

The finding we didn't expect

Before we ran the model comparison, we expected the framework difference to be small. We expected correctness to hold. What we didn't anticipate was how systematic the failure mode was.

gpt-4o-mini didn't get the date arithmetic right on some runs and wrong on others. It got it wrong 20 out of 20 times, by exactly the same margin. That's not a probabilistic sampling failure — that's a knowledge gap baked into the model at this temperature and this task phrasing.

The implication is that per-task model validation matters more than framework benchmarking for production agent decisions. You can swap LangGraph for Pydantic AI or back again in an afternoon. You can't patch a model's arithmetic mid-deployment.

The data

Full benchmark with methodology, raw JSONL, and harness: benchclaw.io/langgraph-vs-pydantic-ai-benchmark/

Harness repo (Apache 2.0, citable): github.com/benchclawio/harness

If you want to reproduce it, the harness runs offline on any machine with Python and an OpenAI key.

LangGraph 1.2.9, pydantic-ai-slim[openai] 2.13.0, gpt-4o (temperature 0) and gpt-4o-mini (temperature 0). Benchmarked 2026-07-24. All runs logged; nothing omitted.

We Cut Our Agent's Token Bill by 21%. One Task in Four Got Nothing.

benchclawio — Tue, 18 Aug 2026 08:53:02 +0000

We turned on progressive disclosure for an agent with 20 tools. Deferring the tool schemas cut input tokens by about 30% and total cost by about 21%. Good result, shipped, write the blog post.

Then I broke the numbers down by task, and the average turned out to be hiding something. Three of our four task types saved 21% to 30%. The fourth saved nothing on one transport and cost 12.3% more on the other.

This post is about that fourth task, and about why the number you should care about when you defer tool schemas is not the mean.

All 80 runs are public. Every command below is one I ran while writing this, with its real output pasted in. You can reproduce the whole thing offline for free.

The setup

Four business tasks, each needing exactly one tool. Twenty tool schemas registered, always. Two arms:

always-on: all 20 schemas serialised into every request
deferred: all 20 wrapped in Pydantic AI's DeferredLoadingToolset, so the model has to search for a capability and load it before calling it

Twenty runs per cell, sequential, no retries, gpt-4o at temperature 0, parallel_tool_calls=False. Run on 2026-08-06 against pydantic-ai-slim[openai]==2.24.0. Both transports, Chat Completions and the Responses API, because tool search executes server-side on one and through a local fallback on the other.

The whole thing cost $0.279585 to run.

The averages, which are fine

cell     mean_in    stdev    min    max
A-chat    1361.5     10.3   1350   1372
B-chat     945.2    190.8    780   1264
A-resp    1353.8     10.6   1342   1364
B-resp    1001.0    263.1    807   1477

A is always-on, B is deferred. Input tokens down 30.6% on Chat Completions and 26.1% on Responses. Cost down 21.2% and 16.8%. Correctness was 80/80 exact matches, so nothing broke.

But look at the standard deviation column. The always-on arms sit within 22 tokens of each other across every run. The deferred arms swing across a 484 and a 670 token range. Deferral made the input size roughly twenty times more variable.

My first assumption was run-to-run nondeterminism: the model writes a slightly different search query each time, gets back a slightly different set of schemas, and the prompt size wobbles. That assumption was wrong, and checking it is what produced the actual finding.

Decomposing the variance

Here is the check. Group by task inside each cell instead of pooling the cell:

curl -sL -o bc025.jsonl https://raw.githubusercontent.com/benchclawio/harness/main/results/bc025-progressive-disclosure-2026-08-06/bc025-scored-raw-2026-08-06.jsonl

python3 -c "
import json, statistics as s, collections
rows=[json.loads(l) for l in open('bc025.jsonl')]
for c in ('A-resp','B-resp'):
    print('==', c)
    g=collections.defaultdict(list)
    for r in rows:
        if r['cell']==c: g[r['task_id']].append(r['metrics']['tokens_in'])
    for t,v in sorted(g.items()):
        print(f'   {t:22s} n={len(v)} mean={s.fmean(v):7.1f} sd={s.stdev(v):6.1f}')
"

Real output:

== A-resp
   currency-conversion    n=5 mean= 1342.0 sd=   0.0
   defect-threshold       n=5 mean= 1364.0 sd=   0.0
   inventory-reorder      n=5 mean= 1345.0 sd=   0.0
   shipment-delay         n=5 mean= 1364.0 sd=   0.0
== B-resp
   currency-conversion    n=5 mean=  930.2 sd=   2.7
   defect-threshold       n=5 mean= 1431.8 sd=  95.6
   inventory-reorder      n=5 mean=  807.0 sd=   0.0
   shipment-delay         n=5 mean=  835.0 sd=   0.0

Two things fall out of this.

Run-to-run variance is near zero in both arms. Repeat the same task five times under deferral and you mostly get the identical token count, standard deviation 0.0. The nondeterminism I assumed was there is not there, or is small enough not to matter.

The spread is between tasks, not between runs. Under always-on, all four tasks land within 22 tokens of each other, because the 20 schemas dominate and the task text barely moves the total. Under deferral that flattening disappears and the tasks spread from 807 to 1,432 tokens.

That reframes the whole thing. Deferral does not make your cost noisy. It makes your cost task-dependent. Always-on charges you the same amount whatever the user asks. Deferred charges you according to what the model decides to search for and load, which means your bill now tracks your traffic mix.

The task that lost money

Once cost is a function of the task, some tasks can come out behind. Here is the per-task cost, both transports:

python3 -c "
import json, statistics as s, collections
rows=[json.loads(l) for l in open('bc025.jsonl')]
g=collections.defaultdict(list)
for r in rows: g[(r['cell'],r['task_id'])].append(r['metrics']['cost_usd'])
hdr=f\"{'task':22} {'A-resp':>10} {'B-resp':>10} {'delta':>8}   {'A-chat':>10} {'B-chat':>10} {'delta':>8}\"
print(hdr)
for t in ('currency-conversion','defect-threshold','inventory-reorder','shipment-delay'):
    ar=s.fmean(g[('A-resp',t)]); br=s.fmean(g[('B-resp',t)])
    ac=s.fmean(g[('A-chat',t)]); bc=s.fmean(g[('B-chat',t)])
    print(f'{t:22} {ar:10.6f} {br:10.6f} {br/ar*100-100:+7.1f}%   {ac:10.6f} {bc:10.6f} {bc/ac*100-100:+7.1f}%')
"

Real output:

task                       A-resp     B-resp    delta       A-chat     B-chat    delta
currency-conversion      0.003795   0.003001   -20.9%     0.003795   0.002791   -26.5%
defect-threshold         0.003944   0.004428   +12.3%     0.003840   0.003810    -0.8%
inventory-reorder        0.003873   0.002707   -30.1%     0.003828   0.002770   -27.6%
shipment-delay           0.003970   0.002828   -28.8%     0.003846   0.002691   -30.0%

defect-threshold is the outlier. On the Responses API it cost 12.3% more with progressive disclosure on. On Chat Completions it saved 0.8%, which after four decimal places is a rounding error and not a saving.

Its deferred prompt came in at 1,431.8 input tokens against 807.0 for the cheapest task in the same cell. That is 625 extra tokens of loaded schema for a task that, like every other task in the suite, needed exactly one tool.

I want to be careful about the mechanism here, because the bundle does not record it. Every deferred run made exactly 2 tool searches and 3 model requests, defect-threshold included, so it is not doing extra round-trips. The most likely explanation is that its search matched more of the 20 capabilities and pulled more schemas back into the prompt than the other tasks did. But the raw JSONL logs token counts and the tool that was finally called, not the schema set the search returned, so I cannot prove that from the published data. Treat it as the obvious inference, not a measurement.

What the distribution looks like

The cost distribution per run makes the point better than a standard deviation does:

python3 -c "
import json
rows=[json.loads(l) for l in open('bc025.jsonl')]
for c in ('B-chat','B-resp'):
    v=sorted(r['metrics']['cost_usd'] for r in rows if r['cell']==c)
    print(c, [f'{x:.6f}' for x in v]); print()
"

Real output:

B-chat ['0.002470', '0.002587', '0.002717', '0.002717', '0.002717', '0.002717', '0.002740', '0.002800', '0.002800', '0.002800', '0.002815', '0.002845', '0.002845', '0.002845', '0.002845', '0.003810', '0.003810', '0.003810', '0.003810', '0.003810']

B-resp ['0.002707', '0.002707', '0.002707', '0.002707', '0.002707', '0.002828', '0.002828', '0.002828', '0.002828', '0.002828', '0.002992', '0.002992', '0.002992', '0.002992', '0.003037', '0.003922', '0.004497', '0.004573', '0.004573', '0.004573']

That is not a bell curve with a fat tail. It is a cluster and then a cliff, and the cliff is one task type.

On the Responses API, 5 of 20 deferred runs cost more than the average always-on run ($0.003895). The worst deferred run cost $0.004573, which is 17.4% above the always-on mean. The optimisation that saves 16.8% on average was, for a quarter of these runs, not an optimisation.

What I'd actually take from this

Compute your saving per task type, not per corpus. A single blended percentage is the one number that cannot tell you whether to ship this. If your traffic is 80% the defect-threshold shape, deferral loses you money while your dashboard reports a saving.

The saving is capped by the schema share of your prompt. Deferral removes tool schemas. It does not remove your system prompt, the user message, the conversation history, or the tool results coming back. In our tasks the schemas were roughly a third of the request, so removing nearly all of them saved roughly a third of input tokens. If you have five tools and a 4,000-token system prompt, there is nothing here for you.

Budget the extra round-trip as a certainty. Always-on completed in 2 model requests. Deferred took 3, in all 40 deferred runs, on both transports. Not an average with spread, a constant. If your latency budget is per-request rather than per-token, you are buying a variable token reduction with a fixed 50% increase in requests.

Do not verify deferral through the framework's own view of its tools. AgentInfo.function_tools lists a deferred tool both before and after it loads, and the local search_tools fallback is present either way. That surface tells you what the agent knows about, not what got serialised to the provider. Every number in this post comes from provider-reported request token counts instead, which is the only thing that maps to the invoice.

Accuracy was not the thing that broke. All 80 runs produced exact matches, 20 out of 20 in every cell. I expected correctness to be the risk and it was not, though our tasks each needed exactly one capability. Tasks needing several loads would compound the round-trips and give the model more chances to pick wrong. We did not test that.

Reproduce it

The bundle is 80 raw runs, both manifests with SHA-256s, the deterministic task-suite generator, the worker, the collector and the analysis script:

https://github.com/benchclawio/harness/tree/main/results/bc025-progressive-disclosure-2026-08-06

python3 analyze_bc025.py regenerates the confidence intervals. python3 bc025_capabilities.py regenerates the task suite byte for byte. Both are offline and cost nothing, because they read the recorded runs rather than calling a model.

The full benchmark, including the bootstrap confidence intervals and the part about where the widely quoted "90% to 98% savings" figure comes from, is written up at benchclaw.io.

One caveat on the version, since it moved while we were running: pydantic-ai-slim 2.25.0 landed on PyPI hours before these runs. We benchmarked 2.24.0 and then diffed the tags. _tool_search.py and toolsets/deferred_loading.py, the entire mechanism under test, are unchanged between them. We have not re-run on 2.25.0.

I Tried to Verify an AI Agent Benchmark. Here's the Bundle I Wish Everyone Shipped

benchclawio — Fri, 14 Aug 2026 06:51:35 +0000

Nearly every AI agent benchmark you read is unfalsifiable.

Not wrong, necessarily - unfalsifiable. There's a blog post with a bar chart, a claim that framework A beat framework B, and no way for you to check it. No run count. No model version. No raw output. Often no cost. You are asked to trust a summary statistic produced by people with an interest in the result.

We publish agent benchmarks, so this is our problem too. This post is about the evidence bundle we settled on, and how you can pull one down and take it apart in about two minutes. Every command below is one I actually ran while writing this, with its real output pasted in.

The claim we're going to try to break

From one of our pilot runs: LangGraph 1.2.9 and Pydantic AI 2.13.0 both completed 20 of 20 tasks under gpt-4o, at a total spend of $0.094275.

That's the sort of sentence you'd normally have to take on faith. Let's not.

Two minutes to verify it yourself

The bundle is a directory in a public repo. Pull it:

BASE="https://raw.githubusercontent.com/benchclawio/harness/main/results/gpt-4o-vs-gpt-4o-mini-tool-calling-2026-07-24"
for f in SHA256SUMS README.md gpt4o-pilot-manifest-v0.4.0.json \
         scored-pilot-gpt4o-raw-2026-07-24.jsonl \
         scored-pilot-raw-2026-07-24.jsonl \
         scored-pilot-analysis-2026-07-24.json \
         scored-pilot-gpt4o-analysis-2026-07-24.json \
         real-pilot-status-manifest-v0.3.0.json; do
  curl -sfO "$BASE/$f"
done

First question: is this the same data we published, or has something drifted?

sha256sum -c SHA256SUMS

README.md: OK
gpt4o-pilot-manifest-v0.4.0.json: OK
real-pilot-status-manifest-v0.3.0.json: OK
scored-pilot-analysis-2026-07-24.json: OK
scored-pilot-gpt4o-analysis-2026-07-24.json: OK
scored-pilot-gpt4o-raw-2026-07-24.jsonl: OK
scored-pilot-raw-2026-07-24.jsonl: OK

That's the cheapest integrity control there is and almost nobody ships it. It costs one line in your run script and it means a reader can tell the difference between the file you published and a file someone edited afterwards.

Now the actual test. Ignore our summary entirely and recompute it from the raw runs:

import json, collections

rows = [json.loads(l) for l in open('scored-pilot-gpt4o-raw-2026-07-24.jsonl')]

by = collections.defaultdict(lambda: [0, 0])
for r in rows:
    b = by[r['subject']]
    b[0] += 1
    b[1] += r['completed']

for s, (n, ok) in sorted(by.items()):
    print(f"{s:<32} {ok}/{n}")
print("total cost  $", round(sum(r['metrics']['cost_usd'] for r in rows), 6))

langgraph_1_2_9_gpt4o_live       20/20
pydantic_ai_2_13_0_gpt4o_live    20/20
total cost  $ 0.094275

The claim reproduces to the cent, from the raw file, without going through our analysis code. That is the whole point of the exercise. Our published scored-pilot-gpt4o-analysis-2026-07-24.json says total_cost_usd: 0.094275 and total_runs: 40 - and now you know the summary wasn't doing anything clever on the way there.

The part that actually earns trust: finding the failure

A benchmark where everything passes tells you very little. The same four tasks under gpt-4o-mini:

import json, collections

rows = [json.loads(l) for l in open('scored-pilot-raw-2026-07-24.jsonl')]

by = collections.defaultdict(lambda: [0, 0])
for r in rows:
    b = by[r['task_id']]
    b[0] += 1
    b[1] += r['completed']

for t, (n, ok) in by.items():
    print(f"{t:<32} {ok}/{n}{'  <-- FAILED' if ok == 0 else ''}")

inventory-reorder                10/10
dependent-shipping-quote         10/10
recover-stale-revision           10/10
refund-policy-minimal-tools      0/10  <-- FAILED

Zero out of ten. Now the question a raw bundle can answer and a bar chart cannot: what kind of failure was it?

bad = [r for r in rows if r['task_id'] == 'refund-policy-minimal-tools']
print('status  :', set(r['status'] for r in bad))
print('failure :', set(str(r['failure']) for r in bad))
print('tool calls:', set(r['metrics']['tool_calls'] for r in bad))

status  : {'failure'}
failure : {"{'type': 'invalid_final_answer', 'stage': 'scoring',
            'message_sanitized': 'final output does not exactly match expected output'}"}
tool calls: {2}

That is the interesting shape. stage: scoring, not stage: execution. The agent made both tool calls it was supposed to make, raised no exception, hit no timeout, and returned a confident, well-formed, wrong answer. (For the curious: it computed a 19-day window inclusive where the policy needed 18 exclusive, then applied a correct eligibility rule to a wrong number.)

If all we had shipped was "75% completion rate", you would have no way to tell that apart from a crash loop or a rate limit. The failure taxonomy is doing real work here, and it's three fields.

The wart I found writing this post

Honesty is cheaper than getting caught. While writing the snippets above I hit this:

m = [json.loads(l) for l in open('scored-pilot-raw-2026-07-24.jsonl')]
g = [json.loads(l) for l in open('scored-pilot-gpt4o-raw-2026-07-24.jsonl')]
print('mini keys:', sorted(m[0].keys()))
print('4o   keys:', sorted(g[0].keys()))

mini keys: ['completed', 'failure', 'metrics', 'model_id', 'run_index', 'status', 'subject_id', 'task_id']
4o   keys: ['completed', 'failure', 'metrics', 'model_id', 'run_index', 'score', 'status', 'subject', 'task_id', 'wall_time_outer_s']

Two files in the same bundle, and one calls it subject_id while the other calls it subject. The gpt-4o file also gained score and wall_time_outer_s. That's schema drift between two runs a few hours apart, and it's why the two snippets above key on different field names - which I'd rather explain than quietly paper over.

It doesn't invalidate anything: the numbers are the numbers, and the checksums prove the files haven't moved since. But it's a real defect, it's the kind of thing a raw bundle exposes and a summary hides, and the fix is boring - a schema_version on every row and a validator in CI, which is what we should have had.

You are going to find this in your own bundles the first time someone reads them properly. Better that than nobody ever reading them.

What actually goes in the bundle

Stripped down, here's what we ended up shipping per run set, and why each piece is there:

File	Job
`-raw-.jsonl`	One JSON object per run. The source of truth. Everything else is derived.
`-analysis-.json`	Our summary. Included so you can check it against the raw.
`-manifest-.json`	What we declared before running: model, temperature, task suite, run count.
`SHA256SUMS`	Integrity. One line to generate, ends an entire category of argument.
`README.md`	What this run was and - importantly - what it is not eligible to claim.

The manifest is the one people skip and it's the one that matters most. Ours pins model_id, temperature: 0, parallel_tool_calls: false, and the per-token pricing used to compute cost. Declaring the run count before execution is what stops a benchmark quietly becoming "we ran it until it looked right".

And the README carries the eligibility verdict. That pilot is explicitly marked not publication-eligible - the task suite changed mid-run, and there was an OOM restart. We still published the bundle, because a flawed run you can inspect is worth more than a clean-looking one you can't.

The checklist, if you're building one

Nothing here is hard. It's about ten lines of extra code in your run loop:

One row per run, JSONL, written as you go. Not at the end - you want the partial data when something dies at run 32 of 40.
Record tokens, cost, wall time, tool calls per run, not just the aggregate.
Taxonomise failures with a stage. execution vs scoring is the single most informative bit you can store.
Write the manifest before the first run, and include the declared run count.
Ship SHA256SUMS.
Version your row schema, which I clearly need to take my own advice on.
State what the run may not be used to claim.

If a benchmark you're reading doesn't ship these, that isn't proof it's wrong. It just means neither of you can find out.

The full harness, the task suite and every bundle mentioned here are at
github.com/benchclawio/harness. Our write-up of the
harness itself is at benchclaw.io/harness, and the methodology
we hold ourselves to - including what disqualifies a run from publication - is at
benchclaw.io/methodology.

If you spot something wrong in those numbers, that's the idea. Open an issue.