I Tried to Verify an AI Agent Benchmark. Here's the Bundle I Wish Everyone Shipped

benchclawio — Fri, 14 Aug 2026 06:51:35 +0000

Nearly every AI agent benchmark you read is unfalsifiable.

Not wrong, necessarily - unfalsifiable. There's a blog post with a bar chart, a claim that framework A beat framework B, and no way for you to check it. No run count. No model version. No raw output. Often no cost. You are asked to trust a summary statistic produced by people with an interest in the result.

We publish agent benchmarks, so this is our problem too. This post is about the evidence bundle we settled on, and how you can pull one down and take it apart in about two minutes. Every command below is one I actually ran while writing this, with its real output pasted in.

The claim we're going to try to break

From one of our pilot runs: LangGraph 1.2.9 and Pydantic AI 2.13.0 both completed 20 of 20 tasks under gpt-4o, at a total spend of $0.094275.

That's the sort of sentence you'd normally have to take on faith. Let's not.

Two minutes to verify it yourself

The bundle is a directory in a public repo. Pull it:

BASE="https://raw.githubusercontent.com/benchclawio/harness/main/results/gpt-4o-vs-gpt-4o-mini-tool-calling-2026-07-24"
for f in SHA256SUMS README.md gpt4o-pilot-manifest-v0.4.0.json \
         scored-pilot-gpt4o-raw-2026-07-24.jsonl \
         scored-pilot-raw-2026-07-24.jsonl \
         scored-pilot-analysis-2026-07-24.json \
         scored-pilot-gpt4o-analysis-2026-07-24.json \
         real-pilot-status-manifest-v0.3.0.json; do
  curl -sfO "$BASE/$f"
done

First question: is this the same data we published, or has something drifted?

sha256sum -c SHA256SUMS

README.md: OK
gpt4o-pilot-manifest-v0.4.0.json: OK
real-pilot-status-manifest-v0.3.0.json: OK
scored-pilot-analysis-2026-07-24.json: OK
scored-pilot-gpt4o-analysis-2026-07-24.json: OK
scored-pilot-gpt4o-raw-2026-07-24.jsonl: OK
scored-pilot-raw-2026-07-24.jsonl: OK

That's the cheapest integrity control there is and almost nobody ships it. It costs one line in your run script and it means a reader can tell the difference between the file you published and a file someone edited afterwards.

Now the actual test. Ignore our summary entirely and recompute it from the raw runs:

import json, collections

rows = [json.loads(l) for l in open('scored-pilot-gpt4o-raw-2026-07-24.jsonl')]

by = collections.defaultdict(lambda: [0, 0])
for r in rows:
    b = by[r['subject']]
    b[0] += 1
    b[1] += r['completed']

for s, (n, ok) in sorted(by.items()):
    print(f"{s:<32} {ok}/{n}")
print("total cost  $", round(sum(r['metrics']['cost_usd'] for r in rows), 6))

langgraph_1_2_9_gpt4o_live       20/20
pydantic_ai_2_13_0_gpt4o_live    20/20
total cost  $ 0.094275

The claim reproduces to the cent, from the raw file, without going through our analysis code. That is the whole point of the exercise. Our published scored-pilot-gpt4o-analysis-2026-07-24.json says total_cost_usd: 0.094275 and total_runs: 40 - and now you know the summary wasn't doing anything clever on the way there.

The part that actually earns trust: finding the failure

A benchmark where everything passes tells you very little. The same four tasks under gpt-4o-mini:

import json, collections

rows = [json.loads(l) for l in open('scored-pilot-raw-2026-07-24.jsonl')]

by = collections.defaultdict(lambda: [0, 0])
for r in rows:
    b = by[r['task_id']]
    b[0] += 1
    b[1] += r['completed']

for t, (n, ok) in by.items():
    print(f"{t:<32} {ok}/{n}{'  <-- FAILED' if ok == 0 else ''}")

inventory-reorder                10/10
dependent-shipping-quote         10/10
recover-stale-revision           10/10
refund-policy-minimal-tools      0/10  <-- FAILED

Zero out of ten. Now the question a raw bundle can answer and a bar chart cannot: what kind of failure was it?

bad = [r for r in rows if r['task_id'] == 'refund-policy-minimal-tools']
print('status  :', set(r['status'] for r in bad))
print('failure :', set(str(r['failure']) for r in bad))
print('tool calls:', set(r['metrics']['tool_calls'] for r in bad))

status  : {'failure'}
failure : {"{'type': 'invalid_final_answer', 'stage': 'scoring',
            'message_sanitized': 'final output does not exactly match expected output'}"}
tool calls: {2}

That is the interesting shape. stage: scoring, not stage: execution. The agent made both tool calls it was supposed to make, raised no exception, hit no timeout, and returned a confident, well-formed, wrong answer. (For the curious: it computed a 19-day window inclusive where the policy needed 18 exclusive, then applied a correct eligibility rule to a wrong number.)

If all we had shipped was "75% completion rate", you would have no way to tell that apart from a crash loop or a rate limit. The failure taxonomy is doing real work here, and it's three fields.

The wart I found writing this post

Honesty is cheaper than getting caught. While writing the snippets above I hit this:

m = [json.loads(l) for l in open('scored-pilot-raw-2026-07-24.jsonl')]
g = [json.loads(l) for l in open('scored-pilot-gpt4o-raw-2026-07-24.jsonl')]
print('mini keys:', sorted(m[0].keys()))
print('4o   keys:', sorted(g[0].keys()))

mini keys: ['completed', 'failure', 'metrics', 'model_id', 'run_index', 'status', 'subject_id', 'task_id']
4o   keys: ['completed', 'failure', 'metrics', 'model_id', 'run_index', 'score', 'status', 'subject', 'task_id', 'wall_time_outer_s']

Two files in the same bundle, and one calls it subject_id while the other calls it subject. The gpt-4o file also gained score and wall_time_outer_s. That's schema drift between two runs a few hours apart, and it's why the two snippets above key on different field names - which I'd rather explain than quietly paper over.

It doesn't invalidate anything: the numbers are the numbers, and the checksums prove the files haven't moved since. But it's a real defect, it's the kind of thing a raw bundle exposes and a summary hides, and the fix is boring - a schema_version on every row and a validator in CI, which is what we should have had.

You are going to find this in your own bundles the first time someone reads them properly. Better that than nobody ever reading them.

What actually goes in the bundle

Stripped down, here's what we ended up shipping per run set, and why each piece is there:

File	Job
`-raw-.jsonl`	One JSON object per run. The source of truth. Everything else is derived.
`-analysis-.json`	Our summary. Included so you can check it against the raw.
`-manifest-.json`	What we declared before running: model, temperature, task suite, run count.
`SHA256SUMS`	Integrity. One line to generate, ends an entire category of argument.
`README.md`	What this run was and - importantly - what it is not eligible to claim.

The manifest is the one people skip and it's the one that matters most. Ours pins model_id, temperature: 0, parallel_tool_calls: false, and the per-token pricing used to compute cost. Declaring the run count before execution is what stops a benchmark quietly becoming "we ran it until it looked right".

And the README carries the eligibility verdict. That pilot is explicitly marked not publication-eligible - the task suite changed mid-run, and there was an OOM restart. We still published the bundle, because a flawed run you can inspect is worth more than a clean-looking one you can't.

The checklist, if you're building one

Nothing here is hard. It's about ten lines of extra code in your run loop:

One row per run, JSONL, written as you go. Not at the end - you want the partial data when something dies at run 32 of 40.
Record tokens, cost, wall time, tool calls per run, not just the aggregate.
Taxonomise failures with a stage. execution vs scoring is the single most informative bit you can store.
Write the manifest before the first run, and include the declared run count.
Ship SHA256SUMS.
Version your row schema, which I clearly need to take my own advice on.
State what the run may not be used to claim.

If a benchmark you're reading doesn't ship these, that isn't proof it's wrong. It just means neither of you can find out.

The full harness, the task suite and every bundle mentioned here are at
github.com/benchclawio/harness. Our write-up of the
harness itself is at benchclaw.io/harness, and the methodology
we hold ourselves to - including what disqualifies a run from publication - is at
benchclaw.io/methodology.

If you spot something wrong in those numbers, that's the idea. Open an issue.