62% vs 58% on the Same Benchmark: How Many Tasks Before the Gap Means Anything?

Two teams run the same agent benchmark on the same model with the same scaffold. Team A runs 100 tasks and reports 62%. Team B runs 250 and reports 58%. Both publish their task lists and pass rates. Neither is cheating.

Is the four-point gap real, or did the two teams stop at different points?

Wei-Jung Huang's paper (arXiv:2607.12338, 14 July 2026, KDD 2026 Agentic AI Evaluation and Trustworthiness workshop) answers this without running any new model. It shuffles the task order of records that three public agent benchmarks already published, then asks whether the decision after the first 15%, 25%, 50%... of tasks matches the decision after all of them.

Replay, not rerun

No new data, only a change in how much you see:

• SWE-bench Lite: 552 ordered comparisons (276 unordered pairs), median 300 shared tasks
• SWE-bench Verified: 156 ordered (78 unordered), median 496
• AppWorld: 306 ordered (153 unordered), 585 tasks, 36 packages (18 systems × 2 splits)
• tau-bench: 12 ordered (6 unordered), 278 entries per system over airline, retail, and telecom, with cost fields

Each pair gets 500 reveal orders and 200 bootstraps (tau-bench: 2,000 and 500). A real evaluation runs once in one order; replay estimates how often a comparison stalls if you stop early.

Three numbers that wear a percent sign

0 pp threshold: A wins if strictly above B on the completed task set, so one task can flip the whole decision. The strictest setting, no margin.

5 pp budget grid: a step size, not an effect threshold — 5% to 95% in 5-point increments. The paper separately tests 5 pp and 10 pp effect thresholds; don't merge them.

Unresolved rate: comparisons where the policy picks neither side and asks for more tasks or declines to conclude. Three gates hold together: at most 5% conditional decision error among decided comparisons, at most 5% coverage failure, at most 25% unresolved. That last is 25%, not 5%: being right when you decide doesn't help if you rarely decide.

Four benchmarks, four answers

At 0 pp on the 5 pp grid:

• AppWorld: 15% — about 88 of 585 tasks, ~21% unresolved; still 15% at 5 and 10 pp
• tau-bench: 25% — about 70 of 278 entries; 30% at 5 pp, 35% at 10 pp
• SWE-bench Verified: 90% — about 446 of 496, 24.22% unresolved
• SWE-bench Lite: 95% is still not enough. At 25%, error and coverage are in range, yet 55.92% of comparisons stay unresolved; at 95% it is 27.25%, above the ceiling.

What differs is the distribution of unresolved comparisons and coverage gaps: Lite decides accurately at 25% but leaves over half undecided; Verified only reaches 24.22% unresolved at 90%.

AppWorld arrives early because a coverage-aware bootstrap under difficulty grouping covers the required groups at 15%. A forced stratified policy needs 50%; forced uniform random fails coverage 99.96% of the time at 25%. On tau-bench, cheapest tasks first spends 11.51% of cost at 25% with 100% coverage failure.

Why the answers move

Five levers, none intrinsic to a benchmark: Lite has no sufficient budget at 0 pp but 55% at 5 pp and 35% at 10 pp; one-task slack on a coarse grid cuts its 0 pp budget to 75%, while repository × difficulty leaves more unresolved; low error often means few decisions rather than few mistakes; keeping one direction per unordered pair leaves 275 positives against 1 negative at 0 pp, so the paper targets ordered comparisons; and AppWorld's 15% moves to 10% or 25% as the truncation parameter a goes 0.10, 0.05, 0.02. A proportion is not a count: Lite's 95% of 300 is about 285 tasks, Verified's 90% of 496 about 446.

What this means when you read a paper

"We ran 100 tasks and A beat B by 0.5 points" is not a result. The paper's checklist asks five things:

1. What is the threshold, in percentage points?
2. How were tasks selected — random, stratified, cheapest first?
3. What is the coverage rule, and how many tasks per group?
4. What is the decision rule — bootstrap tail, paired normal, McNemar?
5. Is there a cap on unresolved comparisons, and is the actual rate reported?

Missing any one, "0.5 points higher" is "no conclusion yet": the same records go from no sufficient budget to 55% by changing the threshold.

What this paper cannot show

First, replay is not a new experiment: "It does not estimate performance on future tasks, because the replay orders permute a fixed completed record." It tells you how much of this batch was needed, not whether 15% survives a fresh batch.

Second, every conclusion depends on the threshold, coverage rule, grouping, and policy parameters. Move 0 pp to 5 or 10 pp and Lite goes from nothing to 55% or 35%; one-task slack on a coarse grid brings its 0 pp budget to 75%; AppWorld's 15% drifts from 10% to 25% with the truncation parameter. These are design choices, so a budget must travel with its direction and coverage rule.

Third, only three benchmark families are covered. OSWorld, WebArena, AgentDojo, AndroidWorld, Terminal-Bench, AgentBench, and MiniWoB++ were not replayed, and tau-bench's grid holds only 12 ordered comparisons (6 pairs). Don't extrapolate "AppWorld is 15%" to other interactive benchmarks.

Fourth, the public records are filtered leaderboard entries: SWE-bench keeps only near-complete entries covering at least 95% of a split, and AppWorld uses 36 public packages. Voluntary submissions, not random samples, so composition changes move the estimate.

Fifth, the cost findings hold only on tau-bench. Neither SWE-bench nor AppWorld publishes a cost field, so "cheap tasks first" is checkable on tau-bench alone.

Sixth, 5% decision error, 5% coverage failure, and 25% unresolved are the paper's shared targets, not an industry standard. Change the targets and the budgets change: a framework for computing a budget, not a universal "15% is enough".

One rule you can use now

When you see "we ran X tasks and A beat B by 0.5 points", check three things: is the decision threshold stated, were tasks selected and covered by repository, difficulty, or domain, and is the unresolved share reported. If any is missing, file it as "no conclusion yet". A task proportion is not a decision rule; without one it is just a cost figure.

More of my writing on AI evaluation: hhfa008.github.io/ai/writing

This is part two of a short series on reading AI evaluations. Part one: Same Model, Same Scaffold, 10 Runs.

Disclosure: drafted with AI assistance and edited by the author.