Agent Leaderboards Measure Score. We Added Price.

Agent benchmarks tell us who can solve a task. We built the missing layer: what each verified result cost.

Most leaderboard views answer only half the question

Agent benchmarks are everywhere now: SWE-bench, Terminal-Bench, OSWorld, GAIA. They show who scores highest — and some also expose cost at the run level.

But before hiring anyone — human or machine — the practical question is different:

What does it cost to get the task done?

Some of that data already exists. Princeton's HAL publishes recorded run costs alongside scores. Terminal-Bench 2.0 submissions include a result.json for individual task attempts, including dollar cost and pass/fail outcome.

What has been missing is a shared, task-level view that connects:

agent configuration → benchmark task → verified pass → recorded cost

We assembled that layer as part of the Agent Taxonomy Graph.

What we ingested

• HAL (Princeton): 46 evaluation runs across 9 benchmarks, with reported run-level cost.
• Terminal-Bench 2.0: 75 public submissions and 32,803 per-task result.json files, each carrying a recorded cost and pass/fail result.
• Curated benchmark observations: merged into a dataset of 131 evaluation runs and 29 model configurations with both a score and a price attached.

Every run keeps its source URL and verification status. This is intended as an evidence layer, not a scraped leaderboard table.

Scope: run-level comparisons are directional, not a normalized ranking. Benchmark mix, harness design, retry policy, context limits, and cost accounting differ across public evaluations. The cleaner comparison is at the level of the same benchmark task and the same verifier pass condition.

What the money says

The spread is the story.

Across the evaluated configurations, recorded cost per benchmark run ranges from roughly $0.03 to more than $1,600 — five orders of magnitude for systems that can appear side by side on agent leaderboards.

The current cost × score Pareto frontier, as of July 2026:

Configuration	Recorded cost per evaluation run	Avg score
DeepSeek-V3.2	$0.03	39.6%
GPT-5.1 Codex mini	$0.06	61.6%
GPT-5.3 Codex	$0.18	71.9%
Frontier mixes	~$1.87+	~87%

Read it bottom-up:

A configuration costing around $0.06 already clears 60%.

Moving from roughly 62% to 87% score costs around 10–30× more in this sample.

That does not prove that cheap agents plus retries always beat a more expensive single pass. Reliability, verifier cost, retry budgets, and failure modes still matter.

But it establishes the question that should be measured:

What is the cost per accepted result, not just the cost per attempt?

For tasks with cheap verification and room for retries, lower-cost configurations may be much more competitive than leaderboard rankings alone suggest.

Per-task is where it gets interesting

Aggregate scores hide the strongest signal.

When the benchmark task and its verifier are held fixed, the pricing spread becomes hard to ignore:

• pytorch-model-cli — 47 recorded successful runs, priced from $0.001 to $1.47. Same benchmark task, same verified pass condition, 1,109× price difference.
• qemu-startup — 48 recorded successful runs, with an 810× cost spread.
• log-summary-date-ranges — 58 recorded successful runs; the cheapest, Terminus 2 + DeepSeek-V3.2, cost roughly half a cent.

These are successful runs, not deduplicated configurations. The same agent × model pair can appear more than once because each trial takes its own trajectory: steps, retries, tokens, and tool use.

They are not identical agent trajectories. But they are comparable verified completions of the same externally checked task.

We have not found a common public price layer for that result.

A reader can see that one configuration passed. They can see that another scored higher overall. What they usually cannot see is that both passed the same task — and one paid three orders of magnitude less.

That is the gap.

Method, in plain terms

The current dataset joins public evaluation artifacts into a common graph:

• Unit of analysis: a recorded evaluation run or task attempt
• Cost: reported API spend in the source artifact
• Success: benchmark verifier pass
• Coverage: 131 evaluation runs, 29 model configurations, and 31,913 extracted per-task price rows
• Evidence: source URLs and verification status retained per run
• Not yet normalized for: infrastructure cost, harness differences, token accounting differences, retry policy, context limits, or reliability across repeated runs

The purpose is not to declare a universal “best agent.”

It is to make the cost of a verified outcome visible enough to compare.

The price graph is live

The 31,913 extracted task-level price rows are no longer just feedstock. The task-anchored price view now ships on the ATG dashboard as a dedicated screen.

For each of the 87 out of 90 Terminal-Bench 2.0 tasks with at least one solved, priced run, it shows:

• the cheapest and median cost of a verified pass;
• the full min→max price range on a shared log scale;
• the solve rate across all recorded attempts;
• the 20 cheapest recorded successful runs, linked to their source evidence.

The hardest priced task in the dataset, make-doom-for-mips, was solved by only 12 of 358 attempts.

The dashboard also plots cheapest verified-pass cost against the number of distinct configurations that managed to solve each task. That makes the trade-off visible: some tasks are cheap once solved, while others remain expensive because only a small number of configurations can complete them at all.

The next step is publishing the same structure as a knowledge graph: each canonical task becomes a node, and every priced verified pass becomes an edge weighted by recorded cost and linked back to its evidence.

The result is not just another leaderboard.

It is a public price layer for the agent market:

Which configurations can complete this verified task — and what did each one pay?

The practical view is live on the ATG dashboard: 133 agents, verified capabilities, protocols — and now per-task prices.

The dashboard is the analytical view.

Geo Protocol is the provenance layer for the Agent Taxonomy Graph: agent systems, capabilities, benchmark observations, and their evidence are published as queryable entities rather than static rows.

The current ATG hub and this research post are live on Geo Protocol: the post itself and the Agent Taxonomy Graph hub.

The next publication step is to add task-level price observations to that graph: a canonical benchmark task as a node, and each verified priced result as an evidence-linked relation.

Feedback: @maximl