I've put our benchmark numbers on Hugging Face, as an open dataset with a small leaderboard Space on top, so anyone can pull the rows and argue with them.
The top of Forge 2.0 is the interesting bit. Jev Router sits first with a median of 0.746, and OpenRouter's generation records show 117 of its 118 calls went to GPT-6.1 Sol. Sol on its own is third, at 0.5933. Same model doing nearly all the work, and a better result. It isn't a clean comparison. Jev and Sol sit in different comparison groups on our board, Sol's builds ran five days earlier on an older goose build, and the router also picks a reasoning effort for each call. I wouldn't call it settled on two builds each anyway. The full write-up is on our blog.
What's in it
94 rows today, one per model per board, across three boards. Gauntlet 7.2 has the model build a payments app. Forge 1.0 and Forge 2.0 have it build or upgrade an Atlassian Forge app for Jira. Each row has the score, the per-build scores, the inference cost where we recorded one, and a link back to the run on leanzero.net, where the checks, screenshots and, where we recorded one, the video are.
It syncs from the site on its own, every three hours. The ranking comes from the site's own code, so the Space and leanzero.net show the same order. A check this morning caught the dataset one site deploy behind (fixed, and the sync now checks the whole board).
Read it before you trust it
Gauntlet and Forge 1.0 rows are one run each, so there's real run-to-run noise, and I wouldn't read much into two models a few hundredths apart. On Forge 2.0 a row is the median of two independent builds. Those two can be far apart. gemini-4-argon scored 0.7926 and then 0.2943. Cost is the inference spend we recorded. Only one Gauntlet row has a cost so far. The Forge 1.0 figures are lower bounds because the billing record was incomplete. Most Forge 2.0 figures are complete.
If you want to poke at it:
import pandas as pd
url = "https://huggingface.co/datasets/Mihai-LeanZero/leanzero-agentic-benchmarks/resolve/main/data/leaderboard.jsonl"
df = pd.read_json(url, lines=True)
f2 = df[df.version == "forge-2.0"].sort_values("rank")
f2["build_scores"] = f2["build_scores"].apply(lambda b: [round(x, 4) for x in b])
print(f2[["rank", "model_name", "score", "build_scores"]].head(10).to_string(index=False))
The part I want your take on
These aren't SWE-bench or Terminal-Bench style tasks. They're long agentic builds of a real app, scored by checks we wrote ourselves, so they have our blind spots too. If you run agents on real work, does the order on these boards match what you see day to day? Or is there a model you'd swear sits too high or too low? I'd like to hear which one and why.
Dataset: Mihai-LeanZero/leanzero-agentic-benchmarks
Leaderboard: the Space
Full results and evidence: leanzero.net/agentic-benchmarks
Top comments (0)