DEV Community

supachai jaturaprom
supachai jaturaprom

Posted on Edited on

Same Model, 11 Harnesses: What the Harness Leaderboard v0.1.9 Tells Engineers

I did not run these trials. Every number below comes from the public Harness Leaderboard v0.1.9 (Hugging Face Space by Joel Niklaus), as I read it on 20 November 2026, including its Method tab. The board is updated often, so check the live page before you quote a number.

TL;DR

  • Same model, same weights, same tasks, 11 different coding-agent harnesses. Macro-average resolve rate runs from 42.7% (OpenClaw) to 52.5% (OpenCode), a 10-point spread. On a single model the spread reaches 22 points.
  • Only three harnesses sit on the accuracy-vs-cost Pareto frontier: OpenClaw (cheapest), Pi (cheapest per solved task) and OpenCode (most accurate).
  • Re-running the same harness already flips 24.1% of tasks. Swapping the harness flips 27.1%. Swapping the model flips 33.8%. The harness matters, but less than the model, and run-to-run noise is large.
  • The best harness depends on the model. Only OpenCode and Kimi Code are tied for first on all four models.
  • The most useful part of the board for engineers is the list of harness bugs it found: wrong default context windows, silent reply caps, compaction that never fires.

Harness Leaderboard v0.1.9 Pareto chart: macro-average resolve rate vs cost per trial for 11 coding-agent harnesses on the same open models

Chart: screenshot of the Harness Leaderboard v0.1.9 Pareto view (All models, All efforts). Chart and data belong to the leaderboard author.

Why this board is worth reading

Most leaderboards rank a model, or a model inside one fixed agent. In practice you pick both: a model, and the harness that drives it. The harness owns the system prompt, tool schemas, edit format, history compaction, retries and subagents. The Harness Leaderboard holds the model fixed and swaps only the harness, so the difference you see is the harness.

Setup

Harnesses (11, each pinned): Claude Code 2.1.300, Codex 0.162.1, Pi 1.2.0, OpenCode 2.2.6, Hermes Agent v0.22.1, mini-swe-agent 2.4.6, Kimi Code 2.2.0, Qwen Code 0.25.2, OpenClaw 2026.11.1, DeepSeek Harness 0.2.2-rc.1, Grok Build 1.0.77.

Models (4, all open):

  • GLM-5.3-Flash and DeepSeek-V4.1-Flash with a 1,048,576-token context
  • Flint and Ripple-65B-Preview with a 262,144-token context

The two context sizes are deliberate: the smaller window forces harnesses to compact history, so the board also tests compaction.

Benchmarks:

  • Terminal-Bench 2.1: 87 of 89 tasks (build, debug, configure in a terminal)
  • DeepSWE v1.1: 111 of 113 tasks (long-horizon features and fixes in real Go, Python, TS/JS and Rust repos)

Every harness runs every task three times per model and reasoning effort: 39,204 attempts on the board. Runs use HF Sandbox, HF Jobs and Harbor, and trajectories are published in a public bucket.

How they keep it fair

Held equal for every harness:

  • Model weights and reasoning effort. A metering proxy inside each sandbox enforces the effort on every model call and counts tokens, so cost and time are measured outside the harness.
  • Task text, time limit, internet access and scoring.
  • No per-task step, token or cost limit.
  • The model's context window, longest reply and image support are passed to every harness through its own settings, so no harness has to guess.

Left to the harness: prompts, tools, compaction, retries and subagents, as shipped.

Providers must pass an equivalence check before they serve trials: same weights and published precision, similar reasoning length, matching known-answer outputs, clean tool calls, and token counts that match billing. Cost is priced at the cheapest checked provider's list price, whichever provider served the trial.

Scoring:

  • Task score = share of attempts that pass. Benchmark score = mean over tasks. Headline = mean of the two benchmarks, with a 95% CI.
  • A paired bootstrap decides which harnesses are tied with the leader.
  • Infrastructure failures are re-run (up to twice) and excluded. Agent errors and timeouts count as fails.
  • Access to the answer counts as a fail even if the grader passed it: reading hidden tests or the reference solution, or fetching the upstream fix.

Results (All models, All efforts)

Harness Macro (95% CI) TB 2.1 DeepSWE $/trial $/solve
OpenCode 52.5% (49.8–55.3) 60.9% 44.1% $0.52 $1.11
Kimi Code 51.7% (49.0–54.4) 59.8% 43.6% $0.63 $1.37
Grok Build 50.3% (47.7–53.0) 57.1% 43.5% $0.66 $1.57
DeepSeek Harness 49.9% (47.2–52.7) 56.9% 43.0% $0.57 $1.31
mini-swe-agent 49.9% (47.2–52.6) 58.8% 41.0% $0.59 $1.41
Pi 49.4% (46.7–52.1) 57.4% 41.4% $0.48 $1.02
Hermes Agent 49.3% (46.5–52.1) 56.2% 42.4% $0.61 $1.35
Claude Code 48.8% (46.2–51.5) 55.2% 42.4% $0.63 $1.51
Codex 47.5% (44.8–50.2) 58.4% 36.6% $0.52 $1.23
Qwen Code 43.9% (41.3–46.5) 52.7% 35.1% $0.52 $1.33
OpenClaw 42.7% (40.0–45.4) 49.4% 36.0% $0.36 $1.13

By the paired bootstrap, only Kimi Code is tied with OpenCode (−0.9). Everyone else is behind the leader: Grok Build −2.2, DeepSeek Harness −2.6, mini-swe-agent −2.6, Pi −3.1, Hermes Agent −3.2, Claude Code −3.7, Codex −5.0, Qwen Code −8.6, OpenClaw −9.8.

The Pareto frontier

On accuracy vs cost per trial, three harnesses are not beaten on both axes:

  • OpenClaw: $0.36 per trial, 42.7%. Cheapest, least accurate.
  • Pi: $0.48 per trial, 49.4%, and the lowest cost per solved task at $1.02.
  • OpenCode: $0.52 per trial, 52.5%. Most accurate.

The other eight are dominated. On these numbers OpenCode is both cheaper and more accurate than Kimi Code, Grok Build, Claude Code and Hermes Agent. The cost range across harnesses is about 1.8x.

Harness effect vs noise

This is the section I would read first. The board counts a task as "flipped" when one attempt passes and another fails.

Comparison Flip rate (macro)
Same harness, re-run 24.1%
Swap the harness 27.1%
Same harness, swap the effort 27.1%
Same harness, swap the model 33.8%

Swapping the harness adds only 3.0 points of flips beyond re-run noise (+4.0 on Terminal-Bench 2.1, +2.0 on DeepSWE). So:

  • The model still matters more than the harness.
  • A single run tells you very little. If one attempt flips a quarter of tasks by itself, a one-off "agent A beat agent B" demo is mostly noise.

The winner depends on the model

  • Only OpenCode and Kimi Code are tied for first on all four models.
  • Hermes Agent is tied for first on GLM-5.3-Flash and Flint, but 11th on DeepSeek-V4.1-Flash.
  • Codex is tied for first on DeepSeek-V4.1-Flash and Ripple-65B-Preview, but 11th on GLM-5.3-Flash (47.4%).
  • On Ripple-65B-Preview, best vs worst is 22 points: Kimi Code 56.4% vs OpenClaw 34.4%.

Use the all-models average as a starting point. If you have already chosen your model, look at that model's column.

Harness bugs the board found

These are worth knowing even if you never look at the rankings, because they will bite you when you point a harness at a model it does not know:

  • OpenClaw asks for at most 8,192 tokens per reply when it is not told the model's longest reply, although its documentation says it then sends no limit. It also sends earlier reasoning back only to providers it recognizes, never to a custom OpenAI-compatible endpoint, unless you turn on a setting.
  • Codex assumes a 272,000-token window for unknown models. Its window setting cannot raise that, only a model catalog file can. It also cuts command output at 10,000 bytes.
  • Claude Code lowers its reply limit to the room left in the context only on Anthropic's servers.
  • DeepSeek Harness sets its compaction point from the model's longest reply, so when the longest reply equals the context window (Flint), automatic compaction never fires.
  • Grok Build sends images it opens to the model, even when the model is set up as text-only.

The lesson: when you run a harness against a model that is not its vendor's own, set the context window and max output tokens explicitly. Do not trust the defaults.

Caveats

  • Open models only. Claude Code and Codex scores here show how they drive GLM, DeepSeek, Flint and Ripple, not how they drive Claude or GPT.
  • Harnesses ran on different days. Provider speed changes day to day while the per-task time limit stays fixed, so a harness that ran on a fast day fits more calls into the limit.
  • Cost uses list prices. Vendor-specific cache discounts and subscriptions are not modelled.
  • Two benchmarks only. Nothing here covers network automation, infra ops or your own codebase.
  • The board is versioned and changing. These are v0.1.9 numbers.

What I take from it

  1. On open models, OpenCode and Kimi Code are the most consistent picks. If cost per solved task matters most, look at Pi.
  2. More expensive is not better. Grok Build and Claude Code have the highest cost per trial and neither is on the frontier.
  3. Before claiming one agent beats another, run it several times. Re-running the same harness already flips 24% of tasks.
  4. Always pass model limits to the harness yourself.
  5. If you are choosing for a team, run a small eval on your own tasks and track pass rate, cost per solve, and the share of tasks solved on all three attempts. Unattended work needs the last one.

Source


Thai version: https://dev.to/supachai/omedlediiywkan-11-harness-aan-harness-leaderboard-v019-aebb-engineer-37bb

Top comments (0)