No synthetic puzzles, no contaminated suites: 25 tasks harvested from PRs merged in 2026 across five languages, graded by each project's own held-out tests. Four agents at stock settings. octomind with an open model solved 24/25 - ahead of Claude Code with Opus - while the same model in another harness solved 19 at double the cost. Here's how we built the benchmark and what it taught us.
We wanted one number we could actually trust: if you hand a coding agent the kind of task a maintainer faces on a normal Tuesday - a real bug, a real feature request, in a real codebase - how often does it deliver a fix the project's own test suite accepts?
None of the public benchmarks could give us that number, so we built octobench. This post is the story of how, what broke along the way, and what the scoreboard says.
Why not an existing benchmark
Two problems kept biting us with the popular suites.
Contamination. The well-known benchmarks are old enough that frontier models have seen the fixes - sometimes literally, commit by commit - in training data. At one point we caught a model running git show on a commit hash it had no business knowing about. Whatever that measures, it isn't problem-solving.
Taste-grading. Many benchmark tasks are graded by tests that assert an implementer's arbitrary choices: an internal variable name, the exact wording of an error message. An agent can write a maintainer-grade fix and fail because it phrased an error differently than the original author. That measures mimicry, not engineering.
So the design goals were: real tasks, fresh tasks, fair grading.
25 tasks, 5 languages, mostly newer than the models
We harvested every task from a real, recently-merged pull request in a respected open-source project: werkzeug, click, anyio, pydantic, twig, carbon, symfony, guzzle, commonmark, uuid, rayon, chrono, bytes, serde-json, fmt, yaml-cpp, catch2, spdlog, eslint, fastify, undici, pino, pino-pretty.
That list is deliberate on two axes. Languages: five of them - python, php, rust, c++, js - because an agent that's great at python and lost in a CMake build is not a general coding agent. Freshness: every fix was merged in 2026, mostly after current model training cutoffs - several within days of harvesting. One case, chrono's reversed date iterators, was merged the same morning we picked it. A benchmark you can re-harvest as cutoffs advance is a benchmark that can't go stale.
The scenario mix runs from one-line crash fixes to multi-file features, and each case reconstructs the moment before the fix existed:
- setup.sh checks out the repository at the commit before the fix - as a single-commit shallow clone, so the answer isn't hiding in the git object store - prepares the toolchain, and removes the git remote so the agent can't just fetch the upstream fix.
- The agent gets a task prompt and works in the repo like a hired contractor.
- validate.sh runs after the agent finishes: it fetches the merged fix's test files - which the agent has never seen - overwrites whatever the agent may have done to the test suite, and runs exactly those tests. The project's own tests, written by the project's own maintainers, decide pass or fail.
- An LLM judge separately grades work quality from the diff and logs.
Before any case entered the benchmark it had to prove itself fail-to-pass: held-out tests must fail on the pre-fix code and pass with the real merged fix applied. No proof, no case.
Writing prompts that don't cheat - in either direction
For 20 tasks we ran a dedicated agent (octomind's developer:reverse-spec) over each merged commit to reconstruct the request that plausibly produced it, then curated by hand with one rule - the derivability rule: everything the hidden tests assert must be derivable from the prompt. Bug fixes got a short informal prompt, because any correct fix passes. Features whose tests pin public API names or exact output formats got a spec-tight version, because a real requester would state a wire format.
For the other 5 - one per language - we used the actual GitHub issue text, verbatim, trimmed only of fix-leaking sections (one author had helpfully written "I have a PR ready that does X"). That tests something different: turning a user-shaped bug report, sometimes with a screenshot instead of expected output, into a maintainer-grade fix.
A few favorites, by the skill they isolate:
- anyio's cancel-scope spin - a 4-line fix that requires understanding why an asyncio event loop pins a CPU core. Tiny diff, deep async reasoning.
- guzzle's cookie prefixes - 27 lines with three independent traps; the naive startsWith("__Secure-") && !secure fix fails half the tests.
- commonmark's fenced-code tabs - a user reported "code blocks sometimes lose their first character" with a screenshot. The held-out fixtures include cases the issue never mentions - only a true root-cause fix passes, a symptom-patch fails.
- pino-pretty's control characters - two hidden tests exist purely to punish over-broad sanitizing: they pass on the broken code and fail any fix that doesn't respect the trust boundary.
Eating our own dogfood
Before the final run, three reviewers read every held-out assertion against every prompt. The audit caught real problems - in our benchmark, not the agents. One case's tests asserted the exact prose of an error message the issue never quoted; two frontier agents produced perfect fixes (one even picked the same error-code number as the maintainer) and both failed on wording. We replaced the case. Another case's hidden tests contradicted its own issue text. A third had its held-out tests inside a source file, so restoring gold tests wholesale would silently revert a correct fix.
The meta-lesson: benchmark infrastructure fails in ways that look exactly like model failures. Every anomaly we chased - a judge scoring 0 on a passing run, a "solved" case with an empty diff - deserved a real root-cause, and about half were ours.
The scoreboard
Four agents, each at its stock, out-of-the-box single-agent invocation. No tuning, no custom prompts.
| Agent | Solved | Judge Σ / 2500 | Cost | Wall Time |
|---|---|---|---|---|
| octomind + glm-5.2 | 24/25 | 2264 | $63.43 | 3.6h |
| claude code + claude-opus-5 | 23/25 | 2262 | $81.79 | 6.7h |
| codex + gpt-5.6-sol | 21/25 | 2127 | $14.86 | 1.0h |
| opencode + glm-5.2 | 19/25 | 2093 | $129.54 | 3.3h |
Four things the table says:
The harness matters as much as the model. octomind and opencode ran the same model on the same endpoint at the same prices - a pure harness A/B. octomind solved 24, opencode 19, at half the cost. The difference is context discipline (opencode pushed 30M input tokens through one task where octomind needed 8M) and supervision: the cases opencode dropped are exactly the deep-root-cause, multi-trap ones where an unsupervised agent declares victory too early.
An open model beat Opus while paying full price for every token. glm-5.2 ran via Ollama cloud, which has no prompt caching - every agent turn re-bought its full context at list price. Opus, meanwhile, billed ~97% of its context re-reads at 1/10 cache rates. glm's dollar figure is its worst case, and it still came out ahead on solves, cost, and time. On any cache-enabled endpoint the cost gap becomes a chasm.
Speed has a thoroughness tax. Codex is remarkable - 2-4 minutes per case, $0.59 median - but all four of its failures are cases that punish not checking one more caller, one more surface, one more trap.
No task was impossible. Every case was solved by at least one agent, and the failure classes were clean: the trust-boundary case caught opus, codex, and opencode; the deep root-cause parser case caught everyone except octomind; and the one case both glm harnesses failed identically is a genuine model blind spot, not a harness artifact.
Where the harness edge comes from
A big share of "context discipline" is simply how the agent finds code. Stock claude code, codex, and opencode navigate a repository the same way: grep, read the file, grep again, read more - and every re-read is context the model pays for on every subsequent turn. octomind's developer:general ships with octocode out of the box: structural search that finds the exact symbol, signature, or code pattern directly, so the agent jumps to the three functions that matter instead of paging through files that don't. Enable semantic indexing and it also searches the codebase by meaning - "where is cookie validation handled" as a query, not a grep pattern. Fewer wrong files read is fewer tokens re-bought, and that compounds into the 8M-vs-30M input-token gap the A/B exposed.
The other share is supervision. During dry runs an agent once "finished" a task by announcing "Now let me create a plan and implement the fix" - and stopping, with a zero diff. In a non-interactive run a text-only turn ends the session, so octomind added a deterministic guard: if a turn ends with no action while the agent's self-reported status is still in progress, it gets sent back to work. In the final run that failure mode was gone - and the cases opencode dropped are precisely the ones where nothing questioned an early declaration of victory.
Reproduce it
Everything is public and pinned: the full per-case table and reproduction guide at the exact commit of this run, per-case agent traces and judge verdicts in the run artifacts, and the case-harvesting pipeline - the reverse-spec agent, the derivability rules, the fail-to-pass verifier - documented so the set keeps growing as training cutoffs advance.
The result we care most about isn't the ranking. It's the A/B: same model, same endpoint, five more tasks solved at half the cost. Model quality is table stakes now - the harness is where the leverage is. That's the thesis octomind is built on, and now we have the number.
Top comments (0)