- Book: Prompt Engineering Pocket Guide
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
Claude Opus 4.6 scored 58.0% on Terminal-Bench 2.0 with Claude Code driving it, according to the Terminal-Bench 2.1 release notes. Inside Meta-Harness, an agent scaffold published by the stanford-iris-lab group on GitHub, the same model scored 76.4% on the same 89 tasks, five trials each.
That is an 18-point swing, and the model never changed. Meanwhile the thread in your team chat is still about Opus vs GPT vs Gemini.
The model is one dial. The harness around it, meaning the tools it gets, the way it edits files, what goes into its context and how it recovers from a bad turn, is another dial. On public agent leaderboards, the second one moves scores about as much as the first. So the benchmark you should trust is your agent on your repo.
What Terminal-Bench actually ranks
Terminal-Bench gives an agent a task in a container and checks the final state with tests. The leaderboard doesn't rank models. It ranks agent-model pairs. The Terminal-Bench 2.1 leaderboard, as published on Snorkel AI's site, lists 18 entries, and five models appear under two different agents. Each pairs the vendor's own CLI (Claude Code, Codex CLI or Gemini CLI) with Terminus 2, the harness the leaderboard credits to the Terminal-Bench team:
| Model | Vendor's own agent | Terminus 2 |
|---|---|---|
| Claude Fable 5 | Claude Code: 83.8% ±1.2 | 80.4% ±1.2 |
| GPT-5.5 | Codex CLI: 83.1% ±1.1 | 78.0% ±1.2 |
| Claude Opus 4.7 | Claude Code: 68.9% ±1.4 | 66.1% ±1.4 |
| Gemini 3 Pro | Gemini CLI: 65.8% ±1.4 | 73.9% ±1.3 |
| Gemini 3.1 Pro | Gemini CLI: 65.8% ±1.7 | 65.6% ±1.7 |
Row by row, the vendor agent doesn't always win. Claude Code and Codex CLI beat Terminus 2 with their own models, by 2.8 to 5.1 points. Gemini 3 Pro scored 8.1 points higher in Terminal-Bench's generic harness than in Google's own CLI. Gemini 3.1 Pro came out level in both.
Compare that 8.1-point harness gap to a model upgrade on the same board. Inside Claude Code, going from Opus 4.7 to Opus 4.8 moved the score from 68.9% to 78.9%. That is 10 points for a model upgrade against 8.1 for a different wrapper around one model, and 8.1 is the biggest such gap on this board.
The gap is as big as a model generation
Two rows are an anecdote. A survey paper from June 2026, From Question Answering to Task Completion, went through the Terminal-Bench 2.0 leaderboard and the public submission logs (Section 7.4). Across the 20 models with at least three harness results, the median gap between a model's best and worst harness was 13.6 points. For scale, the authors note that a stronger model generation often adds more than 10 points inside a fixed harness.
Their table for Claude Opus 4.6 on Terminal-Bench 2.0 shows how wide it gets:
| Harness | Score | Input tokens (K) | Timeout rate |
|---|---|---|---|
| Meta-Harness | 76.4% | 755.0 | 7.9% |
| Terminus-KIRA | 74.7% | 618.6 | 18.2% |
| Terminus 2 | 62.9% | 79.4 | 18.2% |
The same paper puts Gemini 3.1 Pro anywhere from 59.4% to 80.2% and GPT-5.3 Codex from 64.7% to 78.4%, depending on the harness.
The authors are careful here, and you should be too: "The resulting evidence is not a randomized ablation." Public submissions can differ in prompts, versions and budgets. Some of an 18-point gap may be reasoning effort or token budget rather than pure harness design. Part of Claude Code's 58.0% also came from broken tasks that 2.1 later fixed. If budgets and prompts move the score that much, they are part of what you are choosing when you pick an agent, and none of them are the model.
What a harness changes
Tools. mini-swe-agent is "some 100 lines of python" and "does not have any tools other than bash." Terminus 2 gives the model a single tool too, but it is an interactive tmux session where the model sends keystrokes, scrolls output and can open extra shells. Terminus-KIRA, built on Terminus 2, swaps its JSON/XML parsing for the model's native tools parameter. The model gets the same task either way; only the way it is asked to act differs.
Context. Meta-Harness extends KRAFTON's Terminus-KIRA. Before the agent loop starts, it collects a snapshot of the sandbox (working directory, files, available languages, package managers, memory) and puts it in the first prompt. Its README says that saves 2 to 5 early exploration turns. Terminus 2 summarizes the middle of the history when free tokens drop below a threshold (8,000 by default) and keeps the system prompt and task intact. Look at the input-token column above: Meta-Harness fed Opus 4.6 roughly ten times the input of Terminus 2.
Edit format. aider's leaderboard records which edit format each run used. Qwen2.5-Coder-32B-Instruct has two runs there: 16.4% returning whole files and 8.0% returning diffs. The diff run went through a different hosting provider, and only 71.6% of its edits were well-formed, so edit format is not the only variable. Same weights, half the score. Aider's own edit format docs say it plainly: "Different models work better or worse with different edit formats."
Recovery. Timeouts, retries and when to give up are harness decisions. In the survey's table, Terminus-KIRA timed out on 18.2% of Opus 4.6 trials and Meta-Harness on 7.9%.
Even the harness ranking moves
Terminal-Bench 2.1 fixed 28 of the 89 tasks in 2.0: dependencies that had changed, resource budgets too small for valid approaches, instructions that didn't match their tests. Claude Code with Opus 4.6 went from 58.0% to 70.1%, the largest gain the release notes report. Terminus 2 with Opus 4.6 scored 63.8% on 2.1.
Put that next to the survey's 62.9% for Terminus 2 on 2.0. On the old task set, Claude Code trailed Terminus 2 with the same model. On the fixed task set, it led by 6.3 points. The model was the same; only the tasks changed, and the lead flipped. So part of the 18-point gap in the opening was broken tasks hitting one harness harder than another. That is still a harness effect, and it is the kind you won't see until you run your own tasks.
Your repo is another task set. It has its own build quirks, its own test runner and its own idea of a clean diff. A harness that wins on 89 containerised terminal tasks has not been tested on any of that.
If you do want a clean model ranking, Vals AI's Terminal-Bench 4.0 board runs every model through mini-swe-agent with "a single tool, bash." As of October 16, 2026, Claude Opus 5.5 leads it at 65.15%, and Vals notes its 4.0 scores are not directly comparable to 2.1, where Vals's own runs used Terminus 2. It is a fair model comparison, but unless a 100-line bash loop is the agent you code with every day, it isn't your setup.
Race two harnesses on your repo
The useful experiment holds the model still and swaps the harness. Pick the model you already pay for and run it in two agents on the same ticket. OpenCode can sign in to OpenAI with a ChatGPT Plus or Pro plan or an API key, so GPT in Codex vs GPT in OpenCode is one pairing. For Claude in OpenCode, use an Anthropic API key: OpenCode's docs say Anthropic "explicitly prohibits" using Claude Pro/Max plans through third-party plugins.
Give each agent its own branch and folder, cut from the same commit:
git worktree add -b race/claude-code ../race-claude-code
git worktree add -b race/opencode ../race-opencode
Start one agent in each folder with the identical prompt. When both stop, compare what each one left behind. A new worktree has no node_modules, so the loop installs before it tests. Swap npm ci && npm test for your own install and test commands:
for wt in ../race-claude-code ../race-opencode; do
echo "== $wt"
git -C "$wt" status --short
git -C "$wt" diff --stat main
(cd "$wt" && npm ci && npm test) && echo PASS || echo FAIL
done
git diff --stat main covers tracked changes whether the agent committed them or not, and status --short catches new files. Then score each run on what you care about:
- Do the tests pass without you touching anything?
- How many files and lines changed, and were they the right ones?
- How many times did you have to step in or answer a question?
- What did the run cost in tokens or dollars?
- Did it edit lockfiles, CI config or files outside the task?
Do it on three to five real tickets, not one. Official Terminal-Bench 2.0 submissions run five trials per task (survey, Section 7.4) for a reason: one run on one ticket tells you very little.
Running the race in Hermes IDE
I build Hermes IDE, so weigh this section with that in mind. You don't need it for the experiment: the two git commands above do the isolation. Hermes sets up both worktrees and shows the runs side by side.
Press ⌘N (Ctrl+Shift+N on Windows and Linux) and each task starts on its own hermes/<name> branch in its own worktree, with the task as the agent's first prompt. New worktrees get their dependencies in seconds. You pick the agent, model, effort and account per task. The agent runs its own interface in a real terminal, so you are testing that agent's harness and not a Hermes copy of it.
The ⌘N agent picker in Hermes 2.3.4: Claude Code and Codex installed, OpenCode, Aider, Antigravity CLI and the other supported agents listed, plus Custom agent.
Start the ticket in Claude Code, then press ⌘N again and start the same ticket in OpenCode with the same Claude model, on an API key per the note above. If you're testing GPT instead, race Codex against OpenCode. Split panes put both sessions side by side, and the sidebar shows each session's status, context use, cost and branch. Spend is what the agent reports, or an estimate from its transcript marked as estimated.
When both are done, the Review Desk (⌘G, Ctrl+Shift+G on Windows and Linux) shows each run's changes by turn or by file, so you can see how each harness got there. Land the winner. The other branch is plain git, so you can delete it or keep it for the next comparison.
Hermes 2.3.4: Codex and Claude Code side by side, each in its own hermes/ worktree, with status, context use, cost and branch per session in the sidebar.
The judging is still yours: your tests and your read of the diff. Hermes is free, source-available under BSL 1.1, and runs on macOS, Windows and Linux. It is not affiliated with Nous Research, Anthropic, OpenAI, Google or any other agent vendor named here. Download it or read the code on GitHub.
Next time someone asks which model is best, ask them which harness they ran it in. Then ask what it scored on their repo.
If this was useful
A lot of what separates Meta-Harness from Terminus 2 is what reaches the model and when: a snapshot of the environment in the first prompt, a summary when the window fills, the instructions wrapped around each tool. That is prompt and context engineering, and it is the part of the harness you control in your own AGENTS.md and system prompts. My Prompt Engineering Pocket Guide is the short read on it. If you run the race above in Hermes IDE and something doesn't behave the way I described, open an issue on GitHub.
Top comments (0)