Update — treat this as a historical data point, not a current claim. Two things have superseded it. Terminal-Bench has moved to 4.0, which recalibrated task resources, dropped saturated tasks and standardised an 8-hour agent timeout — 2.x scores are not comparable to it. And the headless runner that produced these numbers dispatched reasoning one notch below each model's declared ceiling, while published leaderboard entries run the ceiling. Both are fixed; the run has not been repeated.
The caveat in the post below still stands and matters more than the result: a 3-task gap at k=1 is within noise, and this does not show Atlarix is ahead of opencode. There is also no Atlarix SWE-bench number — if you have seen one attributed to Atlarix, it is invented. Current results and methodology: atlarix.dev/benchmark.
I build Atlarix, an agent workstation for open-weight models. The core claim behind it is that the harness — retrieval, tool surface, control loop — is what lets an open-weight model perform, not just the model's raw weights. This post is me trying to falsify that claim with a controlled run, and publishing every output file so you can check it.
Short version: on Terminal-Bench 2.0, single attempt, Atlarix resolved 42/89 and opencode resolved 39/89 on the same model. That 3-task gap is within k=1 noise — I'm not claiming a win. What it shows is that the harness isn't bottlenecking the model. Details and caveats below; raw files at the end.
The experiment
The only variable is the harness. Everything else is pinned identical across both agents.
-
Benchmark:
terminal-bench/terminal-bench-2— all 89 tasks, one isolated container each, automated verifiers. -
Model:
minimax/minimax-m3, routed through OpenRouter, pinned to a single provider at fp8 — identical for both harnesses. -
Infrastructure: Harbor on Modal (
-e modal), one container per task. -
Attempts: single attempt,
-k 1. -
Timeout: native,
--timeout-multiplier 1(same for both). -
Retries:
--max-retries 3(same for both). - Tool calling: native function-calling forced, no text-tool shim.
Commands
# Atlarix harness
harbor run -d terminal-bench/terminal-bench-2 \
-m openai/minimax/minimax-m3 \
-n 24 -k 1 -y --timeout-multiplier 1 --max-retries 3 \
-e modal --agent-import-path atlarix_tb:AtlarixAgent
# opencode harness (same model + provider + infra)
harbor run -d terminal-bench/terminal-bench-2 \
-m bench/minimax/minimax-m3 \
-n 24 -k 1 -y --timeout-multiplier 1 --max-retries 3 \
-e modal --agent-import-path atlarix_tb.opencode_proxy:BenchOpenCodeAgent
(-n 24 is concurrency — how many containers run in parallel — not a task count. All 89 tasks run.)
Results
| Harness | Resolved | Score |
|---|---|---|
| Atlarix | 42 / 89 | 47% |
| opencode | 39 / 89 | 44% |
Read this before you read the table
k=1 means one sample per task. The official Terminal-Bench leaderboard requires k=5 specifically to measure run-to-run variance. A 3-task difference at k=1 is inside that noise band. So this is not a leaderboard result and not a claim that Atlarix beats opencode. The honest takeaway: an open-weight model performs about as well under Atlarix as under a strong existing harness — the harness isn't holding it back.
~25% of tasks timed out — for both harnesses. At native timeout (×1), roughly a quarter of tasks hit AgentTimeoutError on each side and count as unresolved. So the sub-50% absolute scores aren't all capability failures; a meaningful share are wall-clock on heavy tasks. The timeout ceiling is identical for both agents, so the comparison stays fair — but that's why neither number is higher.
The one config change (full disclosure)
Atlarix's desktop app asks for human approval before every file write and command — a core safety feature. Benchmarks run unattended, so I grant that approval once via an explicit operator flag (ATLARIX_AUTONOMOUS_DANGER=1). Without it, any task needing an install or privileged command is blocked and fails.
This is not an advantage over opencode — every agent auto-approves to run an automated benchmark; it's inherent to running unattended. Stating it for full transparency. The flag is off by default; the interactive app always asks.
Reproduce it
The exact Atlarix bundle I ran is a public, Electron-free headless build: atlarix-headless-linux-amd64.tar.gz. The benchmark is the open-source Harbor framework. The raw Harbor result files — per-task pass/fail for both harnesses — are published unedited. Nothing is hand-typed.
Everything (raw result.json for both sides, summary.csv, exact bundle, full setup): atlarix.dev/benchmark
What's next
- More open-weight models, so no claim rests on one.
- The official Terminal-Bench (k=5) submission — on the roadmap.
- More benchmarks beyond terminal tasks.
If you spot something wrong in the result files, that's the point — tell me.
Built in Nairobi.
Top comments (0)