How our open source coding agent scored 84.5% on Terminal-Bench 2.1 for $27

Pavelisko — Wed, 11 Nov 2026 19:46:13 +0000

AgenC Core is an open source (MIT) coding agent for the terminal. Our goal is to multiply model capability per dollar of inference: to get more verified work out of a model for every dollar spent on it.

To test that goal, we ran Terminal-Bench 2.1 with DeepSeek V4.1 Flash. AgenC Core scored 84.5%, and the whole run cost $27. Every entry on the official leaderboard spent at least $134. All 445 trials are public, with full trajectories.

This post covers how we ran it, what the number does and does not mean, and what the benchmark taught us about our own harness.

Why the harness matters

Terminal-Bench measures how well an agent completes real tasks in a terminal: building software, repairing a git history, configuring servers, recovering data. Each trial gives the agent one instruction inside a container, and the task's own tests grade the result.

A leaderboard row is always an agent and a model together. The model supplies the reasoning. The harness around it decides how that reasoning turns into finished work: how the task is planned, which tools exist, how progress is kept, and how results are checked before the agent says it is done. With the same model, a better harness finishes more of the work for the same money.

How we ran it

The official Terminal-Bench 2.1 package, pinned to the leaderboard's dataset revision: 89 tasks, 5 trials per task, 445 trials.
The official timeouts and resources. No task timeout was changed.
AgenC Core 0.19.24 with DeepSeek V4.1 Flash at max reasoning effort, run through Harbor, the benchmark's runner.
Contamination handling. An egress filter refused known routes to benchmark answers, and a local gateway held the API key and logged every call. In 5 passing trials the agent had still seen a task's solution or tests. We counted those as failures, which lowered the score from 85.6% to 84.5%. Late finishes also count as failures, and there were none.
Cost. $27 is the figure Harbor Hub lists for the run, the same basis as every leaderboard entry. At DeepSeek's list prices for the recorded request times it came to $27.21, across about 1.2 billion tokens with a 97% cache hit rate.

The run is public on Harbor Hub: job 2bf2c84d.

The results

Agent and model	Score	Cost
AgenC Core, DeepSeek V4.1 Flash (max)	84.5%	$27
Codex, GPT-6 Astra (high)	87.4%	$774
Claude Code, Fable 5 (xhigh)	83.8%	$553
Cursor CLI, Grok 4.5 (high)	79.3%	$134

The AgenC Core row is our public run. The other rows are entries on the official Terminal-Bench 2.1 leaderboard as of October 31, 2026, and Cursor CLI's is the cheapest of them. The top entry scored 2.9 percentage points more and cost more than 28 times as much.

Two caveats. Each row pairs an agent with a model, so this compares combinations, not agents alone. And our run is not on the official leaderboard: community submissions for 2.1 are closed, and only runs done by the maintainers are added. We have asked them to run AgenC Core (issue #253), and the adapter they would need is public.

What the benchmark found in our harness

The most useful part of running the benchmark was reading the transcripts. Across our runs, they exposed problems that had nothing to do with the model:

Services died when a command returned. The shell tool stopped every process the agent had started once the command finished, so a server the agent set up was gone by the time the grader ran. Three of the four failures in each of our early runs were tasks of that shape: nginx, a git server with a deploy hook, an sshd server. The tool result now says when leftover processes were stopped, and exec_command takes detach: true to start a service that outlives the session.
A safety policy blocked ordinary commands. Even with approvals bypassed, the shell write policy refused any command it could not analyse, which included everything containing $(...): 51 of 459 shell calls in one run. Under a full bypass with no sandbox those guards are now lifted, and protected paths stay protected.
The agent declared success without checking. Once the harness bugs were fixed, the remaining failures were tasks the agent had called done: sessions that ended early, with final messages claiming checks the transcript did not contain. Non-interactive sessions now carry a completion contract: restate the task as checkable requirements, run the checks before the final message, report what was verified. A completion gate enforces it in code: a final answer is held until it is backed by checks that ran after the last change.
Long sessions broke compaction. A 55-minute session with 191 tool calls reached the auto-compaction threshold and failed. Our planner assumed 4 bytes per token, the provider counted 2.29, and the summary request was rejected as too large. The planner now bounds its input at 2 bytes per token and splits large histories.
A dropped connection ended a trial. A truncated stream came back as a generic provider error that no retry rule recognized. It is now a typed error, retried like any stalled stream.

None of these were model problems, and each fix applies to every model the harness runs. That is what "multiply model capability" means in practice.

Reproduce it

The Harbor adapter is in the repository at runtime/eval/harbor/agenc_agent.py. It installs the pinned release through the public installer and checks its SHA-256, answers the project trust prompt in the real terminal UI, never changes task timeouts, and writes an ATIF trajectory for every trial.

uv tool install harbor
git clone https://github.com/tetsuo-ai/agenc-core
cd agenc-core
export PYTHONPATH=$PWD/runtime/eval/harbor
DEEPSEEK_API_KEY=... harbor run \
  -d terminal-bench/terminal-bench-2-1@sha256:7d7bdc1cbedad549fc1140404bd4dc45e5fd0ea7c4186773687d177ad3a0699a \
  -a agenc_agent:Agenc -m deepseek/deepseek-flash \
  --ak version=0.19.24 --ak reasoning_effort=max \
  -k 5 -n 4 -o ./jobs/agenc-tb21

Our published run also had the egress filter and the logging gateway described above, which are not part of the adapter. The full method is in docs/eval/terminal-bench.md.

Try AgenC Core

AgenC Core works with OpenAI, Anthropic, Grok, Gemini, DeepSeek, OpenRouter and other providers, or with local models through Ollama and LM Studio. It runs in the terminal or headless in scripts and CI with agenc -p, and its Goal mode takes a project through implementation, testing and an independent review.

macOS and Linux:

curl -fsSL https://get.agenc.ag/install.sh | sh

Windows (PowerShell):

iwr -useb https://get.agenc.ag/install.ps1 | iex

Then run agenc onboard to choose a provider.

Source: github.com/tetsuo-ai/agenc-core
Install and docs: get.agenc.ag
Full results and every leaderboard entry: get.agenc.ag/terminal-bench

If you run it on another model or another benchmark, we would like to see your numbers.

Tiny tool, big help: automatically choose the best Ollama model for your setup

Pavelisko — Tue, 05 Aug 2025 11:44:09 +0000

DEV Community: Pavelisko