How our open source coding agent scored 84.5% on Terminal-Bench 2.1 for $27 with DeepSeek V4.1 Flash

Acidburn007 — Wed, 11 Nov 2026 19:46:13 +0000

AgenC Core, our open source coding agent for the terminal, scored 84.5% on Terminal-Bench 2.1 with DeepSeek V4.1 Flash. The inference cost for all 445 trials was $27.

Every entry on the official leaderboard spent at least $134. The five entries that scored higher spent between $463 and $1,070. Our run is public on Harbor Hub, the site that hosts the leaderboard, with every trajectory. It is not an official leaderboard entry.

Our goal is to multiply model capability per dollar of inference. This post covers the method and the limits of the number, then the harness bugs that earlier runs exposed.

What Terminal-Bench measures, and what an AI agent harness is

Terminal-Bench is a benchmark for agents that work in a terminal, with tasks such as debugging async code and fixing security vulnerabilities. Each trial puts the agent in a container with one instruction, and the task's own tests grade the result.

A score belongs to an agent and a model together. An AI agent harness is everything around the model: the tools it can call, the shell that runs its commands, context compaction, retries, and the rule for when a task counts as done. AgenC Core is the harness, under the MIT license. You choose the model.

How we ran it

Dataset. The official Terminal-Bench 2.1 package, pinned to the leaderboard's dataset revision: 89 tasks, 5 trials per task, 445 trials. Official timeouts and resources. No task timeout was changed.
Agent and model. AgenC Core 0.19.24 in Light mode, its default, with DeepSeek V4.1 Flash (deepseek/deepseek-flash) at max reasoning effort, run through Harbor, the benchmark's runner.
Launch. One headless turn per trial, task on stdin: agenc -p --budget max --dangerously-bypass-approvals-and-sandbox --output-format stream-json, plus --deadline at the trial's deadline minus 2 seconds. A trial's agent/launch.json on Harbor Hub records the arguments.
Contamination. An egress filter refused known routes to benchmark answers, and a local gateway held the API key and logged every call. In 5 passing trials the agent had still seen a task's solution or tests. We counted those as failures, which lowers the score from 85.6% to 84.5% (376 of 445 trials). Harbor Hub shows the unadjusted average reward, 0.86. Late finishes also count as failures, and there were none.
Cost. Harbor Hub lists $27.21 of inference, about $0.06 per trial, over 1.2 billion tokens with a 97% cache hit rate. It is priced at DeepSeek's list price for the time of each request.

All 445 trials are public: Harbor Hub job 2bf2c84d, run on October 29 and 30, 2026 (UTC).

Results: score and cost

Agent and model	Score	Cost	On the official leaderboard
Codex, GPT-6 Astra (high)	87.4% ± 1.8%	$773.50	Yes, top entry
AgenC Core, DeepSeek V4.1 Flash (max)	84.5%	$27.21	No, public run
Claude Code, Fable 5 (xhigh)	83.8% ± 2.3%	$552.67	Yes
Cursor CLI, Grok 4.5 (high)	79.3% ± 2.9%	$134.09	Yes, cheapest entry

Leaderboard rows are from the official Terminal-Bench 2.1 leaderboard on Harbor Hub, read on November 11, 2026. It has 22 entries, all listed on get.agenc.ag/terminal-bench.

That is a higher score per dollar than any leaderboard entry. The cheapest entry cost about five times as much and scored 5.2 points lower. The top entry scored 2.9 points higher and cost more than 28 times as much.

The caveats:

It is not an official entry. Community submissions for 2.1 are closed, and only runs done by the maintainers are added. We asked them to run AgenC Core on November 8, 2026 (issue #253). Maintainers review the trajectories of leaderboard submissions. Our contamination review is our own.
It compares combinations. Each row pairs an agent with a model, so this compares combinations, not agents alone. Our eval docs say model choice moves scores far more than the harness does. No leaderboard entry uses a DeepSeek model, so there is no same-model comparison here.
The margins overlap. The leaderboard prints a margin with each score, from ± 1.4% to ± 3.3%. Our run has none. The 0.7 point gap to Claude Code with Fable 5 is inside that entry's margin.
It is the later of two public runs. Our earlier run with the same model, started October 12, 2026 on an older build, has a raw average reward of 0.76 against 0.86 here, and 61 timeouts against none (job 71f39f3b). The adapter and run setup also changed, so the gap is not a clean measure of the harness.
It used a lot of tokens. Only one leaderboard entry lists more than our 1.2 billion. The cost is low because the model is cheap per token and most of the prompt came from cache.

The harness bugs benchmarking found

The useful part of a benchmark run is the transcripts. The findings below come from runs before this one: an 11-task pilot and a full run on Terminal-Bench 2.0 with DeepSeek V4 Pro, a Terminal-Bench 4.0 run with Grok 4.6, and trials with GPT-6 Astra. All five were fixed in September 2026, before release 0.19.24.

Services died when a command returned. The shell tool stopped every process the agent had started once the command finished, and the tool result never said so. Pilot transcripts show the model starting nginx four different ways and finding nothing listening each time. Three of our four pilot failures had this shape. The result now reports stopped processes, and exec_command takes detach: true for a service that must outlive the session.
A safety policy blocked ordinary commands. Even with approvals bypassed, the shell write policy refused any command it could not analyze. That covered everything containing $(...): 51 of 459 shell calls in one run. Under a full bypass with no sandbox those guards are now lifted. Protected paths stay protected.
The agent said it was done without checking. With those bugs fixed, the remaining failures were tasks the agent had declared finished. GPT-6 Astra sessions averaged 19 minutes, against 35 to 47 for the Codex leaderboard rows. Final messages claimed checks the transcript did not contain. The cause was our prompt. Headless runs received the interactive instructions, which tell the model to stop and report to a human who will steer. Non-interactive sessions now get a completion contract: restate the task as checkable requirements, run the checks before the final message, never end the turn waiting for input, and report what was verified.
A long session broke compaction. A 55-minute Grok 4.6 session with 191 tool calls reached the auto-compaction threshold and died. The planner assumed 4 bytes per token and packed the whole history into one summary request. The provider counted 2.29 and rejected it. The planner now budgets 2 bytes per token and splits large histories.
A dropped connection ended 1.5 hours of work. An upstream socket drop truncated a response stream in a Grok trial. It came back as a generic provider error that no retry rule recognized. A truncated stream is now a typed error, retried like a stalled stream.

We also built a completion gate that holds a final answer until executed checks back it. It is off by default for JSON output, which is how the adapter runs the agent, so it did not run in the published trials. In our measured benchmark trials it changed no result (0.19.19 release notes).

None of these was a model failure, and the fixes apply to any model the harness runs. This post contains no ablation, so we attach no score to any single fix.

Reproduce the run

The Harbor adapter is runtime/eval/harbor/agenc_agent.py. It installs the pinned release through the public installer and checks its SHA-256, answers the trust prompt in the real terminal UI, never changes task timeouts, and writes an ATIF trajectory per trial. You need Harbor, a clone of the repository and a DeepSeek API key.

uv tool install harbor            # the published runs used 0.23.0
export PYTHONPATH=/path/to/agenc-core/runtime/eval/harbor
DEEPSEEK_API_KEY=... harbor run \
  -d terminal-bench/terminal-bench-2-1@sha256:7d7bdc1cbedad549fc1140404bd4dc45e5fd0ea7c4186773687d177ad3a0699a \
  -a agenc_agent:Agenc -m deepseek/deepseek-flash \
  --ak version=0.19.24 --ak reasoning_effort=max \
  -k 5 -n 4 -o ./jobs/agenc-tb21

Three things differ from our published run:

The published run used an internal adapter tied to our bench host. The public one was written afterwards as its counterpart (PR #3428) and validated on four live trials, not a full rerun.
The egress filter and the logging gateway are not part of the adapter.
The published trials passed --deadline with a 2 second margin. Until agenc-core#3433 is merged, the adapter leaves it off. Add --ak cli_deadline=true --ak deadline_margin_sec=2 to match.

The full method is in docs/eval/terminal-bench.md.

Try AgenC Core

AgenC Core works with OpenAI, Anthropic, Grok, Gemini, DeepSeek, OpenRouter and other providers, or with local models through Ollama and LM Studio. It runs in the terminal UI or headless in scripts and CI with agenc -p.

macOS and Linux:

curl -fsSL https://get.agenc.ag/install.sh | sh

Windows (PowerShell):

iwr -useb https://get.agenc.ag/install.ps1 | iex

Then run agenc onboard to choose a provider.

Source: github.com/tetsuo-ai/agenc-core
Install and docs: get.agenc.ag
Setup as a DeepSeek coding agent, the way it ran here: get.agenc.ag/deepseek

If you run it on another model or another benchmark, we would like to see your numbers.

Tiny tool, big help: automatically choose the best Ollama model for your setup

Acidburn007 — Tue, 05 Aug 2025 11:44:09 +0000

DEV Community: Acidburn007