DEV Community

Acidburn007
Acidburn007

Posted on Edited on AI-assisted

How our open source coding agent scored 84.5% on Terminal-Bench 2.1 for $27 with DeepSeek V4.1 Flash

#agents #ai #llm #opensource

AgenC Core, our open source coding agent for the terminal, scored 84.5% on Terminal-Bench 2.1 with DeepSeek V4.1 Flash. The inference cost for all 445 trials was $27.

Every entry on the official leaderboard spent at least $134. The five entries that scored higher spent between $463 and $1,070. Our run is public on Harbor Hub, the site that hosts the leaderboard, with every trajectory. It is not an official leaderboard entry.

Our goal is to multiply model capability per dollar of inference. This post covers the method and the limits of the number, then the harness bugs that earlier runs exposed.

What Terminal-Bench measures, and what an AI agent harness is

Terminal-Bench is a benchmark for agents that work in a terminal, with tasks such as debugging async code and fixing security vulnerabilities. Each trial puts the agent in a container with one instruction, and the task's own tests grade the result.

A score belongs to an agent and a model together. An AI agent harness is everything around the model: the tools it can call, the shell that runs its commands, context compaction, retries, and the rule for when a task counts as done. AgenC Core is the harness, under the MIT license. You choose the model.

How we ran it

  • Dataset. The official Terminal-Bench 2.1 package, pinned to the leaderboard's dataset revision: 89 tasks, 5 trials per task, 445 trials. Official timeouts and resources. No task timeout was changed.
  • Agent and model. AgenC Core 0.19.24 in Light mode, its default, with DeepSeek V4.1 Flash (deepseek/deepseek-flash) at max reasoning effort, run through Harbor, the benchmark's runner.
  • Launch. One headless turn per trial, task on stdin: agenc -p --budget max --dangerously-bypass-approvals-and-sandbox --output-format stream-json, plus --deadline at the trial's deadline minus 2 seconds. A trial's agent/launch.json on Harbor Hub records the arguments.
  • Contamination. An egress filter refused known routes to benchmark answers, and a local gateway held the API key and logged every call. In 5 passing trials the agent had still seen a task's solution or tests. We counted those as failures, which lowers the score from 85.6% to 84.5% (376 of 445 trials). Harbor Hub shows the unadjusted average reward, 0.86. Late finishes also count as failures, and there were none.
  • Cost. Harbor Hub lists $27.21 of inference, about $0.06 per trial, over 1.2 billion tokens with a 97% cache hit rate. It is priced at DeepSeek's list price for the time of each request.

All 445 trials are public: Harbor Hub job 2bf2c84d, run on October 29 and 30, 2026 (UTC).

Results: score and cost

Agent and model Score Cost On the official leaderboard
Codex, GPT-6 Astra (high) 87.4% ± 1.8% $773.50 Yes, top entry
AgenC Core, DeepSeek V4.1 Flash (max) 84.5% $27.21 No, public run
Claude Code, Fable 5 (xhigh) 83.8% ± 2.3% $552.67 Yes
Cursor CLI, Grok 4.5 (high) 79.3% ± 2.9% $134.09 Yes, cheapest entry

Leaderboard rows are from the official Terminal-Bench 2.1 leaderboard on Harbor Hub, read on November 11, 2026. It has 22 entries, all listed on get.agenc.ag/terminal-bench.

Score against cost on Terminal-Bench 2.1. AgenC Core with DeepSeek V4.1 Flash sits at 84.5% and $27, apart from the leaderboard entries, which all cost $134 or more. Leaderboard data from Harbor Hub, October 31, 2026.

That is a higher score per dollar than any leaderboard entry. The cheapest entry cost about five times as much and scored 5.2 points lower. The top entry scored 2.9 points higher and cost more than 28 times as much.

The caveats:

  • It is not an official entry. Community submissions for 2.1 are closed, and only runs done by the maintainers are added. We asked them to run AgenC Core on November 8, 2026 (issue #253). Maintainers review the trajectories of leaderboard submissions. Our contamination review is our own.
  • It compares combinations. Each row pairs an agent with a model, so this compares combinations, not agents alone. Our eval docs say model choice moves scores far more than the harness does. No leaderboard entry uses a DeepSeek model, so there is no same-model comparison here.
  • The margins overlap. The leaderboard prints a margin with each score, from ± 1.4% to ± 3.3%. Our run has none. The 0.7 point gap to Claude Code with Fable 5 is inside that entry's margin.
  • It is the later of two public runs. Our earlier run with the same model, started October 12, 2026 on an older build, has a raw average reward of 0.76 against 0.86 here, and 61 timeouts against none (job 71f39f3b). The adapter and run setup also changed, so the gap is not a clean measure of the harness.
  • It used a lot of tokens. Only one leaderboard entry lists more than our 1.2 billion. The cost is low because the model is cheap per token and most of the prompt came from cache.

The harness bugs benchmarking found

The useful part of a benchmark run is the transcripts. The findings below come from runs before this one: an 11-task pilot and a full run on Terminal-Bench 2.0 with DeepSeek V4 Pro, a Terminal-Bench 4.0 run with Grok 4.6, and trials with GPT-6 Astra. All five were fixed in September 2026, before release 0.19.24.

  • Services died when a command returned. The shell tool stopped every process the agent had started once the command finished, and the tool result never said so. Pilot transcripts show the model starting nginx four different ways and finding nothing listening each time. Three of our four pilot failures had this shape. The result now reports stopped processes, and exec_command takes detach: true for a service that must outlive the session.
  • A safety policy blocked ordinary commands. Even with approvals bypassed, the shell write policy refused any command it could not analyze. That covered everything containing $(...): 51 of 459 shell calls in one run. Under a full bypass with no sandbox those guards are now lifted. Protected paths stay protected.
  • The agent said it was done without checking. With those bugs fixed, the remaining failures were tasks the agent had declared finished. GPT-6 Astra sessions averaged 19 minutes, against 35 to 47 for the Codex leaderboard rows. Final messages claimed checks the transcript did not contain. The cause was our prompt. Headless runs received the interactive instructions, which tell the model to stop and report to a human who will steer. Non-interactive sessions now get a completion contract: restate the task as checkable requirements, run the checks before the final message, never end the turn waiting for input, and report what was verified.
  • A long session broke compaction. A 55-minute Grok 4.6 session with 191 tool calls reached the auto-compaction threshold and died. The planner assumed 4 bytes per token and packed the whole history into one summary request. The provider counted 2.29 and rejected it. The planner now budgets 2 bytes per token and splits large histories.
  • A dropped connection ended 1.5 hours of work. An upstream socket drop truncated a response stream in a Grok trial. It came back as a generic provider error that no retry rule recognized. A truncated stream is now a typed error, retried like a stalled stream.

We also built a completion gate that holds a final answer until executed checks back it. It is off by default for JSON output, which is how the adapter runs the agent, so it did not run in the published trials. In our measured benchmark trials it changed no result (0.19.19 release notes).

None of these was a model failure, and the fixes apply to any model the harness runs. This post contains no ablation, so we attach no score to any single fix.

Reproduce the run

The Harbor adapter is runtime/eval/harbor/agenc_agent.py. It installs the pinned release through the public installer and checks its SHA-256, answers the trust prompt in the real terminal UI, never changes task timeouts, and writes an ATIF trajectory per trial. You need Harbor, a clone of the repository and a DeepSeek API key.

uv tool install harbor            # the published runs used 0.23.0
export PYTHONPATH=/path/to/agenc-core/runtime/eval/harbor
DEEPSEEK_API_KEY=... harbor run \
  -d terminal-bench/terminal-bench-2-1@sha256:7d7bdc1cbedad549fc1140404bd4dc45e5fd0ea7c4186773687d177ad3a0699a \
  -a agenc_agent:Agenc -m deepseek/deepseek-flash \
  --ak version=0.19.24 --ak reasoning_effort=max \
  -k 5 -n 4 -o ./jobs/agenc-tb21
Enter fullscreen mode Exit fullscreen mode

Three things differ from our published run:

  • The published run used an internal adapter tied to our bench host. The public one was written afterwards as its counterpart (PR #3428) and validated on four live trials, not a full rerun.
  • The egress filter and the logging gateway are not part of the adapter.
  • The published trials passed --deadline with a 2 second margin. Until agenc-core#3433 is merged, the adapter leaves it off. Add --ak cli_deadline=true --ak deadline_margin_sec=2 to match.

The full method is in docs/eval/terminal-bench.md.

Try AgenC Core

AgenC Core works with OpenAI, Anthropic, Grok, Gemini, DeepSeek, OpenRouter and other providers, or with local models through Ollama and LM Studio. It runs in the terminal UI or headless in scripts and CI with agenc -p.

macOS and Linux:

curl -fsSL https://get.agenc.ag/install.sh | sh
Enter fullscreen mode Exit fullscreen mode

Windows (PowerShell):

iwr -useb https://get.agenc.ag/install.ps1 | iex
Enter fullscreen mode Exit fullscreen mode

Then run agenc onboard to choose a provider.

If you run it on another model or another benchmark, we would like to see your numbers.

Top comments (2)

Collapse
 
reidmarlow profile image
Reid Marlow •

Publishing all 445 trajectories and scoring the 5 contaminated passes as failures makes the $27 run genuinely comparable to the leaderboard entries.

The two harness fixes that stand out are detached process lifecycles and the compaction estimator. Stopping child processes when a shell call returns silently breaks server-setup tasks, though detach: true also requires process-group cleanup on teardown so orphaned listeners do not hold ports across retries. And a 4 bytes-per-token heuristic holds up on English prose while dense JSON tool traces and stack logs drop near 2.29 bytes per token and overflow the summarizer window mid-run.

Collapse
 
pavelevich profile image
Pavelisko •

Thanks ❤️‍🔥, both good points. AgenC deliberately never stops a detached service: it runs in its own session with its own log, and the tool result reports its pid and log path, so the agent or the user stops it with kill . In Terminal-Bench the container teardown is the cleanup, which is why the grader can still reach the server. Outside a container, orphaned listeners are a real risk; the agent is told to check a server’s known URL before starting another one after a reconnect, and process-group cleanup on retry is worth adding. On compaction, agreed: tool-heavy histories were the failing case, which is why the planner now budgets 2 bytes per token and splits large histories across several summary calls