DEV Community

Santosh kumar
Santosh kumar

Posted on Originally published at agentfield.ai

DeepSWE results: ten coding agents on one model (2026)

I work on codeaf at AgentField, one of the harnesses compared here. This post first appeared on agentfield.ai.

CodeAF, the open-source coding agent by AgentField, solved 62 of 113 DeepSWE tasks (54.9%) in senior-dev mode, first of ten harnesses in AgentField's DeepSWE comparison, where every harness ran the same model (deepseek/deepseek-v4-flash-0731 through OpenRouter). It also had the lowest cost per solved issue: 39.9 cents, at 22 cents a task. Every number is in arms.csv.

At a glance

Result CodeAF (senior-dev mode) solved 62 of 113 (54.9%), first of ten harnesses in this comparison
Model, for every harness deepseek/deepseek-v4-flash-0731 through OpenRouter
Cost 22 cents a task, 39.9 cents per solved issue (the lowest of the ten)
Next best mini-swe-agent, 56 of 113 (49.6%), 1.9x CodeAF's cost per solved issue
With newer models the same mode solved 77.9% on DeepSeek V4.1 Flash and 69.0% on Kimi K3

The full table

Every row ran the full DeepSWE set on the same model. "Cost per solved issue" is each harness's spend divided by tasks solved, shown as a multiple of CodeAF's 39.9 cents. "Mean time" is mean agent wall time per task.

Harness Solved (of 113) Cost per task Cost per solved issue Mean time
CodeAF (senior-dev) 62, 54.9% 22 cents 1x 54 min
mini-swe-agent 56, 49.6% 38 cents 1.9x 44 min
Codex CLI 51, 45.1% 37 cents 2.1x 46 min
Pi 42, 37.2% 35 cents 2.4x 52 min
Oh My Pi (omp) 31, 27.4% 50 cents 4.5x 49 min
OpenCode 30, 26.5% 50 cents 4.8x 48 min
Kilo Code 30, 26.5% 48 cents 4.6x 54 min
Claude Code 16, 14.2% 19 cents 3.4x 32 min
DeepSeek Harness 16, 14.2% 150 cents 26.6x 94 min
Muse Code 3, 2.7% 12 cents 11.3x 16 min

30 of 113 is 26.5%. The method page prints 26.6% for OpenCode and Kilo Code, rounded from the stored reward rate of 0.2655; the task count is the same. In arms.csv the CodeAF row is named senior-dev, because the binary had an earlier name on the day of the run. At a shell, the same mode is codeaf senior-dev; in a conversation it is /senior-dev (senior-dev docs).

Results by slice

The per-task records, one CSV per harness with 113 rows each, show where the lead came from:

  • Head to head: against each of the nine other harnesses, CodeAF solved more of the tasks where the two disagreed. Against mini-swe-agent, 29 to 23. See head to head.
  • Harder tasks: on the 76 medium and hard tasks CodeAF solved 36, against 30 for mini-swe-agent and 29 for Codex CLI. See by difficulty.
  • Go and Python: CodeAF solved 21 of 34 Go tasks (next: 17) and 20 of 34 Python tasks (next: 18). See by language.
  • Cost: 39.9 cents per solved task, against 76.9 for the next cheapest. See cost per solved issue.

How the comparison was run

Benchmark full DeepSWE set, 113 tasks
Model deepseek/deepseek-v4-flash-0731 through OpenRouter
Verifiers official DeepSWE at 0b9fabb
Budget 3 hours per task
Isolation four shards per harness, a dedicated OpenRouter key per harness

How the columns are derived, from the method page:

  • Solved counts tasks the verifier passed. A task with no verifier outcome counts as unsolved. Five tasks produced none: Codex CLI 1, Pi 1, Oh My Pi 2, OpenCode 1.
  • Cost per task is billed OpenRouter spend divided by 113, so tasks without a verifier result stay in the denominator. Oh My Pi, OpenCode, Kilo Code, DeepSeek Harness, Claude Code and Muse Code did not record per-attempt cost; their figure is that harness's total spend.
  • Cost per solved issue is spend divided by tasks solved.

Tested versions

Harness Version tested Release date (UTC)
CodeAF first stable release containing senior-dev mode: v0.5.0 (integration commit dd0fcc654, #1488) 2026-09-30
mini-swe-agent 2.4.6 2026-07-23
Codex CLI 0.154.0 2026-09-09
Pi 0.85.1 2026-09-05
Oh My Pi 18.1.17 2026-09-10
OpenCode 1.18.30 2026-09-09
Kilo Code 7.6.2 2026-09-10
Claude Code 2.1.267 2026-09-09
DeepSeek Harness (@deepseek-ai/dsh) 0.1.5-rc.1 2026-09-10
Muse Code 1.1.1-R2514.1 not independently established

versions.csv holds full source revisions and release timestamps.

DeepSWE versus SWE-bench Verified

DeepSWE, published by Datacurve, is "a benchmark for measuring frontier coding agents on original, long-horizon software engineering tasks": 113 tasks across TypeScript, Go, Python, JavaScript and Rust, each graded by a verifier that "accepts any solution whose observable behavior is correct" (DeepSWE README). SWE-bench Verified is "a human-validated subset of 500 SWE-bench instances."

AgentField's comparison holds the model fixed and changes the harness, so it measures what the agent around the model adds. Every harness was graded by the same verifiers on the same tasks.

Senior-dev mode with newer models

With newer models, the same mode solved 77.9% on DeepSeek V4.1 Flash and 69.0% on Kimi K3. That is 88 of 113 solved (exact 95% CI 69.1% to 85.1%) with DeepSeek V4.1 Flash and 78 of 113 (59.6% to 77.4%) with Kimi K3.

How to reproduce

The scripts that run and grade DeepSWE tasks are in bench/deepswe/ in the CodeAF repository. One task through CodeAF:

bench/deepswe/run.sh <task-id> <model-id> <seed-tag>
Enter fullscreen mode Exit fullscreen mode

It needs a linux/amd64 codeaf binary in CODEAF_BIN, a provider key, and the DeepSWE task corpus, which Datacurve distributes as a gated dataset. The compare-* scripts beside it install a pinned comparator, freeze its container image and run it. The per-task records of this campaign are public, so every table on this page can be recounted from them.

Questions people ask

Is CodeAF number one on DeepSWE?

CodeAF was first of ten harnesses in AgentField's DeepSWE comparison, solving 62 of 113 tasks (54.9%) with every harness on DeepSeek V4 Flash. The comparison holds the model fixed and ranks the harness around it.

Which AI coding agent scores highest on SWE-bench style benchmarks right now?

When the model is held fixed (DeepSeek V4 Flash) and the harness changes, CodeAF solved the most DeepSWE tasks of the ten harnesses in AgentField's comparison, 62 of 113 (54.9%), ahead of mini-swe-agent (56) and Codex CLI (51).

What is the most cost-efficient coding agent per solved issue?

In AgentField's DeepSWE comparison on DeepSeek V4 Flash, CodeAF had the lowest cost per solved issue, 39.9 cents, ahead of mini-swe-agent (1.9x), Codex CLI (2.1x) and Claude Code (3.4x).

Does this mean CodeAF beats Claude Code?

On DeepSeek V4 Flash, in AgentField's DeepSWE comparison, CodeAF solved 62 of 113 tasks and Claude Code 2.1.267 solved 16. Both ran DeepSeek V4 Flash, not Claude, so the result measures the harness on that model, not Claude Code with Claude models.

Can I reproduce these results?

Yes. The run and grading scripts are in bench/deepswe/ in the CodeAF repository, and the per-task records of all ten harnesses are public, so every count here can be checked. The DeepSWE task corpus is a gated dataset from Datacurve.

Sources

For how CodeAF compares beyond this benchmark, see CodeAF vs Claude Code and CodeAF vs Codex CLI. For the 3-hour, unattended way senior-dev works, see a coding agent that runs for hours, and for the terminal side, terminal coding agents.

Last updated 2026-11-21. Every figure checked against arms.csv and versions.csv at the method page; harness versions are those of the 2026-09-10 to 12 campaign.

Top comments (0)