DEV Community

Santosh kumar
Santosh kumar

Posted on Originally published at agentfield.ai

CodeAF head to head with nine coding agents on DeepSWE

I work on codeaf at AgentField, one of the harnesses compared here. This post first appeared on agentfield.ai.

Compared task by task on the 113 DeepSWE tasks, with every harness on the same model (deepseek/deepseek-v4-flash-0731), CodeAF in senior-dev mode solved more tasks that the other harness failed than the other way round, against each of the nine harnesses in AgentField's comparison. Against mini-swe-agent: 29 tasks only CodeAF solved, 23 only mini-swe-agent solved. The widest is Muse Code: 60 against 1.

A total score hides how two agents differ. Two harnesses can both solve about half the tasks and still solve different halves. Pairing them task by task shows which one you would rather have had on the tasks where they disagree.

At a glance

Opponent Only CodeAF solved Only the opponent solved Both solved Neither
mini-swe-agent 29 23 33 28
Codex CLI 25 14 37 37
Pi 33 13 29 38
Oh My Pi 44 13 18 38
OpenCode 40 8 22 43
Kilo Code 41 9 21 42
DeepSeek Harness 53 7 9 44
Claude Code 51 5 11 46
Muse Code 60 1 2 50

Each row adds up to 113. "Solved" means the official DeepSWE verifier passed the task. A task with no verifier result counts as not solved, as on the main results page.

Method

This page reads the per-task records of the comparison on the main DeepSWE results page: ten harnesses, the full 113-task set, one model for all of them through OpenRouter, the official verifiers at 0b9fabb, 3 hours per task. The records are one CSV per harness, 113 rows each, in the campaign folder. In those files the CodeAF row is named senior-dev, the mode that ran.

Any row of the table can be recounted from those files. Join them, then compare the reward column of two harnesses on each task:

awk 'FNR>1 || NR==1' tasks/*.csv > all-arms.csv
Enter fullscreen mode Exit fullscreen mode

Results

Against the three strongest other harnesses, CodeAF's margin on disagreements was 6 tasks over mini-swe-agent, 11 over Codex CLI and 20 over Pi. Against the rest it was 31 or more.

The "Neither" column matters too. 15 tasks were solved by no harness at all, so even the best pairing leaves a block of tasks neither agent could do on this model.

Questions people ask

Does CodeAF beat mini-swe-agent on DeepSWE?

Yes, in AgentField's comparison on DeepSeek V4 Flash: CodeAF in senior-dev mode solved 62 of 113 tasks and mini-swe-agent 56. On the tasks where they disagreed, CodeAF solved 29 that mini-swe-agent missed and mini-swe-agent solved 23 that CodeAF missed.

Does CodeAF beat Codex CLI on DeepSWE?

On DeepSeek V4 Flash, CodeAF solved 62 tasks and Codex CLI 0.154.0 solved 51. Task by task, 25 were solved only by CodeAF and 14 only by Codex CLI; both solved 37. Codex CLI here drove DeepSeek, not an OpenAI model.

How does CodeAF compare with Claude Code task by task?

With both on DeepSeek V4 Flash, CodeAF solved 51 tasks that Claude Code 2.1.267 failed, and Claude Code solved 5 that CodeAF failed. That measures the harness on DeepSeek; it does not say how Claude Code performs with Claude models.

Can I check these numbers myself?

Yes. The per-task CSVs for all ten harnesses are public in the CodeAF repository, linked under Method. Each row has the task, its band and language, the verifier reward, the fail-to-pass and pass-to-pass test counts, wall time and exit status.

Sources

Last updated 2026-11-21. Every count recomputed from the per-task CSVs; versions are those on the main results page.

Top comments (0)