<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Santosh kumar</title>
    <description>The latest articles on DEV Community by Santosh kumar (@santoshkumarradha).</description>
    <link>https://dev.to/santoshkumarradha</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4257906%2Ff5ff7767-d52c-4b96-8453-911c03dd1331.jpg</url>
      <title>DEV Community: Santosh kumar</title>
      <link>https://dev.to/santoshkumarradha</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/santoshkumarradha"/>
    <language>en</language>
    <item>
      <title>DeepSWE results: ten coding agents on one model (2026)</title>
      <dc:creator>Santosh kumar</dc:creator>
      <pubDate>Sat, 21 Nov 2026 20:06:55 +0000</pubDate>
      <link>https://dev.to/santoshkumarradha/deepswe-results-ten-coding-agents-on-one-model-2026-3moh</link>
      <guid>https://dev.to/santoshkumarradha/deepswe-results-ten-coding-agents-on-one-model-2026-3moh</guid>
      <description>&lt;p&gt;&lt;em&gt;I work on &lt;a href="https://github.com/Agent-Field/CodeAF" rel="noopener noreferrer"&gt;codeaf&lt;/a&gt; at AgentField, one of the harnesses compared here. This post first appeared on &lt;a href="https://agentfield.ai/benchmarks/deepswe" rel="noopener noreferrer"&gt;agentfield.ai&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;CodeAF, the open-source coding agent by AgentField, solved 62 of 113 DeepSWE tasks (54.9%) in senior-dev mode, first of ten harnesses in AgentField's DeepSWE comparison, where every harness ran the same model (&lt;code&gt;deepseek/deepseek-v4-flash-0731&lt;/code&gt; through OpenRouter). It also had the lowest cost per solved issue: 39.9 cents, at 22 cents a task. Every number is in &lt;a href="https://github.com/Agent-Field/CodeAF/tree/d049b13406dc9179f2265608bafca3255f3c5f3a/docs/benchmarks/deepswe" rel="noopener noreferrer"&gt;&lt;code&gt;arms.csv&lt;/code&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  At a glance
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Result&lt;/td&gt;
&lt;td&gt;CodeAF (senior-dev mode) solved 62 of 113 (54.9%), first of ten harnesses in this comparison&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Model, for every harness&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;deepseek/deepseek-v4-flash-0731&lt;/code&gt; through OpenRouter&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cost&lt;/td&gt;
&lt;td&gt;22 cents a task, 39.9 cents per solved issue (the lowest of the ten)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Next best&lt;/td&gt;
&lt;td&gt;mini-swe-agent, 56 of 113 (49.6%), 1.9x CodeAF's cost per solved issue&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;With newer models&lt;/td&gt;
&lt;td&gt;the same mode solved 77.9% on DeepSeek V4.1 Flash and 69.0% on Kimi K3&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  The full table
&lt;/h2&gt;

&lt;p&gt;Every row ran the full DeepSWE set on the same model. "Cost per solved issue" is each harness's spend divided by tasks solved, shown as a multiple of CodeAF's 39.9 cents. "Mean time" is mean agent wall time per task.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Harness&lt;/th&gt;
&lt;th&gt;Solved (of 113)&lt;/th&gt;
&lt;th&gt;Cost per task&lt;/th&gt;
&lt;th&gt;Cost per solved issue&lt;/th&gt;
&lt;th&gt;Mean time&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;CodeAF (senior-dev)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;62, 54.9%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;22 cents&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;1x&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;54 min&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;mini-swe-agent&lt;/td&gt;
&lt;td&gt;56, 49.6%&lt;/td&gt;
&lt;td&gt;38 cents&lt;/td&gt;
&lt;td&gt;1.9x&lt;/td&gt;
&lt;td&gt;44 min&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Codex CLI&lt;/td&gt;
&lt;td&gt;51, 45.1%&lt;/td&gt;
&lt;td&gt;37 cents&lt;/td&gt;
&lt;td&gt;2.1x&lt;/td&gt;
&lt;td&gt;46 min&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Pi&lt;/td&gt;
&lt;td&gt;42, 37.2%&lt;/td&gt;
&lt;td&gt;35 cents&lt;/td&gt;
&lt;td&gt;2.4x&lt;/td&gt;
&lt;td&gt;52 min&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Oh My Pi (omp)&lt;/td&gt;
&lt;td&gt;31, 27.4%&lt;/td&gt;
&lt;td&gt;50 cents&lt;/td&gt;
&lt;td&gt;4.5x&lt;/td&gt;
&lt;td&gt;49 min&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;OpenCode&lt;/td&gt;
&lt;td&gt;30, 26.5%&lt;/td&gt;
&lt;td&gt;50 cents&lt;/td&gt;
&lt;td&gt;4.8x&lt;/td&gt;
&lt;td&gt;48 min&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Kilo Code&lt;/td&gt;
&lt;td&gt;30, 26.5%&lt;/td&gt;
&lt;td&gt;48 cents&lt;/td&gt;
&lt;td&gt;4.6x&lt;/td&gt;
&lt;td&gt;54 min&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Claude Code&lt;/td&gt;
&lt;td&gt;16, 14.2%&lt;/td&gt;
&lt;td&gt;19 cents&lt;/td&gt;
&lt;td&gt;3.4x&lt;/td&gt;
&lt;td&gt;32 min&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DeepSeek Harness&lt;/td&gt;
&lt;td&gt;16, 14.2%&lt;/td&gt;
&lt;td&gt;150 cents&lt;/td&gt;
&lt;td&gt;26.6x&lt;/td&gt;
&lt;td&gt;94 min&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Muse Code&lt;/td&gt;
&lt;td&gt;3, 2.7%&lt;/td&gt;
&lt;td&gt;12 cents&lt;/td&gt;
&lt;td&gt;11.3x&lt;/td&gt;
&lt;td&gt;16 min&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;30 of 113 is 26.5%. The method page prints 26.6% for OpenCode and Kilo Code, rounded from the stored reward rate of 0.2655; the task count is the same. In &lt;code&gt;arms.csv&lt;/code&gt; the CodeAF row is named &lt;code&gt;senior-dev&lt;/code&gt;, because the binary had an earlier name on the day of the run. At a shell, the same mode is &lt;code&gt;codeaf senior-dev&lt;/code&gt;; in a conversation it is &lt;code&gt;/senior-dev&lt;/code&gt; (&lt;a href="https://agentfield.ai/docs/codeaf/senior-dev" rel="noopener noreferrer"&gt;senior-dev docs&lt;/a&gt;).&lt;/p&gt;

&lt;h2&gt;
  
  
  Results by slice
&lt;/h2&gt;

&lt;p&gt;The &lt;a href="https://github.com/Agent-Field/CodeAF/tree/9b4344aaf56048c995cb9caa33369449cf137832/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks" rel="noopener noreferrer"&gt;per-task records&lt;/a&gt;, one CSV per harness with 113 rows each, show where the lead came from:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Head to head:&lt;/strong&gt; against each of the nine other harnesses, CodeAF solved more of the tasks where the two disagreed. Against mini-swe-agent, 29 to 23. See &lt;a href="https://agentfield.ai/benchmarks/deepswe-head-to-head" rel="noopener noreferrer"&gt;head to head&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Harder tasks:&lt;/strong&gt; on the 76 medium and hard tasks CodeAF solved 36, against 30 for mini-swe-agent and 29 for Codex CLI. See &lt;a href="https://agentfield.ai/benchmarks/deepswe-by-difficulty" rel="noopener noreferrer"&gt;by difficulty&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Go and Python:&lt;/strong&gt; CodeAF solved 21 of 34 Go tasks (next: 17) and 20 of 34 Python tasks (next: 18). See &lt;a href="https://agentfield.ai/benchmarks/deepswe-by-language" rel="noopener noreferrer"&gt;by language&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cost:&lt;/strong&gt; 39.9 cents per solved task, against 76.9 for the next cheapest. See &lt;a href="https://agentfield.ai/benchmarks/coding-agent-cost-per-solved-issue" rel="noopener noreferrer"&gt;cost per solved issue&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  How the comparison was run
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Benchmark&lt;/td&gt;
&lt;td&gt;full DeepSWE set, 113 tasks&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Model&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;deepseek/deepseek-v4-flash-0731&lt;/code&gt; through OpenRouter&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Verifiers&lt;/td&gt;
&lt;td&gt;official DeepSWE at &lt;code&gt;0b9fabb&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Budget&lt;/td&gt;
&lt;td&gt;3 hours per task&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Isolation&lt;/td&gt;
&lt;td&gt;four shards per harness, a dedicated OpenRouter key per harness&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;How the columns are derived, from the &lt;a href="https://github.com/Agent-Field/CodeAF/tree/d049b13406dc9179f2265608bafca3255f3c5f3a/docs/benchmarks/deepswe" rel="noopener noreferrer"&gt;method page&lt;/a&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Solved&lt;/strong&gt; counts tasks the verifier passed. A task with no verifier outcome counts as unsolved. Five tasks produced none: Codex CLI 1, Pi 1, Oh My Pi 2, OpenCode 1.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cost per task&lt;/strong&gt; is billed OpenRouter spend divided by 113, so tasks without a verifier result stay in the denominator. Oh My Pi, OpenCode, Kilo Code, DeepSeek Harness, Claude Code and Muse Code did not record per-attempt cost; their figure is that harness's total spend.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cost per solved issue&lt;/strong&gt; is spend divided by tasks solved.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Tested versions
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Harness&lt;/th&gt;
&lt;th&gt;Version tested&lt;/th&gt;
&lt;th&gt;Release date (UTC)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;CodeAF&lt;/td&gt;
&lt;td&gt;first stable release containing senior-dev mode: &lt;a href="https://github.com/Agent-Field/CodeAF/releases/tag/v0.5.0" rel="noopener noreferrer"&gt;v0.5.0&lt;/a&gt; (integration commit &lt;code&gt;dd0fcc654&lt;/code&gt;, &lt;a href="https://github.com/Agent-Field/CodeAF/pull/1488" rel="noopener noreferrer"&gt;#1488&lt;/a&gt;)&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;mini-swe-agent&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/SWE-agent/mini-swe-agent/releases/tag/v2.4.6" rel="noopener noreferrer"&gt;2.4.6&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-07-23&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Codex CLI&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/openai/codex/releases/tag/rust-v0.154.0" rel="noopener noreferrer"&gt;0.154.0&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-09&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Pi&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/earendil-works/pi/releases/tag/v0.85.1" rel="noopener noreferrer"&gt;0.85.1&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-05&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Oh My Pi&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/can1357/oh-my-pi/releases/tag/v18.1.17" rel="noopener noreferrer"&gt;18.1.17&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-10&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;OpenCode&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/anomalyco/opencode/releases/tag/v1.18.30" rel="noopener noreferrer"&gt;1.18.30&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-09&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Kilo Code&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/Kilo-Org/kilocode/releases/tag/v7.6.2" rel="noopener noreferrer"&gt;7.6.2&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-10&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Claude Code&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/anthropics/claude-code/releases/tag/v2.1.267" rel="noopener noreferrer"&gt;2.1.267&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-09&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DeepSeek Harness (&lt;code&gt;@deepseek-ai/dsh&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/deepseek-ai/deepseek-harness/releases/tag/dsh-v0.1.5-rc.1" rel="noopener noreferrer"&gt;0.1.5-rc.1&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-10&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Muse Code&lt;/td&gt;
&lt;td&gt;1.1.1-R2514.1&lt;/td&gt;
&lt;td&gt;not independently established&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;a href="https://github.com/Agent-Field/CodeAF/tree/d049b13406dc9179f2265608bafca3255f3c5f3a/docs/benchmarks/deepswe" rel="noopener noreferrer"&gt;&lt;code&gt;versions.csv&lt;/code&gt;&lt;/a&gt; holds full source revisions and release timestamps.&lt;/p&gt;

&lt;h2&gt;
  
  
  DeepSWE versus SWE-bench Verified
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://deepswe.datacurve.ai/" rel="noopener noreferrer"&gt;DeepSWE&lt;/a&gt;, published by Datacurve, is "a benchmark for measuring frontier coding agents on original, long-horizon software engineering tasks": 113 tasks across TypeScript, Go, Python, JavaScript and Rust, each graded by a verifier that "accepts any solution whose observable behavior is correct" (&lt;a href="https://github.com/datacurve-ai/deep-swe" rel="noopener noreferrer"&gt;DeepSWE README&lt;/a&gt;). &lt;a href="https://www.swebench.com/verified.html" rel="noopener noreferrer"&gt;SWE-bench Verified&lt;/a&gt; is "a human-validated subset of 500 SWE-bench instances."&lt;/p&gt;

&lt;p&gt;AgentField's comparison holds the model fixed and changes the harness, so it measures what the agent around the model adds. Every harness was graded by the same verifiers on the same tasks.&lt;/p&gt;

&lt;h2&gt;
  
  
  Senior-dev mode with newer models
&lt;/h2&gt;

&lt;p&gt;With newer models, the same mode solved 77.9% on DeepSeek V4.1 Flash and 69.0% on Kimi K3. That is 88 of 113 solved (exact 95% CI 69.1% to 85.1%) with DeepSeek V4.1 Flash and 78 of 113 (59.6% to 77.4%) with Kimi K3.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to reproduce
&lt;/h2&gt;

&lt;p&gt;The scripts that run and grade DeepSWE tasks are in &lt;a href="https://github.com/Agent-Field/CodeAF/tree/dev/bench/deepswe" rel="noopener noreferrer"&gt;&lt;code&gt;bench/deepswe/&lt;/code&gt;&lt;/a&gt; in the CodeAF repository. One task through CodeAF:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;bench/deepswe/run.sh &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;task-id&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;model-id&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;seed-tag&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It needs a linux/amd64 &lt;code&gt;codeaf&lt;/code&gt; binary in &lt;code&gt;CODEAF_BIN&lt;/code&gt;, a provider key, and the DeepSWE task corpus, which Datacurve distributes as a &lt;a href="https://huggingface.co/datasets/datacurve/deep-swe" rel="noopener noreferrer"&gt;gated dataset&lt;/a&gt;. The &lt;code&gt;compare-*&lt;/code&gt; scripts beside it install a pinned comparator, freeze its container image and run it. The &lt;a href="https://github.com/Agent-Field/CodeAF/tree/9b4344aaf56048c995cb9caa33369449cf137832/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks" rel="noopener noreferrer"&gt;per-task records&lt;/a&gt; of this campaign are public, so every table on this page can be recounted from them.&lt;/p&gt;

&lt;h2&gt;
  
  
  Questions people ask
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Is CodeAF number one on DeepSWE?
&lt;/h3&gt;

&lt;p&gt;CodeAF was first of ten harnesses in AgentField's DeepSWE comparison, solving 62 of 113 tasks (54.9%) with every harness on DeepSeek V4 Flash. The comparison holds the model fixed and ranks the harness around it.&lt;/p&gt;

&lt;h3&gt;
  
  
  Which AI coding agent scores highest on SWE-bench style benchmarks right now?
&lt;/h3&gt;

&lt;p&gt;When the model is held fixed (DeepSeek V4 Flash) and the harness changes, CodeAF solved the most DeepSWE tasks of the ten harnesses in AgentField's comparison, 62 of 113 (54.9%), ahead of mini-swe-agent (56) and Codex CLI (51).&lt;/p&gt;

&lt;h3&gt;
  
  
  What is the most cost-efficient coding agent per solved issue?
&lt;/h3&gt;

&lt;p&gt;In AgentField's DeepSWE comparison on DeepSeek V4 Flash, CodeAF had the lowest cost per solved issue, 39.9 cents, ahead of mini-swe-agent (1.9x), Codex CLI (2.1x) and Claude Code (3.4x).&lt;/p&gt;

&lt;h3&gt;
  
  
  Does this mean CodeAF beats Claude Code?
&lt;/h3&gt;

&lt;p&gt;On DeepSeek V4 Flash, in AgentField's DeepSWE comparison, CodeAF solved 62 of 113 tasks and Claude Code 2.1.267 solved 16. Both ran DeepSeek V4 Flash, not Claude, so the result measures the harness on that model, not Claude Code with Claude models.&lt;/p&gt;

&lt;h3&gt;
  
  
  Can I reproduce these results?
&lt;/h3&gt;

&lt;p&gt;Yes. The run and grading scripts are in &lt;code&gt;bench/deepswe/&lt;/code&gt; in the CodeAF repository, and the per-task records of all ten harnesses are public, so every count here can be checked. The DeepSWE task corpus is a gated dataset from Datacurve.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://github.com/Agent-Field/CodeAF/tree/d049b13406dc9179f2265608bafca3255f3c5f3a/docs/benchmarks/deepswe" rel="noopener noreferrer"&gt;DeepSWE method page&lt;/a&gt;, with &lt;code&gt;arms.csv&lt;/code&gt; and &lt;code&gt;versions.csv&lt;/code&gt; beside it: every number on this page, checked 2026-11-21.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/Agent-Field/CodeAF/tree/9b4344aaf56048c995cb9caa33369449cf137832/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks" rel="noopener noreferrer"&gt;Per-task records, one CSV per harness&lt;/a&gt;, checked 2026-11-21.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/Agent-Field/CodeAF/tree/dev/bench/deepswe" rel="noopener noreferrer"&gt;&lt;code&gt;bench/deepswe/&lt;/code&gt;&lt;/a&gt; in the CodeAF repository, checked 2026-11-21.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://agentfield.ai/docs/codeaf/senior-dev" rel="noopener noreferrer"&gt;senior-dev docs&lt;/a&gt;, checked 2026-11-21.&lt;/li&gt;
&lt;li&gt;DeepSWE: &lt;a href="https://deepswe.datacurve.ai/" rel="noopener noreferrer"&gt;deepswe.datacurve.ai&lt;/a&gt;, &lt;a href="https://github.com/datacurve-ai/deep-swe" rel="noopener noreferrer"&gt;README&lt;/a&gt;, &lt;a href="https://huggingface.co/datasets/datacurve/deep-swe" rel="noopener noreferrer"&gt;dataset&lt;/a&gt;, checked 2026-11-21.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.swebench.com/verified.html" rel="noopener noreferrer"&gt;SWE-bench Verified&lt;/a&gt;, checked 2026-11-21.&lt;/li&gt;
&lt;li&gt;Release pages of each harness, linked in the versions table, checked 2026-11-21.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For how CodeAF compares beyond this benchmark, see &lt;a href="https://agentfield.ai/compare/codeaf-vs-claude-code" rel="noopener noreferrer"&gt;CodeAF vs Claude Code&lt;/a&gt; and &lt;a href="https://agentfield.ai/compare/codeaf-vs-codex-cli" rel="noopener noreferrer"&gt;CodeAF vs Codex CLI&lt;/a&gt;. For the 3-hour, unattended way senior-dev works, see &lt;a href="https://agentfield.ai/guides/long-running-coding-agent" rel="noopener noreferrer"&gt;a coding agent that runs for hours&lt;/a&gt;, and for the terminal side, &lt;a href="https://agentfield.ai/guides/terminal-coding-agent" rel="noopener noreferrer"&gt;terminal coding agents&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Last updated 2026-11-21. Every figure checked against &lt;code&gt;arms.csv&lt;/code&gt; and &lt;code&gt;versions.csv&lt;/code&gt; at the method page; harness versions are those of the 2026-09-10 to 12 campaign.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>opensource</category>
      <category>benchmark</category>
      <category>programming</category>
    </item>
  </channel>
</rss>
