<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Bhaskar Gurram</title>
    <description>The latest articles on DEV Community by Bhaskar Gurram (@bhaskar_gurram_6bc3033721).</description>
    <link>https://dev.to/bhaskar_gurram_6bc3033721</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4254164%2F032bed89-caa0-412d-a38a-95274a864946.jpg</url>
      <title>DEV Community: Bhaskar Gurram</title>
      <link>https://dev.to/bhaskar_gurram_6bc3033721</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/bhaskar_gurram_6bc3033721"/>
    <language>en</language>
    <item>
      <title>I benchmarked my coding agent against seven others. It tied, and it was not the cheapest.</title>
      <dc:creator>Bhaskar Gurram</dc:creator>
      <pubDate>Thu, 19 Nov 2026 16:07:33 +0000</pubDate>
      <link>https://dev.to/bhaskar_gurram_6bc3033721/i-benchmarked-my-coding-agent-against-seven-others-it-tied-and-it-was-not-the-cheapest-5b8i</link>
      <guid>https://dev.to/bhaskar_gurram_6bc3033721/i-benchmarked-my-coding-agent-against-seven-others-it-tied-and-it-was-not-the-cheapest-5b8i</guid>
      <description>&lt;p&gt;I maintain &lt;a href="https://github.com/bhaskargurram-ai/turnloom" rel="noopener noreferrer"&gt;Turnloom&lt;/a&gt;, a small open-source coding agent. Before telling anyone to use it, I wanted to know how it does next to the agents people already use. So I ran eight of them on the same tasks, with the same model, and metered every model call the same way.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Tasks:&lt;/strong&gt; 19 tasks from Terminal-Bench 2.1, picked by a seeded random sample, one attempt each.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Model:&lt;/strong&gt; OpenAI &lt;code&gt;gpt-5.1&lt;/code&gt; for every agent.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Agents:&lt;/strong&gt; OpenCode, Goose, Codex CLI, OpenHands SDK, mini-SWE-agent, Terminus 2, Aider and Turnloom, each at its defaults.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Metering:&lt;/strong&gt; a small proxy between every agent and the API. Token counts are the provider's own figures, and the agents never see the real key.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The result
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Agent&lt;/th&gt;
&lt;th&gt;Solved&lt;/th&gt;
&lt;th&gt;Cost per task&lt;/th&gt;
&lt;th&gt;Input tokens per task&lt;/th&gt;
&lt;th&gt;Output tokens per task&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;OpenCode&lt;/td&gt;
&lt;td&gt;11 of 19&lt;/td&gt;
&lt;td&gt;$0.091&lt;/td&gt;
&lt;td&gt;110,562&lt;/td&gt;
&lt;td&gt;6,060&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Goose&lt;/td&gt;
&lt;td&gt;11 of 19&lt;/td&gt;
&lt;td&gt;$0.101&lt;/td&gt;
&lt;td&gt;180,595&lt;/td&gt;
&lt;td&gt;5,470&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Terminus 2&lt;/td&gt;
&lt;td&gt;11 of 19&lt;/td&gt;
&lt;td&gt;$0.162&lt;/td&gt;
&lt;td&gt;51,228&lt;/td&gt;
&lt;td&gt;14,056&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Turnloom&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;10 of 19&lt;/td&gt;
&lt;td&gt;$0.160&lt;/td&gt;
&lt;td&gt;70,192&lt;/td&gt;
&lt;td&gt;12,637&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;mini-SWE-agent&lt;/td&gt;
&lt;td&gt;10 of 19&lt;/td&gt;
&lt;td&gt;$0.181&lt;/td&gt;
&lt;td&gt;134,693&lt;/td&gt;
&lt;td&gt;14,796&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;OpenHands SDK&lt;/td&gt;
&lt;td&gt;9 of 19&lt;/td&gt;
&lt;td&gt;$0.186&lt;/td&gt;
&lt;td&gt;326,500&lt;/td&gt;
&lt;td&gt;11,477&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Codex CLI&lt;/td&gt;
&lt;td&gt;6 of 19&lt;/td&gt;
&lt;td&gt;$0.156&lt;/td&gt;
&lt;td&gt;275,188&lt;/td&gt;
&lt;td&gt;9,483&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Aider&lt;/td&gt;
&lt;td&gt;1 of 19&lt;/td&gt;
&lt;td&gt;$0.061&lt;/td&gt;
&lt;td&gt;3,626&lt;/td&gt;
&lt;td&gt;5,745&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Six agents solved between 9 and 11 tasks. With 19 tasks, that is a tie: the 95% interval for 10 of 19 runs from 32% to 73%. Aider's row is not a fair picture of Aider, because the benchmark runner gives it one message and no way to run commands.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I learned
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Sending less input did not make Turnloom cheap.&lt;/strong&gt; Turnloom folds old tool output out of the prompt and keeps it in an archive the model can read back. That worked: among the agents that work through a tool loop, only Terminus 2 sent fewer input tokens. But output tokens cost eight times as much as input, and Turnloom wrote about twice as many as the two cheapest agents. Output was $0.126 of its $0.160 per task.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The likely reason is the API.&lt;/strong&gt; Leaving Aider aside, the agents with the fewest output tokens call OpenAI's Responses API, which carries the model's reasoning from one call to the next. Turnloom uses Chat Completions, which does not. I have not tested that explanation yet. It is now on the roadmap.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A benchmark is a good bug finder.&lt;/strong&gt; The run found three problems in Turnloom, including one where an unattended run ended by telling a reader which commands to type. Nobody was reading. All three are fixed in 0.7.1.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this does not show
&lt;/h2&gt;

&lt;p&gt;It is a small sample, run once, on one model, by the author of one of the agents. The &lt;a href="https://github.com/bhaskargurram-ai/turnloom/blob/main/benchmarks/README.md" rel="noopener noreferrer"&gt;benchmark page&lt;/a&gt; lists every limit, the result of every task for every agent, the token counts of every model call, and the scripts. If you run it and get different numbers, I want to know.&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pip install turnloom
turnloom connect
turnloom run --read-only &lt;span class="s2"&gt;"Explain how this project is laid out"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Turnloom is MIT licensed and works with any OpenAI-compatible model, including local ones through Ollama. If you want to help, the &lt;a href="https://github.com/bhaskargurram-ai/turnloom/discussions/17" rel="noopener noreferrer"&gt;announcement&lt;/a&gt; lists where to start.&lt;/p&gt;

</description>
      <category>opensource</category>
      <category>ai</category>
      <category>python</category>
      <category>showdev</category>
    </item>
  </channel>
</rss>
