DEV Community

Cover image for I benchmarked my coding agent against seven others. It tied, and it was not the cheapest.
Bhaskar Gurram
Bhaskar Gurram

Posted on

I benchmarked my coding agent against seven others. It tied, and it was not the cheapest.

I maintain Turnloom, a small open-source coding agent. Before telling anyone to use it, I wanted to know how it does next to the agents people already use. So I ran eight of them on the same tasks, with the same model, and metered every model call the same way.

The setup

  • Tasks: 19 tasks from Terminal-Bench 2.1, picked by a seeded random sample, one attempt each.
  • Model: OpenAI gpt-5.1 for every agent.
  • Agents: OpenCode, Goose, Codex CLI, OpenHands SDK, mini-SWE-agent, Terminus 2, Aider and Turnloom, each at its defaults.
  • Metering: a small proxy between every agent and the API. Token counts are the provider's own figures, and the agents never see the real key.

The result

Agent Solved Cost per task Input tokens per task Output tokens per task
OpenCode 11 of 19 $0.091 110,562 6,060
Goose 11 of 19 $0.101 180,595 5,470
Terminus 2 11 of 19 $0.162 51,228 14,056
Turnloom 10 of 19 $0.160 70,192 12,637
mini-SWE-agent 10 of 19 $0.181 134,693 14,796
OpenHands SDK 9 of 19 $0.186 326,500 11,477
Codex CLI 6 of 19 $0.156 275,188 9,483
Aider 1 of 19 $0.061 3,626 5,745

Six agents solved between 9 and 11 tasks. With 19 tasks, that is a tie: the 95% interval for 10 of 19 runs from 32% to 73%. Aider's row is not a fair picture of Aider, because the benchmark runner gives it one message and no way to run commands.

What I learned

Sending less input did not make Turnloom cheap. Turnloom folds old tool output out of the prompt and keeps it in an archive the model can read back. That worked: among the agents that work through a tool loop, only Terminus 2 sent fewer input tokens. But output tokens cost eight times as much as input, and Turnloom wrote about twice as many as the two cheapest agents. Output was $0.126 of its $0.160 per task.

The likely reason is the API. Leaving Aider aside, the agents with the fewest output tokens call OpenAI's Responses API, which carries the model's reasoning from one call to the next. Turnloom uses Chat Completions, which does not. I have not tested that explanation yet. It is now on the roadmap.

A benchmark is a good bug finder. The run found three problems in Turnloom, including one where an unattended run ended by telling a reader which commands to type. Nobody was reading. All three are fixed in 0.7.1.

What this does not show

It is a small sample, run once, on one model, by the author of one of the agents. The benchmark page lists every limit, the result of every task for every agent, the token counts of every model call, and the scripts. If you run it and get different numbers, I want to know.

Try it

pip install turnloom
turnloom connect
turnloom run --read-only "Explain how this project is laid out"
Enter fullscreen mode Exit fullscreen mode

Turnloom is MIT licensed and works with any OpenAI-compatible model, including local ones through Ollama. If you want to help, the announcement lists where to start.

Top comments (0)