I maintain Turnloom, a small open-source coding agent. Before telling anyone to use it, I wanted to know how it does next to the agents people already use. So I ran eight of them on the same tasks, with the same model, and metered every model call the same way.
The setup
- Tasks: 19 tasks from Terminal-Bench 2.1, picked by a seeded random sample, one attempt each.
-
Model: OpenAI
gpt-5.1for every agent. - Agents: OpenCode, Goose, Codex CLI, OpenHands SDK, mini-SWE-agent, Terminus 2, Aider and Turnloom, each at its defaults.
- Metering: a small proxy between every agent and the API. Token counts are the provider's own figures, and the agents never see the real key.
The result
| Agent | Solved | Cost per task | Input tokens per task | Output tokens per task |
|---|---|---|---|---|
| OpenCode | 11 of 19 | $0.091 | 110,562 | 6,060 |
| Goose | 11 of 19 | $0.101 | 180,595 | 5,470 |
| Terminus 2 | 11 of 19 | $0.162 | 51,228 | 14,056 |
| Turnloom | 10 of 19 | $0.160 | 70,192 | 12,637 |
| mini-SWE-agent | 10 of 19 | $0.181 | 134,693 | 14,796 |
| OpenHands SDK | 9 of 19 | $0.186 | 326,500 | 11,477 |
| Codex CLI | 6 of 19 | $0.156 | 275,188 | 9,483 |
| Aider | 1 of 19 | $0.061 | 3,626 | 5,745 |
Six agents solved between 9 and 11 tasks. With 19 tasks, that is a tie: the 95% interval for 10 of 19 runs from 32% to 73%. Aider's row is not a fair picture of Aider, because the benchmark runner gives it one message and no way to run commands.
What I learned
Sending less input did not make Turnloom cheap. Turnloom folds old tool output out of the prompt and keeps it in an archive the model can read back. That worked: among the agents that work through a tool loop, only Terminus 2 sent fewer input tokens. But output tokens cost eight times as much as input, and Turnloom wrote about twice as many as the two cheapest agents. Output was $0.126 of its $0.160 per task.
The likely reason is the API. Leaving Aider aside, the agents with the fewest output tokens call OpenAI's Responses API, which carries the model's reasoning from one call to the next. Turnloom uses Chat Completions, which does not. I have not tested that explanation yet. It is now on the roadmap.
A benchmark is a good bug finder. The run found three problems in Turnloom, including one where an unattended run ended by telling a reader which commands to type. Nobody was reading. All three are fixed in 0.7.1.
What this does not show
It is a small sample, run once, on one model, by the author of one of the agents. The benchmark page lists every limit, the result of every task for every agent, the token counts of every model call, and the scripts. If you run it and get different numbers, I want to know.
Try it
pip install turnloom
turnloom connect
turnloom run --read-only "Explain how this project is laid out"
Turnloom is MIT licensed and works with any OpenAI-compatible model, including local ones through Ollama. If you want to help, the announcement lists where to start.
Top comments (0)