Pi Finished First in a 10-Agent Test on a 4B Model. The Question Is Why

Pi won. Goose and Crush weren't far behind. And the most honest thing about this little benchmark is that it refuses to tell you why — which is exactly why it's worth reading.

A user on r/openclaw ran ten agents on Qwen3.5-4B through five deliberately ordinary tasks: read a port from YAML, write and run a stats script, fix a slugify bug, count errors across log files, rename a function across three files. Ten minutes per task, fresh fixture every time, an automated check script doing the judging. No grand claims about intelligence. Just a workbench test — and workbench tests are where the marketing usually dies.

Pi didn't just win, it lapped the field

Pi (1.3.0) finished all five tasks in 1 minute 36 seconds — 10, 14, 24, 19 and 29 seconds per task. Goose (1.55.0) took 2 minutes 51 seconds, Crush (0.98.1) 3 minutes 31 seconds, both also completing all five. That's not a rounding error. On a 4B model, where every wasted token is time you can feel, Pi's lead is real — even if the reason for it isn't proven yet.

The tempting explanation is still just a guess — and the author says so

The author's guess: Pi wins because it ships four tools (read, write, edit, bash) and a short system prompt, so the model barely has to think about tool formats. On a small model, every extra instruction token shows up in the timing. It's an attractive theory. It's also, by the author's own framing, unmeasured. The test logged no token counts and no tool-call counts, so it can't separate tool-format friction from feature bloat, prompt overhead, or plain luck.

Credit where it's due: most benchmark posts would have dressed that guess up as a finding. This one didn't.

The follow-up actually complicates the story

A follow-up writeup of the same 50 runs found six agents passed all five tasks — Pi, Goose, Crush, Cline, Atomic Agent and Opencode — and that tool calls were not the main failure mode. The failures were exact-match edits, overfitting to visible test data, miscounting, and vague instructions followed literally. In other words: the agents mostly didn't fail at calling tools. They failed at doing the job carefully. That doesn't kill the tool-format theory, but it does mean anyone selling you "fewer tools = faster agent" off this test alone is overselling it.

The commenters asked the questions that matter

• Were these single runs? Repeat each agent-task pairing and report pass rates alongside median times, or one lucky run moves the whole ranking.
• Separate startup failures from task failures. An agent that can't start with a backend tells you about compatibility, not about how it would have done the tasks — counting them the same way muddies the ranking.
• Add token and tool-call logging, plus a sub-2B comparison on the same fixtures. Until then, this is a ranking that raises a question, not a test that answers one.

The hardware advice is refreshingly unpretentious

No, you don't need the expensive new setup. The 5B model at Q4 is about a 2.7 GB file, and the author says it runs fine on an 8 or 16 GB Mac or any GPU with 6–8 GB of VRAM — the M5 machine used for the test was overkill. His rule for a 16 GB machine: stick to 4B–9B models at Q4, keep context at 16,000–32,000 tokens, because a big context is what kills speed on smaller Macs. That's advice, not a measured result — but it's the kind of practical honesty the agent-tooling world could use more of.

Go deeper

This is the condensed version — the full analysis covers the task-by-task times, the failure modes, and the next-round questions that would turn this ranking into a real test: Pi Finished First in a 10-Agent Test on a 4B Model. The Question Is Why

---

Originally published on Agentic Affairs (https://agentic-affairs.com)