In my last post, benchmarking a small local LLM exposed a bug in my own harness — the "weak model" was partly my code. That raised the obvious next question: how much does the tool around the model matter, measured across many tools?
So I ran 14 agent-type AI coding tools through the same task set, on the same ordinary PC (Windows 11, Ryzen 5 4500U — no GPU, nothing special), with two model backends:
- BYOK: gemini-3.5-flash-lite via each user's own API key
- Local LLM: qwen3.5:4b on Ollama
Scope: agent-type tools only — things that autonomously read files, edit files, and run commands. Chat assistants and subscription products (Copilot, Claude Code, Cursor, Devin) are out of scope.
What I measured
Real tasks: create a file, search/read and answer, perform a precise edit, diagnose+fix a failing test. Success rates count only the last 2 attempts per task, and results identical to baseline ("did nothing but claimed done") are marked failed.
BYOK results (gemini-3.5-flash-lite)
| Tool | Passes |
|---|---|
| Cline (VS Code ext) | 7/7 (100%) |
| Kilo Code, OpenHands, OpenCode, Aider | 6/6 (100%) |
| Teaspoon IDE (standalone Electron) | 6/7 (86%) |
| Zoo Code, Goose | 5/6 (83%) |
| Continue | 2/5 (40%) |
| SWE-agent, Void, Roo Code | 0–1 attempts (0%) |
Same model, same tasks — and the spread is 100% → 0%. The model is constant; the harness (prompt assembly, tool-call format, retry logic, context handling) is what varies.
Local LLM results (qwen3.5:4b, Ollama)
| Tool | Passes |
|---|---|
| Cline | 6/9 (67%) |
| Teaspoon IDE | 6/9 (67%) |
| Void | 2/3 (67%) |
| Aider | 2/4 (50%) |
| Zoo Code, Kilo Code, Continue, OpenCode, Goose | ~25% |
| Roo Code, OpenHands, Devika, Plandex | 0% |
The weak local model is a harsher judge: tools whose harnesses were tuned for strong cloud APIs fall apart (brittle tool-call formats, no retry on malformed output), while tools with tolerant parsing and explicit command protocols still complete tasks.
What the numbers actually show
- "This model is good/bad" claims usually include the harness. Same backbone model, different success rate — that's tool-side design, not intelligence.
- Small local models are the best harness test. A 4B on CPU has no slack; every weakness in the agent loop surfaces as a visible failure.
- Dead ecosystems show up in measurements, not marketing. Roo Code's repo is archived; attempts got no response on either backend — the bench records it as "not evaluable" instead of pretending otherwise.
- AI-vs-AI reviews add a second axis. I also had ChatGPT, Gemini, Claude, and Devin (SWE-2) score each tool's beginner-friendliness 0–5 by reading the measured data: Teaspoon IDE 3.8, Cline 3.5, everything else ≤2.8.
Caveats
- Scores reflect the last two runs — small n by design (agent runs are slow on this hardware). Full per-attempt history is published per tool.
- SWE-agent and Plandex hit environment constraints (Windows/autocrlf, no native build) — recorded as not evaluable, not failed.
- I'm the author of one tool in the table (Teaspoon IDE). The measurement protocol is identical for all tools and the raw logs are public — judge for yourself.
Links
- Full tables, feature matrix, and per-tool detail pages: https://labo.mond.jp/articles/editor-benchmark.en.html
- All site content: https://labo.mond.jp/index.en.html
- My previous post (the harness bug that started this): https://dev.to/cuculhart/i-thought-the-local-llm-was-weak-it-was-a-bug-in-my-own-harness-5dg0
If you're choosing an agent tool, don't ask "which model is it using" — ask "what does the harness do when the model misbehaves." That's where the success rate lives.
Top comments (2)
Nice design keeping the model fixed. Two things worth logging per run on the Ollama backend, because they can turn a harness result into a config result:
num_ctx. Ollama's default context window is often far smaller than what agents with long system prompts and tool schemas send on the first turn, and the overflow is truncated on the server with only a log warning. That shows up as "did nothing" or malformed tool calls, not as an error. Setting num_ctx explicitly per run (and recording the first request's prompt token count) would separate "can't drive a small model" from "never saw its own instructions".
Tool-call mode. Some tools rely only on native function calling (Roo Code's docs say it uses native tool calling exclusively), others parse tool calls out of text. A model/template pair that doesn't expose tools through Ollama's OpenAI-compatible endpoint fails the first group regardless of harness quality. One column for native vs text tool calls would make the low rows easier to read.
Thanks — both are real confounds, and the first one is a wall I've already hit myself.
num_ctx — agreed, and I have the receipt for that exact failure mode. In the earlier model-level bench on this same machine, Ollama's default (4096) made agent loops silently lose earlier context mid-run — the model's "I read it but can't see the contents" symptom — with nothing on the surface except a server-side warning. So a 4B that "did nothing" can absolutely mean "never saw its own instructions."
The honest gap: per-run context size wasn't logged in this table, and for most tools it can't be injected anyway — num_ctx isn't exposed through the OpenAI-compatible endpoint most of them use. The knobs are the server-side default (OLLAMA_CONTEXT_LENGTH) and whatever each tool surfaces (Cline has a context setting; most don't). The right fix isn't re-scoring but flagging: record the first-turn prompt token count where it's visible, and mark rows whose payload plausibly exceeded the effective window as "context-limited" rather than plain failures. I'll add that.
Tool-call mode — agreed, and it changes how the low rows read. Roo Code is the extreme case: native function calling only, an archived repo, and no response even on BYOK — so its row was already "not evaluable," not "failed." For the other 0% rows, whether Ollama's endpoint exposed tools for this model/template at all is exactly the line between "bad harness" and "incompatible transport." A native-vs-text column is cheap and makes the bottom of the table legible — I'll add it to the feature matrix.