Part 4 of a series. Previously: Part 3 — A Multi-Agent Setup with Hermes.
Part 3 ended on a specific failure: Gemma 4 26B, the best of the local models as a worker, could not orchestrate. It handled one or two sequential hand-offs and then lost the plan — dropping a step, summarising prematurely, or narrating the work instead of delegating it. This entry is the search for a model that could hold a plan together, and it produced two instructive failures before it produced a rule.
The diagnosis: depth versus breadth
The coordinator problem was not a matter of the model being weak. Gemma sustains deep, single-session work well — it will iterate on a single artifact for a long time without drifting. What it cannot sustain is the coordination loop: tracking several delegations, remembering which are outstanding, and assembling their results into a whole. That is a different demand — breadth of attention across many threads rather than depth on one — and it is not what a model built for on-device, single-purpose work was designed to do. The workers had been cast correctly; only the conductor was miscast. The task was to find a model whose strengths matched the conductor's job.
The candidates
Several models were tried in the coordinator's seat. In summary:
-
llama3-groq-tool-use:8b— a tool-calling specialist, ~89% on the Berkeley Function-Calling Leaderboard. Fabricated a tool session outright; worse than the honest model it was meant to replace. -
qwen3:30b(MoE) — in Gemma's speed class and a reasonable candidate, but set aside once a clearer contender appeared. -
gpt-oss:120b(~77 GB MoE) — reasoned better than Gemma, but still stumbled on the agentic loop, and was in any case too large to coexist with the workers; unusable through memory swapping. -
gpt-oss:20b(~14 GB) — small enough to sit alongside the workers; noted as a fallback and taken up in a later entry.
Two of these failed in ways worth seeing in detail.
The tool-use model that only performed tool use
A coordinator's core action is calling other agents, so a model purpose-built for tool calling looked ideal. Instead of calling tools, however, it staged them. Given a real task it produced a fluent transcript of a collaboration that never occurred; the details the log captured were an invented session identifier, a tool that exists nowhere in the system, and a teammate it made up on the spot.
# What the model emitted — a "tool session" that never ran:
session started: <invented session id>
process(action="poll") # no such tool exists in the system
Maya (worker) joined — bio: "enjoys hiking and science fiction" # invented teammate
result: task complete # nothing had actually run
This fails worse than honest confusion, because it fails convincingly. The lesson is narrow but important: a high score on a tool-use benchmark means the model reliably emits one well-formed function call in isolation. It says little about whether the model will track a multi-step task without inventing its own progress, and the two are easily mistaken for one another.
The 120B that reasoned well but would not fit
The opposite kind of model was tried next: gpt-oss:120b, roughly 77 GB. On the merits of the job it was a success — it genuinely reasoned about the plan, sequenced its delegations, and did not fabricate. It ran mostly on CPU and system RAM (the model spills far past the 16 GB of VRAM), so generation was slow, around 9 tokens per second, but slow was tolerable. Slow was not what killed it.
Memory was. The main machine has about 80 GB of RAM; the 120 B coordinator claimed roughly 77 GB of it, and the Gemma workers needed about 20 GB more. The two could not be resident at once, so every delegation forced a swap: evict the 120 B, load the worker, run it, evict the worker, reload the 120 B — on the order of 100 GB of weights shuffled in and out of memory for a single hand-off.
# gpt-oss:120b coordinator — a single "introduce yourself" delegation:
attempt 1: timed out at 121s
attempt 2: completed in 291s # ~100 GB of weights swapped in/out for one hand-off
Nearly five minutes to say hello — not because the model was thinking, but because the machine spent that time moving weights on and off disk. A coordinator that reasons perfectly and answers in five minutes is not a usable coordinator.
Lessons learned
Three lessons came out of the search, and the last is the one that reshaped the rest of the project.
The best worker is not the best orchestrator. Coordination and execution are different jobs with different demands, and a model chosen for one should not be assumed to suit the other. Matching the model to the job — not to a leaderboard, and not to its performance in another role — is the operative principle.
Agent benchmarks measure a narrower thing than they appear to. A top score on a tool-use benchmark reflects skill at producing one correct function call, not reliability across a multi-step task where the model must also track state, wait for results, and refrain from inventing them. For agentic work, the benchmark and the requirement are only loosely related.
A coordinator and its workers must fit in memory together. This is the constraint that mattered most. On a single machine, any coordinator large enough to be worth running must still leave room for the workers to be resident alongside it; if it does not, every interaction pays a swap cost that dwarfs the actual work. The 120 B model was not too slow to reason — it was too large to coexist, and on this hardware that made it unusable regardless of how well it reasoned. Capability that does not fit is not capability.
What follows
The memory wall carries its own solution within it. If the coordinator and the workers cannot share one machine's RAM without evicting each other, then they should not share one machine. Moving the workers onto a second box removes the contention entirely — and, as it turns out, solves a second problem from Part 2 at the same time. That distributed arrangement is the next entry.
Top comments (0)