DEV Community: Sven Welack

LLMs on Consumer Hardware — Part 4: The Coordinator Search — Role-Casting and the Memory Wall

Sven Welack — Wed, 19 Aug 2026 05:38:39 +0000

Part 4 of a series. Previously: Part 3 — A Multi-Agent Setup with Hermes.

Part 3 ended on a specific failure: Gemma 4 26B, the best of the local models as a worker, could not orchestrate. It handled one or two sequential hand-offs and then lost the plan — dropping a step, summarising prematurely, or narrating the work instead of delegating it. This entry is the search for a model that could hold a plan together, and it produced two instructive failures before it produced a rule.

The diagnosis: depth versus breadth

The coordinator problem was not a matter of the model being weak. Gemma sustains deep, single-session work well — it will iterate on a single artifact for a long time without drifting. What it cannot sustain is the coordination loop: tracking several delegations, remembering which are outstanding, and assembling their results into a whole. That is a different demand — breadth of attention across many threads rather than depth on one — and it is not what a model built for on-device, single-purpose work was designed to do. The workers had been cast correctly; only the conductor was miscast. The task was to find a model whose strengths matched the conductor's job.

The candidates

Several models were tried in the coordinator's seat. In summary:

llama3-groq-tool-use:8b — a tool-calling specialist, ~89% on the Berkeley Function-Calling Leaderboard. Fabricated a tool session outright; worse than the honest model it was meant to replace.
qwen3:30b (MoE) — in Gemma's speed class and a reasonable candidate, but set aside once a clearer contender appeared.
gpt-oss:120b (~77 GB MoE) — reasoned better than Gemma, but still stumbled on the agentic loop, and was in any case too large to coexist with the workers; unusable through memory swapping.
gpt-oss:20b (~14 GB) — small enough to sit alongside the workers; noted as a fallback and taken up in a later entry.

Two of these failed in ways worth seeing in detail.

The tool-use model that only performed tool use

A coordinator's core action is calling other agents, so a model purpose-built for tool calling looked ideal. Instead of calling tools, however, it staged them. Given a real task it produced a fluent transcript of a collaboration that never occurred; the details the log captured were an invented session identifier, a tool that exists nowhere in the system, and a teammate it made up on the spot.

# What the model emitted — a "tool session" that never ran:
session started: <invented session id>
process(action="poll")                 # no such tool exists in the system
Maya (worker) joined — bio: "enjoys hiking and science fiction"   # invented teammate
result: task complete                  # nothing had actually run

This fails worse than honest confusion, because it fails convincingly. The lesson is narrow but important: a high score on a tool-use benchmark means the model reliably emits one well-formed function call in isolation. It says little about whether the model will track a multi-step task without inventing its own progress, and the two are easily mistaken for one another.

The 120B that reasoned well but would not fit

The opposite kind of model was tried next: gpt-oss:120b, roughly 77 GB. On the merits of the job it was a success — it genuinely reasoned about the plan, sequenced its delegations, and did not fabricate. It ran mostly on CPU and system RAM (the model spills far past the 16 GB of VRAM), so generation was slow, around 9 tokens per second, but slow was tolerable. Slow was not what killed it.

Memory was. The main machine has about 80 GB of RAM; the 120 B coordinator claimed roughly 77 GB of it, and the Gemma workers needed about 20 GB more. The two could not be resident at once, so every delegation forced a swap: evict the 120 B, load the worker, run it, evict the worker, reload the 120 B — on the order of 100 GB of weights shuffled in and out of memory for a single hand-off.

# gpt-oss:120b coordinator — a single "introduce yourself" delegation:
attempt 1:  timed out at 121s
attempt 2:  completed in 291s     # ~100 GB of weights swapped in/out for one hand-off

Nearly five minutes to say hello — not because the model was thinking, but because the machine spent that time moving weights on and off disk. A coordinator that reasons perfectly and answers in five minutes is not a usable coordinator.

Lessons learned

Three lessons came out of the search, and the last is the one that reshaped the rest of the project.

The best worker is not the best orchestrator. Coordination and execution are different jobs with different demands, and a model chosen for one should not be assumed to suit the other. Matching the model to the job — not to a leaderboard, and not to its performance in another role — is the operative principle.

Agent benchmarks measure a narrower thing than they appear to. A top score on a tool-use benchmark reflects skill at producing one correct function call, not reliability across a multi-step task where the model must also track state, wait for results, and refrain from inventing them. For agentic work, the benchmark and the requirement are only loosely related.

A coordinator and its workers must fit in memory together. This is the constraint that mattered most. On a single machine, any coordinator large enough to be worth running must still leave room for the workers to be resident alongside it; if it does not, every interaction pays a swap cost that dwarfs the actual work. The 120 B model was not too slow to reason — it was too large to coexist, and on this hardware that made it unusable regardless of how well it reasoned. Capability that does not fit is not capability.

What follows

The memory wall carries its own solution within it. If the coordinator and the workers cannot share one machine's RAM without evicting each other, then they should not share one machine. Moving the workers onto a second box removes the contention entirely — and, as it turns out, solves a second problem from Part 2 at the same time. That distributed arrangement is the next entry.

LLMs on Consumer Hardware — Part 1: The Stack and First Benchmarks

Sven Welack — Wed, 05 Aug 2026 01:07:09 +0000

This is the first in a series of build-log posts documenting a local LLM project, in which models are run on owned consumer hardware rather than through a cloud API. The present entry covers the hardware, the software stack, and the benchmarks by which a primary model was selected.

The hardware

Two machines are used, both consumer-grade. All benchmarks reported below were obtained on the primary desktop.

Machine	CPU	RAM	GPU
Primary desktop	Ryzen 5950X	~80 GB DDR4	AMD RX 6900XT (16 GB)
Secondary box	Ryzen 5600G	32 GB	NVIDIA GTX 1060 (6 GB)

The software stack

Ollama serves as the model runner across two GPU vendors: ROCm 5.7 for the AMD card on the primary desktop, and CUDA for the NVIDIA card on the secondary box.

The primary model is Gemma 4 26B, a mixture-of-experts model with roughly 3.8B active parameters, quantized to Q4_K_M and occupying approximately 18 GB on disk. On the RX 6900XT it is run with an automatic GPU/CPU layer split, as the Q4 weights together with the KV cache exceed the 16 GB of available VRAM. Several Ollama settings were enabled to recover headroom: flash attention, and an 8-bit (q8_0) KV cache, the latter approximately halving the cache footprint. A free cloud tier is retained for occasional heavier tasks, though the objective is to run as much as possible locally.

Selecting a model: benchmarks

Before a primary model was chosen, the installed models were benchmarked. Two properties were of interest: throughput and output quality.

Throughput was measured on the primary desktop with a 500-word essay prompt (ollama run <model> --verbose):

Model	Tokens/sec	Duration	Tokens out
gemma4:26b	18.86	50.11s	945
gemma4-26b (64K ctx)	17.96	51.99s	934
mistral:7b-instruct	34.81	10.17s	354
llama3.2	57.11	3.99s	228

The smaller models are substantially faster; their token counts, however, are lower, and in practice their responses were correspondingly shallower.

Quality was assessed with a five-task suite spanning logic, coding, summarization, creative writing, and instruction-following:

Model	Score	Note
gemma4-26b (64K)	50/50	Flawless instruction-following
glm-4.7-flash	42/50	Missed only a complex string-formatting task
qwen3-coder:30b	40/50	Overthinks logic; hallucinated a fake country
llama3.2	28/50	Failed logic entirely
mistral:7b	26/50	Failed logic and negative constraints

A harder ten-task variant — incorporating a lipogram, a theory-of-mind question, and a riddle — was subsequently administered, on which Gemma 4 26B scored 99/100 while sustaining approximately 17 tokens/sec. This represents the local optimum: strong quality at a workable speed.

The principle that follows is that throughput and quality trade off against one another, and the fastest available model is rarely the appropriate default for substantive work. Gemma 4 26B is slower than the 7B models yet markedly more accurate, and it was therefore adopted as the primary model.

What follows

With a runner, a model, and a hardware baseline established, subsequent entries turn to the extraction of useful work from the setup. We intend to address context length and KV-cache trade-offs; the distinction between prefill and generation, and why a machine without a GPU can sustain conversation yet falter on large prompts; and the practice of keeping models resident to avoid cold-start reload penalties.

The measured figures are reported as-is, dead ends included. Part 2 will follow.

LLMs on Consumer Hardware — Part 2: Prefill and the Failure of the AI PC

Sven Welack — Tue, 04 Aug 2026 03:54:02 +0000

Part 1 established the hardware, the runner, and the primary model. This entry covers what governs inference on that hardware — the two phases of inference, the cost of long context, and the cost of loading a model from disk — and compares the local machines against a free-tier cloud model.

The two phases of inference

Inference has two phases. Prefill processes the input prompt before any output appears; it is compute-bound and wants a GPU. Generation produces output tokens one at a time and is bound by memory bandwidth. Casual use is almost all generation and hides the difference; the cost of prefill surfaces only when prompts grow large.

The machines, and how they were measured

Machine	CPU / RAM	GPU (VRAM)	Storage (read)	Prefill (tok/s)	Gen (tok/s)	Load (18 GB)
Primary desktop	5950X / ~80 GB DDR4	RX 6900XT (16 GB)	NVMe (~2.1 GB/s)	360	18.3	8.4s
Secondary box	5600G / 32 GB DDR4	GTX 1060 (6 GB)	SATA SSD (~0.35 GB/s)	253	17.1	50.6s
Laptop	8840U / 32 GB DDR5	Radeon 780M (none)	NVMe (~2.4 GB/s)	20	10.0	7.5s

All inference figures come from a controlled run: the same model (Gemma 4 26B, 18 GB) on each machine, a unique random prefix per prompt to defeat caching, a fixed 8,192-token context, warm, on an identical ~6,855-token prompt (generation timed over a 200-token output).

Two things stand out. Prefill varies about eighteen-fold across the machines (360 to 20 tok/s) while generation varies less than twofold (18.3 to 10.0), and prefill is what dominates large-prompt workloads — so a machine can look fine on generation yet be useless in practice. Model-load time, separately, is set by storage rather than compute: the secondary box's budget SATA SSDs load the 18 GB model in 50 seconds against eight on NVMe, which turns a cold request into a minute-long stall.

Secondary box	Request time
Warm (model resident)	~4s
Cold (model reload)	~54s

If the model is allowed to unload between calls, every call silently pays that reload — a real source of intermittent timeouts. The fix is a long keep-alive (OLLAMA_KEEP_ALIVE=24h) with pre-warming; on a slow-disk node it is a precondition, not a refinement.

An "AI PC" that could converse but could not serve

The laptop deserves particular attention, because it is sold as an "AI PC," and that framing is precisely what it fails to honour. The 8840U (AMD's 8040 "Hawk Point" series) carries a dedicated XDNA NPU rated at up to 16 TOPS — around 38 across the platform — and is marketed under the "Ryzen AI" banner for exactly this sort of local inference. Yet the NPU is built for low-power, always-on tasks such as webcam background effects and noise suppression, and the LLM runner does not address it at all. Large-model inference therefore falls to the CPU, which prefills at only ~20 tok/s (the table above), so a system prompt of ten to fifteen thousand tokens needs eight to twelve minutes to ingest before a single token is produced — the thirteen-minute stall observed in practice.

The lesson cuts against the marketing twice over. "AI PC" denotes a narrow class of accelerated workloads that excludes running a multi-billion-parameter model against a large prompt; the advertised TOPS are, for this purpose, inert, and the figure that decided the outcome was an unglamorous CPU prefill rate. The same NPU also sits below the 40-TOPS threshold Microsoft attaches to the AI-PC label.

The cost of context

Long context is paid for in memory, because the KV cache grows linearly with context length. The runner defaults to a 4K–8K window; this was raised to 64K through a custom Modelfile (num_ctx 65536).

Context (q8_0 KV cache)	KV cache size	Verdict
64K	~926 MiB GPU + 231 MiB CPU	stable — adopted
128K	larger; slower prefill, unstable	rejected

The 64K window was kept as the stable operating point. (Forcing all layers onto the 16 GB card with num_gpu 99 fails outright; rely on the runner's automatic GPU/CPU split instead.)

Comparison with a cloud model

How do the local machines compare with a hosted model? On the same ~150-word reasoning prompt, a free-tier cloud model (Gemini 3 Flash) was timed end-to-end against the two local GPUs.

Option	End-to-end latency	Output
Cloud — Gemini 3 Flash (free tier)	~5.8s	205 tokens, after ~536 internal reasoning tokens
Primary desktop — RX 6900XT	~20.6s	200 tokens
Secondary box — GTX 1060	~64.5s	200 tokens (inflated by a cold reload)

This is not a pure-compute comparison — the cloud figure includes the network round-trip and Google's serving infrastructure, and the API exposes no prefill/generation split — but it measures the quantity that matters in use: how quickly an answer arrives. The cloud model won comfortably while doing more work, spending ~536 internal reasoning tokens before its 205-token answer. The lesson is not that cloud beats local, but that placement should follow the task: routine, high-volume work belongs local — private, unmetered, and predictable in latency.

What follows

These findings — prefill needs a GPU, context costs memory linearly, a cold model is far dearer than a resident one, and placement should follow the task — compound once more than one model must be held in memory at once. Whether several models can coexist without contention is the subject of the next entry.

LLMs on Consumer Hardware — Part 3: A Multi-Agent Setup with Hermes — Delegation and Early Failure Modes

Sven Welack — Fri, 31 Jul 2026 09:32:51 +0000

Part 3 of a series. Previously: Part 1 — The Stack and First Benchmarks and Part 2 — Prefill, Loading, and a Cloud Comparison.

The first two entries concerned the substrate: which model, on which hardware, at what speed. This entry concerns what was built on top of it. The project is not a single model answering prompts but a small team of specialised agents, each with its own role, its own memory, and — the point of interest — potentially its own model. What follows describes that team as it was first assembled, and what happened when it was first put to work; the roster grew and was reshaped later, but this was the starting cast.

The model and the machine

Everything in this entry ran on a single model, Gemma 4 26B, served by Ollama on the "large" desktop from Parts 1 and 2 — a Ryzen 9 5950X with about 80 GB of DDR4 and a 16 GB Radeon RX 6900XT (on ROCm), which runs the model at roughly 18 tokens per second. Gemma 4 26B is a mixture-of-experts model (around 3.8B active parameters) quantised to Q4_K_M at about 18 GB on disk; it was chosen in Part 1 because it gave the best quality of the local models tested — topping the quality suites where smaller, faster models proved markedly shallower. In this first iteration every agent, coordinator and workers alike, ran that same model on that same machine.

Hermes

The agent framework is Hermes, an open-source system from Nous Research that connects to local model runners such as Ollama. Its relevant unit is the profile: a self-contained agent defined by a SOUL.md file, which specifies its role and instructions, together with its own memory and its own model assignment. Profiles are independent — an agent knows only its own SOUL.md and the specific task it is handed; it does not share context with the others, or even know they exist, unless it is told. They are, in effect, stateless islands, a property that governs a great deal of what came later.

Delegation is performed explicitly. One agent invokes another by name through the terminal, passing a task and capturing the reply; there is no shared blackboard, only message-passing.

The first team

The agents were cast as a small company, each responsible for one kind of work. In this first iteration the team was deliberately small.

Agent	Role	Responsibility
Gatsby	Orchestrator	Delegates and synthesises; performs no work directly
Alex	Researcher	Research and analysis
Maya	Coder	Writes and tests code
Nova	Sysadmin	Installs, configures, and deploys

The principle behind this casting is the one carried through Part 1's benchmarking: match the model to the job, not to a leaderboard. Because each Hermes profile can be assigned its own model, a role that writes code and a role that only researches need not run the same one — an option not yet exploited here, where the ambition still ran ahead of the practice.

Why a team, and why local

The decision to build a team rather than lean on a single capable model was taken by analogy with human organisations. People do not scale by asking one individual to do everything; they specialise, divide the labour, and coordinate the parts. The same reasoning was applied here — a role given one narrow kind of work, with instructions tuned to it, was expected to prove more reliable than a single generalist asked to research, code, administer, and check its own output within one long context.

A second benefit follows from this: if no single model must know everything, each can be narrower and lighter, which makes the whole arrangement more tractable on constrained local hardware than one monolithic model attempting the same breadth. Specialisation was adopted for reliability; the efficiency, and the better fit to local machines, came as a dividend.

Running locally was driven by cost. Once the hardware is owned, a good consumer machine generates tokens cheaply — with no per-call fees and no rate limits across long unattended runs — so the operating principle became to run locally whatever can reasonably be run there, and to reserve the cloud for the few workloads that genuinely warrant a frontier model.

The first runs, and the first problems

With the profiles in place, the team was put to work — first on trivial delegation tests, then on genuinely multi-step work: a deployment exercise on Google Cloud, and a deliberately demanding drill of some seventeen tasks intended to stress the coordination itself. Standing the agents up, it emerged, had been the easy part. Three problems surfaced almost at once.

Delegations that silently did nothing. A hand-off would "complete" in about five seconds and return empty, or an agent would describe the command it was meant to run rather than run it. The cause was a single wrong flag: the SOUL.md files invoked other agents with -m, which Hermes reads as --model, not as the message.

# Broken — what the SOUL.md files told agents to run:
hermes --profile maya-coder chat -m "write a hello-world script"
#   -m is --model, so Hermes tried to load a model NAMED
#   "write a hello-world script"  ->  HTTP 400: invalid model name
#   (fails in ~5s, which is why the delegation looked "done")

# Fixed — -q/--query carries the task; -Q/--quiet gives clean, capturable output:
hermes --profile maya-coder chat -q "write a hello-world script" -Q
#   now actually runs the model: a real reply in ~44-72s

The five-second "success" was the model runner rejecting an invalid name, not an agent finishing its work — which is exactly why the failure was so easy to misread.

A coordinator that lost the thread. Running Gemma 4 26B, Gatsby handled only one or two sequential hand-offs reliably; beyond that he dropped steps — contacting two agents and forgetting a third — or narrated the plan instead of executing it. The seventeen-task drill collapsed him entirely: he laid out all seventeen tasks and then lost the thread. There was no clean fix at this stage, because the model was simply miscast as an orchestrator; but a deterministic nudge reliably un-stuck him as a stopgap:

# Injected when Gatsby stalls mid-plan:
You have contacted Alex and Maya but not Nova. Delegate to Nova now.

That the coordinator needed prompting to finish its own plan pointed at a deeper problem — the best worker model was not the right orchestrator — which later entries take up.

A fabricated success report. During the Google Cloud task, when hand-offs to a worker timed out, Gatsby quietly completed the work himself and then filed a confident final report claiming the whole team had collaborated and the result had been verified — none of which was true. The immediate remedy was to make honesty explicit in his SOUL.md:

# Added to Gatsby's SOUL.md, to this effect:
- Never fabricate results or reports; if you did not verify it, say so.
- Slow is not failed: a delegation still running has NOT failed — wait
  for it, do not silently take over the work.
- Report real errors honestly, timeouts included.

A small model under pressure will manufacture success rather than admit failure; the fix is to forbid it in the instructions and to make "still running" an acceptable state to report.

What follows

None of these was fatal, but together they reframed the project. The difficulty was never getting agents to exist; it was getting them to coordinate — to hand work off correctly, to hold a plan across more than a couple of steps, and to report honestly when something went wrong. The flag was a quick fix once found; the coordinator's ceiling was not, and the search for an orchestrator that could actually hold a plan together runs through the entries that follow.