DEV Community: Sven Welack

Running LLMs on Consumer Hardware — Part 7: The Strategist — Judgment on a Cloud Model, Execution Local

Sven Welack — Mon, 19 Oct 2026 10:32:39 +0000

Part 7 of a series. Previously: Part 6 — The Orchestration Ceiling.

Part 6 made the team's execution reliable: by moving the control loop out of the model and into deterministic code, the orchestrator could carry out a plan without dropping steps. But reliable execution answers only half of the problem. A plan carried out faithfully is of little use once it meets a wall the plan did not anticipate — and for that the team needed a capability that execution reliability could not supply: judgment. Judgment was the one thing served poorly by any local model, and it is the single part of the system that runs in the cloud. This entry explains why, and what it bought.

The stall-versus-improvise problem

Earlier entries established Gatsby, the orchestrator, as "the honest operator": reliable, never fabricating a result, never improvising wildly. That discipline is exactly what an orchestrator needs, but it created a tension at the team's edges. When the team hit a genuine blocker — a download that failed, a dependency that was absent — an honest, non-improvising operator has only two available moves, and both are wrong. It can stall: report the obstacle and stop. Or it can improvise: depart from the plan, and risk exactly the kind of unfounded, made-up progress the honesty rules existed to prevent.

What was missing was a third option — adaptation: a way to rethink the approach without either giving up or inventing. The problem was that reliability and creative rethinking are, to some extent, opposing traits, and one agent had been asked to supply both.

Separating judgment from execution

The resolution was to stop asking a single agent to be both dependable and inventive. Execution — the honest, faithful carrying-out of a plan — stays with Gatsby. Judgment — the creative rethinking that a wall calls for — moves to a new role: Sage, the strategist. Gatsby no longer improvises; when he is blocked, he consults Sage, and Sage does the adapting. Judgment lives in Sage; honest execution lives in Gatsby. The escalation was made explicit with a single rule added to Gatsby's instructions — when blocked, consult Sage — so that hitting a wall triggers a consultation rather than a stall or a guess.

Why this role — and only this role — is in the cloud

The operating principle set out earlier in the series was to run locally whatever reasonably can be, and to reserve the cloud for the few workloads that genuinely warrant a frontier model. Strategy is that workload, for two reasons.

First, it is invoked rarely — only when the team is actually stuck — so its cost is negligible. A role called a handful of times across a long run is precisely the kind of workload a free cloud tier serves without strain, and it consumes no local resources at all. Second, it is the one role whose quality turns on depth of reasoning rather than on throughput, and depth is where a frontier model sits in a different class.

That difference is measurable. Asked what makes a strategy good, the cloud model spent 413 internal "thinking" tokens before producing a 24-token answer — it reasoned for roughly seventeen times the length of what it ultimately said. That ratio is the entire point of the role: a strategist should think far more than it speaks. The local models used elsewhere in this system do not reason internally in this way; the capability is specific to the frontier model, and it is exactly what the role requires.

The free-tier reality

The specifics mattered, because the free tier had narrowed by mid-2026. Gemini's Pro models left it on 1 April 2026, leaving only the Flash and Flash-Lite tiers free. The role therefore landed on gemini-3-flash-preview: free, with a one-million-token context window — large enough that the context limits that shaped the local models are simply irrelevant here — and configurable "thinking" levels. A frontier-class strategist, consulted occasionally, at no cost.

Sage in practice

The value showed on a real blocker. The team was building a game and could not download the graphics it wanted; the network was blocked, and the workers treated this as a hard stop. Sage reframed it. The team, it observed, was conflating graphics with external downloads — and the two are not the same thing. It recommended procedural synthesis: generating the graphics in code rather than fetching them, with concrete next steps for each member of the team.

The detail that marked this as judgment rather than a lucky guess was anticipatory. Sage also directed the team to audit the project for other hidden external dependencies — before they hit the same wall somewhere else. That is the difference a reasoning model makes to the role: it did not merely solve the blocker in front of it, but recognised the class of problem behind it and moved to pre-empt the next instance.

A second case showed the same judgment turned the other way — toward stopping. Tasked with installing a set of system packages, Nova, the sysadmin, hit a wall that no amount of reframing could get past: the installation required a sudo password, which the agents did not have and should not have. Here the escalation path proved its worth in the opposite direction. Rather than inventing a workaround or declaring false success, the team routed the blocker up through Sage, and the judgment that came back was to surface it to the human. Credentials are exactly the kind of thing an autonomous system should stop and ask for, and the design held end to end: no fabrication, no stalling in place, just an honest hand-off to the one party who could supply what was missing.

What follows

With judgment and execution separated, the team could at last both stay honest and adapt. But a strategist that reframes a problem well does nothing to guarantee that the work it unblocks is actually verified — and verification turned out to be the team's weakest link. The next entry is about quality assurance, and about why "it ran" and "it works" are not the same claim.

LLMs on Consumer Hardware — Part 6: The Orchestration Ceiling — Why a Bigger Coordinator Doesn't Fix the Loop

Sven Welack — Sun, 18 Oct 2026 05:03:54 +0000

Part 6 of a series. Previously: Part 5 — Distributing the Workers and the Memory Budget.

Part 5 ended with the 77 GB coordinator finally running as it should — resident, no thrashing, the memory problem resolved by moving the workers to a second machine. That made it possible to answer a question Part 4 had deferred: now that the big model can actually run, does a bigger coordinator fix orchestration? The answer was no, and the reason reshaped how the team was run.

The ceiling persists on the big model

With memory no longer the bottleneck, the coordinator's orchestration could be judged on its own merits. It sequenced delegations more reliably than Gemma had — but it did not escape the same failure modes. On longer plans it still dropped steps, issuing some delegations and silently omitting others before writing its summary. It still, on occasion, emitted a delegation as text rather than executing it — Part 4 showed one such case, a hand-off printed as raw JSON and then abandoned. And it periodically returned nothing at all after a tool call, requiring an automatic nudge to continue:

⚠ Model returned empty after tool calls — nudging to continue

That warning recurred often enough to be diagnostic. These were not the mistakes of a weak model; they were the mistakes of a capable model asked to do something it is structurally poor at — hold a long control loop in its head while also doing the reasoning inside it.

The diagnosis: the ceiling is the loop, not the model

The decisive observation was that all three failure modes — dropped steps, commands narrated instead of executed, empty turns — appeared on the large model exactly as they had on the small one. The common factor was not model size. It was the agentic loop itself.

Asking a language model to orchestrate is asking it to be, simultaneously, the planner that decides the steps, the dispatcher that issues each delegation, the bookkeeper that tracks which delegations are still outstanding, and the evaluator that judges the results as they return. That is a great deal of state to carry across many turns, and carrying state across turns is precisely what these models are weakest at. A bigger model raises the ceiling a little; it does not remove it.

The fix: take the loop away from the model

If the loop is the problem, the loop should not be the model's job. The resolution was deterministic orchestration: a small script owns the control loop. It holds the list of tasks, dispatches each delegation, waits for and collects the result, and decides what runs next. The model is invoked only for the two things it is genuinely good at — planning (what are the steps?) and evaluating (is this result acceptable?) — and never has to remember what it has already dispatched, because the script does that.

The improvement was immediate and measurable. A task sequence that the model-driven loop had stalled on partway through — completing three of five steps before losing the thread — ran all five to completion when the same plan was executed by the script. The model had not become more capable. It had simply been relieved of the one job it could not reliably do.

The lesson

The instinct, when an agent loop is unreliable, is to reach for a bigger model. That instinct was wrong here, and expensively so — it was the reasoning behind the whole coordinator search of Part 4, and the memory wall of Part 5, and in the end it did not solve the problem it was chasing. Orchestration reliability is not a model capability that can be bought in larger quantities; it is a property of how the loop is structured. The durable fix was architectural: move the deterministic part — the loop, the bookkeeping, the dispatch — into deterministic code, and reserve the model for the parts that genuinely require judgment. A control loop is a job for a program, not for a language model, and the sooner that division is drawn the more reliable the system becomes.

What follows

Deterministic orchestration made the team's execution reliable: a plan was now carried out without dropped steps or invented progress. But it did nothing for the moment a plan meets a wall — when the right next step is not in the plan at all, and someone has to rethink the approach rather than merely execute it. Reliable execution and good judgment are different capabilities, and the next entry is about supplying the second without compromising the first.

LLMs on Consumer Hardware — Part 5: Distributing the Workers and the Memory Budget

Sven Welack — Wed, 14 Oct 2026 07:40:39 +0000

Part 4 ended against a wall. A coordinator capable enough to be worth running — the 77 GB gpt-oss:120b — could not share a machine with the workers it had to call; every delegation forced roughly 100 GB of model-swapping, and a greeting took nearly five minutes. The conclusion was that a coordinator and its workers must fit in memory together. This entry takes that constraint at its word and does the obvious thing: it stops making them share.

Moving the workers off the machine

The fix was to put the workers on a different machine. The second box introduced in Part 2 — the 5600G with the GTX 1060 — became the worker host; this is how it got that job. With the workers resident there, the main machine's memory was no longer contested: the coordinator had it to itself, and nothing local evicted it between hand-offs.

The effect on the 120B was immediate. The same model that had thrashed itself into uselessness now loaded once and stayed resident for 1 hour 46 minutes across a long run, with no swapping at all. Freed from the swap penalty, its orchestration was also more reliable than Gemma's had been. The lesson was almost deflating in its simplicity: the coordinator had never been the problem. Its placement had.

The second machine thereby earned its keep twice. Part 2 put it there because its discrete GPU fixed prefill — a GPU-less box could not ingest the large prompts in any reasonable time. Moving the workers onto it solved a second, unrelated problem: it freed the main machine's RAM for the coordinator. One hardware decision, two constraints resolved.

One quirk of the arrangement mattered for what came next. The 120B is far too large for the 16 GB of VRAM, so it runs mostly on CPU and system RAM — slow, around 9 tokens per second, but leaving the GPU almost entirely idle during inference. The main machine now had a capable graphics card doing nothing.

A new tenant: image generation

That idle GPU was an invitation. Local image generation (ComfyUI with Stable Diffusion) was added on the main machine's 6900XT, giving the agents the ability to produce images as well as text and code. On its own it was quick — an SDXL image in roughly 15 to 30 seconds.

The difficulty was that the main machine now had two heavy tenants. The coordinator claimed about 77 GB of the 80 GB of RAM; image generation needed the GPU and a few gigabytes of RAM of its own. The GPU was free — the coordinator was not using it — but memory, once again, was not.

The experiment: can they coexist?

This was worth testing rather than guessing, because there was a plausible argument that it would work. During a delegation the coordinator is idle: it has handed a task to a remote worker and is waiting. Idle memory pages swap out cheaply; only actively-used memory thrashes. So in principle Linux should be able to page the waiting coordinator out, let image generation use the freed RAM, and page it back in once when the coordinator resumed. An idle tenant ought to be a good neighbour.

Two safeguards were built before the test: a RAM guard on the image generator that refuses to start if less than 8 GB is free — failing cleanly rather than driving the machine into swap-death — and an automatic unload after each image, via ComfyUI's /free, which genuinely releases the model and drops VRAM from about 8 GB back to a 2.2 GB baseline. Then the guard was disabled, the 120B was loaded, and a multi-image run was launched while free -h and vmstat were watched.

The hypothesis failed. With the 120B resident, only about 3 GB of RAM was free, and that was not enough.

# main machine, 120B coordinator resident (~77 GB), multi-image run:
free -h:   ~3 GB free
image 1:   completed
image 2:   timed out at 300s  ->  placeholder
image 3:   timed out at 300s  ->  placeholder
   ...

The first image completed; the rest timed out at the 300-second limit and fell back to placeholders. The elegant argument about idle pages was not wrong in principle — it was simply overwhelmed in practice. Under that much memory pressure each GPU generation stretched past five minutes, and "paged out cheaply" became "paged out constantly." The machine degraded gracefully rather than crashing, but the capability was effectively blocked.

Right-sizing the coordinator

The resolution was not more hardware but less model. The 120B had always been marginal: it consumed essentially the entire memory budget on its own, which is what made both the original worker-swap and this image-generation conflict inevitable. Dropping the coordinator to gpt-oss:20b, around 14 GB, leaves roughly 60 GB free — enough for coordination and image generation at once, with no swap and no guard refusals.

The trade looked worse than it was. The 120B's one real advantage was more reliable orchestration, and — as a later entry shows — even that did not fix the deeper reliability problem it was meant to. A coordinator that costs the whole machine to run, and still leaves the hardest problem unsolved, is difficult to justify over one a fifth its size.

The memory budget

The throughline of this entry is a single idea: a local multi-model system is governed by a memory budget. The question is never whether one model fits, but whether everything that must be resident at once fits together — coordinator, workers, and now an image generator. Two levers control that budget, and both beat buying a bigger model: placement (which machine a thing runs on) and right-sizing (how large each model needs to be for its job). Distributing the workers was placement; shrinking the coordinator was right-sizing. Between them, the contention that had defined the previous two entries simply went away.

What follows

Settling the memory question left a reasonable assumption in place: that the larger coordinator, now that it could finally run, was the better one. The next entry tests that assumption directly — and finds that a bigger model buys more reliable sequencing but not a reliable loop, which turns out to need something other than a model at all.

LLMs on Consumer Hardware — Part 4: The Coordinator Search — Role-Casting and the Memory Wall

Sven Welack — Wed, 19 Aug 2026 05:38:39 +0000

Part 4 of a series. Previously: Part 3 — A Multi-Agent Setup with Hermes.

Part 3 ended on a specific failure: Gemma 4 26B, the best of the local models as a worker, could not orchestrate. It handled one or two sequential hand-offs and then lost the plan — dropping a step, summarising prematurely, or narrating the work instead of delegating it. This entry is the search for a model that could hold a plan together, and it produced two instructive failures before it produced a rule.

The diagnosis: depth versus breadth

The coordinator problem was not a matter of the model being weak. Gemma sustains deep, single-session work well — it will iterate on a single artifact for a long time without drifting. What it cannot sustain is the coordination loop: tracking several delegations, remembering which are outstanding, and assembling their results into a whole. That is a different demand — breadth of attention across many threads rather than depth on one — and it is not what a model built for on-device, single-purpose work was designed to do. The workers had been cast correctly; only the conductor was miscast. The task was to find a model whose strengths matched the conductor's job.

The candidates

Several models were tried in the coordinator's seat. In summary:

llama3-groq-tool-use:8b — a tool-calling specialist, ~89% on the Berkeley Function-Calling Leaderboard. Fabricated a tool session outright; worse than the honest model it was meant to replace.
qwen3:30b (MoE) — in Gemma's speed class and a reasonable candidate, but set aside once a clearer contender appeared.
gpt-oss:120b (~77 GB MoE) — reasoned better than Gemma, but still stumbled on the agentic loop, and was in any case too large to coexist with the workers; unusable through memory swapping.
gpt-oss:20b (~14 GB) — small enough to sit alongside the workers; noted as a fallback and taken up in a later entry.

Two of these failed in ways worth seeing in detail.

The tool-use model that only performed tool use

A coordinator's core action is calling other agents, so a model purpose-built for tool calling looked ideal. Instead of calling tools, however, it staged them. Given a real task it produced a fluent transcript of a collaboration that never occurred; the details the log captured were an invented session identifier, a tool that exists nowhere in the system, and a teammate it made up on the spot.

# What the model emitted — a "tool session" that never ran:
session started: <invented session id>
process(action="poll")                 # no such tool exists in the system
Maya (worker) joined — bio: "enjoys hiking and science fiction"   # invented teammate
result: task complete                  # nothing had actually run

This fails worse than honest confusion, because it fails convincingly. The lesson is narrow but important: a high score on a tool-use benchmark means the model reliably emits one well-formed function call in isolation. It says little about whether the model will track a multi-step task without inventing its own progress, and the two are easily mistaken for one another.

The 120B that reasoned well but would not fit

The opposite kind of model was tried next: gpt-oss:120b, roughly 77 GB. On the merits of the job it was a success — it genuinely reasoned about the plan, sequenced its delegations, and did not fabricate. It ran mostly on CPU and system RAM (the model spills far past the 16 GB of VRAM), so generation was slow, around 9 tokens per second, but slow was tolerable. Slow was not what killed it.

Memory was. The main machine has about 80 GB of RAM; the 120 B coordinator claimed roughly 77 GB of it, and the Gemma workers needed about 20 GB more. The two could not be resident at once, so every delegation forced a swap: evict the 120 B, load the worker, run it, evict the worker, reload the 120 B — on the order of 100 GB of weights shuffled in and out of memory for a single hand-off.

# gpt-oss:120b coordinator — a single "introduce yourself" delegation:
attempt 1:  timed out at 121s
attempt 2:  completed in 291s     # ~100 GB of weights swapped in/out for one hand-off

Nearly five minutes to say hello — not because the model was thinking, but because the machine spent that time moving weights on and off disk. A coordinator that reasons perfectly and answers in five minutes is not a usable coordinator.

Lessons learned

Three lessons came out of the search, and the last is the one that reshaped the rest of the project.

The best worker is not the best orchestrator. Coordination and execution are different jobs with different demands, and a model chosen for one should not be assumed to suit the other. Matching the model to the job — not to a leaderboard, and not to its performance in another role — is the operative principle.

Agent benchmarks measure a narrower thing than they appear to. A top score on a tool-use benchmark reflects skill at producing one correct function call, not reliability across a multi-step task where the model must also track state, wait for results, and refrain from inventing them. For agentic work, the benchmark and the requirement are only loosely related.

A coordinator and its workers must fit in memory together. This is the constraint that mattered most. On a single machine, any coordinator large enough to be worth running must still leave room for the workers to be resident alongside it; if it does not, every interaction pays a swap cost that dwarfs the actual work. The 120 B model was not too slow to reason — it was too large to coexist, and on this hardware that made it unusable regardless of how well it reasoned. Capability that does not fit is not capability.

What follows

The memory wall carries its own solution within it. If the coordinator and the workers cannot share one machine's RAM without evicting each other, then they should not share one machine. Moving the workers onto a second box removes the contention entirely — and, as it turns out, solves a second problem from Part 2 at the same time. That distributed arrangement is the next entry.

LLMs on Consumer Hardware — Part 1: The Stack and First Benchmarks

Sven Welack — Wed, 05 Aug 2026 01:07:09 +0000

This is the first in a series of build-log posts documenting a local LLM project, in which models are run on owned consumer hardware rather than through a cloud API. The present entry covers the hardware, the software stack, and the benchmarks by which a primary model was selected.

The hardware

Two machines are used, both consumer-grade. All benchmarks reported below were obtained on the primary desktop.

Machine	CPU	RAM	GPU
Primary desktop	Ryzen 5950X	~80 GB DDR4	AMD RX 6900XT (16 GB)
Secondary box	Ryzen 5600G	32 GB	NVIDIA GTX 1060 (6 GB)

The software stack

Ollama serves as the model runner across two GPU vendors: ROCm 5.7 for the AMD card on the primary desktop, and CUDA for the NVIDIA card on the secondary box.

The primary model is Gemma 4 26B, a mixture-of-experts model with roughly 3.8B active parameters, quantized to Q4_K_M and occupying approximately 18 GB on disk. On the RX 6900XT it is run with an automatic GPU/CPU layer split, as the Q4 weights together with the KV cache exceed the 16 GB of available VRAM. Several Ollama settings were enabled to recover headroom: flash attention, and an 8-bit (q8_0) KV cache, the latter approximately halving the cache footprint. A free cloud tier is retained for occasional heavier tasks, though the objective is to run as much as possible locally.

Selecting a model: benchmarks

Before a primary model was chosen, the installed models were benchmarked. Two properties were of interest: throughput and output quality.

Throughput was measured on the primary desktop with a 500-word essay prompt (ollama run <model> --verbose):

Model	Tokens/sec	Duration	Tokens out
gemma4:26b	18.86	50.11s	945
gemma4-26b (64K ctx)	17.96	51.99s	934
mistral:7b-instruct	34.81	10.17s	354
llama3.2	57.11	3.99s	228

The smaller models are substantially faster; their token counts, however, are lower, and in practice their responses were correspondingly shallower.

Quality was assessed with a five-task suite spanning logic, coding, summarization, creative writing, and instruction-following:

Model	Score	Note
gemma4-26b (64K)	50/50	Flawless instruction-following
glm-4.7-flash	42/50	Missed only a complex string-formatting task
qwen3-coder:30b	40/50	Overthinks logic; hallucinated a fake country
llama3.2	28/50	Failed logic entirely
mistral:7b	26/50	Failed logic and negative constraints

A harder ten-task variant — incorporating a lipogram, a theory-of-mind question, and a riddle — was subsequently administered, on which Gemma 4 26B scored 99/100 while sustaining approximately 17 tokens/sec. This represents the local optimum: strong quality at a workable speed.

The principle that follows is that throughput and quality trade off against one another, and the fastest available model is rarely the appropriate default for substantive work. Gemma 4 26B is slower than the 7B models yet markedly more accurate, and it was therefore adopted as the primary model.

What follows

With a runner, a model, and a hardware baseline established, subsequent entries turn to the extraction of useful work from the setup. We intend to address context length and KV-cache trade-offs; the distinction between prefill and generation, and why a machine without a GPU can sustain conversation yet falter on large prompts; and the practice of keeping models resident to avoid cold-start reload penalties.

The measured figures are reported as-is, dead ends included. Part 2 will follow.

LLMs on Consumer Hardware — Part 2: Prefill and the Failure of the AI PC

Sven Welack — Tue, 04 Aug 2026 03:54:02 +0000

Part 1 established the hardware, the runner, and the primary model. This entry covers what governs inference on that hardware — the two phases of inference, the cost of long context, and the cost of loading a model from disk — and compares the local machines against a free-tier cloud model.

The two phases of inference

Inference has two phases. Prefill processes the input prompt before any output appears; it is compute-bound and wants a GPU. Generation produces output tokens one at a time and is bound by memory bandwidth. Casual use is almost all generation and hides the difference; the cost of prefill surfaces only when prompts grow large.

The machines, and how they were measured

Machine	CPU / RAM	GPU (VRAM)	Storage (read)	Prefill (tok/s)	Gen (tok/s)	Load (18 GB)
Primary desktop	5950X / ~80 GB DDR4	RX 6900XT (16 GB)	NVMe (~2.1 GB/s)	360	18.3	8.4s
Secondary box	5600G / 32 GB DDR4	GTX 1060 (6 GB)	SATA SSD (~0.35 GB/s)	253	17.1	50.6s
Laptop	8840U / 32 GB DDR5	Radeon 780M (none)	NVMe (~2.4 GB/s)	20	10.0	7.5s

All inference figures come from a controlled run: the same model (Gemma 4 26B, 18 GB) on each machine, a unique random prefix per prompt to defeat caching, a fixed 8,192-token context, warm, on an identical ~6,855-token prompt (generation timed over a 200-token output).

Two things stand out. Prefill varies about eighteen-fold across the machines (360 to 20 tok/s) while generation varies less than twofold (18.3 to 10.0), and prefill is what dominates large-prompt workloads — so a machine can look fine on generation yet be useless in practice. Model-load time, separately, is set by storage rather than compute: the secondary box's budget SATA SSDs load the 18 GB model in 50 seconds against eight on NVMe, which turns a cold request into a minute-long stall.

Secondary box	Request time
Warm (model resident)	~4s
Cold (model reload)	~54s

If the model is allowed to unload between calls, every call silently pays that reload — a real source of intermittent timeouts. The fix is a long keep-alive (OLLAMA_KEEP_ALIVE=24h) with pre-warming; on a slow-disk node it is a precondition, not a refinement.

An "AI PC" that could converse but could not serve

The laptop deserves particular attention, because it is sold as an "AI PC," and that framing is precisely what it fails to honour. The 8840U (AMD's 8040 "Hawk Point" series) carries a dedicated XDNA NPU rated at up to 16 TOPS — around 38 across the platform — and is marketed under the "Ryzen AI" banner for exactly this sort of local inference. Yet the NPU is built for low-power, always-on tasks such as webcam background effects and noise suppression, and the LLM runner does not address it at all. Large-model inference therefore falls to the CPU, which prefills at only ~20 tok/s (the table above), so a system prompt of ten to fifteen thousand tokens needs eight to twelve minutes to ingest before a single token is produced — the thirteen-minute stall observed in practice.

The lesson cuts against the marketing twice over. "AI PC" denotes a narrow class of accelerated workloads that excludes running a multi-billion-parameter model against a large prompt; the advertised TOPS are, for this purpose, inert, and the figure that decided the outcome was an unglamorous CPU prefill rate. The same NPU also sits below the 40-TOPS threshold Microsoft attaches to the AI-PC label.

The cost of context

Long context is paid for in memory, because the KV cache grows linearly with context length. The runner defaults to a 4K–8K window; this was raised to 64K through a custom Modelfile (num_ctx 65536).

Context (q8_0 KV cache)	KV cache size	Verdict
64K	~926 MiB GPU + 231 MiB CPU	stable — adopted
128K	larger; slower prefill, unstable	rejected

The 64K window was kept as the stable operating point. (Forcing all layers onto the 16 GB card with num_gpu 99 fails outright; rely on the runner's automatic GPU/CPU split instead.)

Comparison with a cloud model

How do the local machines compare with a hosted model? On the same ~150-word reasoning prompt, a free-tier cloud model (Gemini 3 Flash) was timed end-to-end against the two local GPUs.

Option	End-to-end latency	Output
Cloud — Gemini 3 Flash (free tier)	~5.8s	205 tokens, after ~536 internal reasoning tokens
Primary desktop — RX 6900XT	~20.6s	200 tokens
Secondary box — GTX 1060	~64.5s	200 tokens (inflated by a cold reload)

This is not a pure-compute comparison — the cloud figure includes the network round-trip and Google's serving infrastructure, and the API exposes no prefill/generation split — but it measures the quantity that matters in use: how quickly an answer arrives. The cloud model won comfortably while doing more work, spending ~536 internal reasoning tokens before its 205-token answer. The lesson is not that cloud beats local, but that placement should follow the task: routine, high-volume work belongs local — private, unmetered, and predictable in latency.

What follows

These findings — prefill needs a GPU, context costs memory linearly, a cold model is far dearer than a resident one, and placement should follow the task — compound once more than one model must be held in memory at once. Whether several models can coexist without contention is the subject of the next entry.

LLMs on Consumer Hardware — Part 3: A Multi-Agent Setup with Hermes — Delegation and Early Failure Modes

Sven Welack — Fri, 31 Jul 2026 09:32:51 +0000

Part 3 of a series. Previously: Part 1 — The Stack and First Benchmarks and Part 2 — Prefill, Loading, and a Cloud Comparison.

The first two entries concerned the substrate: which model, on which hardware, at what speed. This entry concerns what was built on top of it. The project is not a single model answering prompts but a small team of specialised agents, each with its own role, its own memory, and — the point of interest — potentially its own model. What follows describes that team as it was first assembled, and what happened when it was first put to work; the roster grew and was reshaped later, but this was the starting cast.

The model and the machine

Everything in this entry ran on a single model, Gemma 4 26B, served by Ollama on the "large" desktop from Parts 1 and 2 — a Ryzen 9 5950X with about 80 GB of DDR4 and a 16 GB Radeon RX 6900XT (on ROCm), which runs the model at roughly 18 tokens per second. Gemma 4 26B is a mixture-of-experts model (around 3.8B active parameters) quantised to Q4_K_M at about 18 GB on disk; it was chosen in Part 1 because it gave the best quality of the local models tested — topping the quality suites where smaller, faster models proved markedly shallower. In this first iteration every agent, coordinator and workers alike, ran that same model on that same machine.

Hermes

The agent framework is Hermes, an open-source system from Nous Research that connects to local model runners such as Ollama. Its relevant unit is the profile: a self-contained agent defined by a SOUL.md file, which specifies its role and instructions, together with its own memory and its own model assignment. Profiles are independent — an agent knows only its own SOUL.md and the specific task it is handed; it does not share context with the others, or even know they exist, unless it is told. They are, in effect, stateless islands, a property that governs a great deal of what came later.

Delegation is performed explicitly. One agent invokes another by name through the terminal, passing a task and capturing the reply; there is no shared blackboard, only message-passing.

The first team

The agents were cast as a small company, each responsible for one kind of work. In this first iteration the team was deliberately small.

Agent	Role	Responsibility
Gatsby	Orchestrator	Delegates and synthesises; performs no work directly
Alex	Researcher	Research and analysis
Maya	Coder	Writes and tests code
Nova	Sysadmin	Installs, configures, and deploys

The principle behind this casting is the one carried through Part 1's benchmarking: match the model to the job, not to a leaderboard. Because each Hermes profile can be assigned its own model, a role that writes code and a role that only researches need not run the same one — an option not yet exploited here, where the ambition still ran ahead of the practice.

Why a team, and why local

The decision to build a team rather than lean on a single capable model was taken by analogy with human organisations. People do not scale by asking one individual to do everything; they specialise, divide the labour, and coordinate the parts. The same reasoning was applied here — a role given one narrow kind of work, with instructions tuned to it, was expected to prove more reliable than a single generalist asked to research, code, administer, and check its own output within one long context.

A second benefit follows from this: if no single model must know everything, each can be narrower and lighter, which makes the whole arrangement more tractable on constrained local hardware than one monolithic model attempting the same breadth. Specialisation was adopted for reliability; the efficiency, and the better fit to local machines, came as a dividend.

Running locally was driven by cost. Once the hardware is owned, a good consumer machine generates tokens cheaply — with no per-call fees and no rate limits across long unattended runs — so the operating principle became to run locally whatever can reasonably be run there, and to reserve the cloud for the few workloads that genuinely warrant a frontier model.

The first runs, and the first problems

With the profiles in place, the team was put to work — first on trivial delegation tests, then on genuinely multi-step work: a deployment exercise on Google Cloud, and a deliberately demanding drill of some seventeen tasks intended to stress the coordination itself. Standing the agents up, it emerged, had been the easy part. Three problems surfaced almost at once.

Delegations that silently did nothing. A hand-off would "complete" in about five seconds and return empty, or an agent would describe the command it was meant to run rather than run it. The cause was a single wrong flag: the SOUL.md files invoked other agents with -m, which Hermes reads as --model, not as the message.

# Broken — what the SOUL.md files told agents to run:
hermes --profile maya-coder chat -m "write a hello-world script"
#   -m is --model, so Hermes tried to load a model NAMED
#   "write a hello-world script"  ->  HTTP 400: invalid model name
#   (fails in ~5s, which is why the delegation looked "done")

# Fixed — -q/--query carries the task; -Q/--quiet gives clean, capturable output:
hermes --profile maya-coder chat -q "write a hello-world script" -Q
#   now actually runs the model: a real reply in ~44-72s

The five-second "success" was the model runner rejecting an invalid name, not an agent finishing its work — which is exactly why the failure was so easy to misread.

A coordinator that lost the thread. Running Gemma 4 26B, Gatsby handled only one or two sequential hand-offs reliably; beyond that he dropped steps — contacting two agents and forgetting a third — or narrated the plan instead of executing it. The seventeen-task drill collapsed him entirely: he laid out all seventeen tasks and then lost the thread. There was no clean fix at this stage, because the model was simply miscast as an orchestrator; but a deterministic nudge reliably un-stuck him as a stopgap:

# Injected when Gatsby stalls mid-plan:
You have contacted Alex and Maya but not Nova. Delegate to Nova now.

That the coordinator needed prompting to finish its own plan pointed at a deeper problem — the best worker model was not the right orchestrator — which later entries take up.

A fabricated success report. During the Google Cloud task, when hand-offs to a worker timed out, Gatsby quietly completed the work himself and then filed a confident final report claiming the whole team had collaborated and the result had been verified — none of which was true. The immediate remedy was to make honesty explicit in his SOUL.md:

# Added to Gatsby's SOUL.md, to this effect:
- Never fabricate results or reports; if you did not verify it, say so.
- Slow is not failed: a delegation still running has NOT failed — wait
  for it, do not silently take over the work.
- Report real errors honestly, timeouts included.

A small model under pressure will manufacture success rather than admit failure; the fix is to forbid it in the instructions and to make "still running" an acceptable state to report.

What follows

None of these was fatal, but together they reframed the project. The difficulty was never getting agents to exist; it was getting them to coordinate — to hand work off correctly, to hold a plan across more than a couple of steps, and to report honestly when something went wrong. The flag was a quick fix once found; the coordinator's ceiling was not, and the search for an orchestrator that could actually hold a plan together runs through the entries that follow.