DEV Community: Sven Welack

LLMs Locally on Consumer Hardware — Part 10: How the Setup Evolved — A Timeline of What Changed and Why

Sven Welack — Wed, 11 Nov 2026 07:15:58 +0000

Part 10 of a series — a retrospective appendix. The narrative ran across Parts 1–9; this one lays the evolution out stage by stage.

No part of the setup was chosen for its own sake. Every change — a new machine, a different coordinator, a smaller model — was forced by a specific failure in the stage before it. Below is each stage as a snapshot, with the arrow between snapshots showing what broke and what was added or removed to fix it.

The timeline, stage by stage

Stage 0 — Initial

Machine	What it runs
Main — 5950X · ~80 GB · RX 6900XT	Coordinator and all workers — everything on `gemma4:26b`, local

⬇ Broke: gemma4:26b can't hold a coordination loop beyond ~1–2 delegations.
Added: a search for a dedicated coordinator model (workers stay on gemma).

Stage 1 — Coordinator search (still one machine)

Machine	What it runs
Main — ~80 GB · RX 6900XT	Coordinator: trialing `gpt-oss:120b` (and `groq-tool-use:8b`, which fabricated sessions) · Workers: `gemma4:26b`

⬇ Broke: the 120B (~77 GB) and the gemma workers (~20 GB) can't both be resident in 80 GB RAM → ~100 GB swapped per hand-off; 291 s to say hello.
Removed: the workers from the main machine · Added: a second machine (5600G + GTX 1060).

Stage 2 — Distributed (two machines)

Machine	What it runs
Main — ~80 GB · RX 6900XT	Coordinator: `gpt-oss:120b` (resident; runs on CPU + RAM)
Worker (.114) — 32 GB · GTX 1060	Workers: `gemma4:26b`

⬇ Noticed: with the 120B running on CPU and RAM, the main machine's GPU sits idle.
Added: image generation (ComfyUI + SDXL) on the main 6900XT.

Stage 3 — Image generation added

Machine	What it runs
Main — ~80 GB · RX 6900XT	Coordinator `gpt-oss:120b` (~77 GB RAM) + SDXL image gen (~15 GB VRAM)
Worker (.114)	`gemma4:26b` workers

⬇ Broke: the 120B leaves only ~3 GB RAM free → SDXL generations time out.
Removed: gpt-oss:120b · Added: gpt-oss:20b (~14 GB) in its place.

Stage 4 — Right-sized coordinator

Machine	What it runs
Main — ~80 GB · RX 6900XT	Coordinator `gpt-oss:20b` (~14 GB RAM) + SDXL (~15 GB VRAM) — ~60 GB RAM free
Worker (.114)	`gemma4:26b` workers

⬇ Added over the same period: a cloud strategist (Sage, on Gemini 3 Flash) for the one role needing deep reasoning; the laptop as an SD 1.5 image node; and deterministic orchestration (a script owns the delegation loop, the model only plans and evaluates).

Stage 5 — Final

Where	What it runs
Main — ~80 GB · RX 6900XT	Coordinator `gpt-oss:20b` + SDXL image generation
Worker (.114) — 32 GB · GTX 1060	`gemma4:26b` specialist workers
Laptop (.247) — 32 GB DDR5 · 780M	SD 1.5 image node (512 px)
Cloud	Sage strategist — Gemini 3 Flash (free tier), consulted when blocked

The laptop's detour

The 8840U laptop ran its own small arc inside the larger one.

Attempt	Result
Laptop 8840U / 780M as an LLM worker	CPU prefill ~25 tok/s → 13-minute stalls

⬇ Why: no discrete GPU, so prefill is hopeless for large agent prompts (Part 2).
Repurposed for the one job the hardware could do:

Repurposed	Result
Laptop as an image node	SDXL hangs on the 780M; SD 1.5 @ 512 px works (~37–42 s)

Initial versus final

	Initial setup	Final setup
Machines	1 (main)	3 (main, worker, laptop) + 1 cloud model
Coordinator	`gemma4:26b`, local	`gpt-oss:20b`, local
Workers	`gemma4:26b`, same machine as the coordinator	`gemma4:26b`, dedicated worker machine (.114)
Strategist	—	Gemini 3 Flash, cloud (consulted only when blocked)
Image generation	—	SDXL on the main 6900XT; SD 1.5 on the laptop node (.247)
Orchestration	model-driven loop	deterministic script owns the loop; model only plans and evaluates

The pattern

Read top to bottom, the chain has a shape. Nearly every change was one of two moves: placement — which machine a thing runs on — or right-sizing — how large each model needs to be for its job. The workers moved for placement; the coordinator shrank for right-sizing; the strategist went to the cloud because that was the right placement for the one role that needed deep reasoning. Not once did the answer turn out to be "buy a bigger model and hope." The capability was almost always already there in the hardware on hand; the work was in arranging it so each piece had room to do the thing it was good at.

Running LLMs Locally on Consumer Hardware — Part 9: Image Generation on AMD, and a Creative Capstone

Sven Welack — Tue, 03 Nov 2026 14:09:38 +0000

Part 9 of a series. Previously: Part 8 — The QA Problem.

The team could research, write code, administer a machine, verify its own work, and adapt when blocked. One thing it could not do was make a picture. This final entry is about adding that last capability — which on AMD hardware is less trivial than it sounds — and about the capstone project that used it: an attempt by the team to write and illustrate its own animated story.

The setup, at a glance

Before the procedure, the shape of it — three machines, each given the job its hardware actually suits.

Machine	Hardware	What it runs	Memory used	Headroom
Main ("gatsby")	Ryzen 5950X · ~80 GB RAM · RX 6900XT, 16 GB VRAM	Coordinator (`gpt-oss:20b`) + SDXL image generation	coordinator ~14 GB RAM (GPU idle); SDXL ~15 GB VRAM at peak	~60 GB RAM free
Worker (.114)	Ryzen 5600G · 32 GB RAM · GTX 1060, 6 GB VRAM	Specialist workers (Gemma 4 26B)	~18 GB model, spilling across 6 GB VRAM + system RAM	~14 GB RAM free
Laptop (.247)	Ryzen 8840U · 32 GB DDR5 · Radeon 780M (iGPU, no dGPU)	SD 1.5 image node, 512 px	light; the iGPU shares system RAM	ample — but SDXL will not run here

Image generation runs on the main machine's 6900XT, and the card is free to host it because the coordinator barely touches the GPU: too large for 16 GB of VRAM, it runs on the CPU and system RAM and leaves the card idle during inference. The two tenants therefore contend for different resources — the coordinator for RAM, SDXL for VRAM — so the real constraint is how much RAM the coordinator leaves behind:

Main machine (~80 GB RAM, 16 GB VRAM)	RAM used	VRAM used	RAM free	Image generation
Coordinator `gpt-oss:120b` + SDXL	~77 GB	~15 GB	~3 GB	times out
Coordinator `gpt-oss:20b` + SDXL	~14 GB	~15 GB	~60 GB	runs

The VRAM column is identical in both rows — the GPU was never the bottleneck. Right-sizing the coordinator from the 120B down to the 20B (Part 5) is what opened the ~60 GB of headroom that lets generation and coordination run at once. (SDXL's ~15 GB is the peak during a generation; ComfyUI's /free drops it back to about 2.2 GB when idle.)

Getting ComfyUI onto an AMD card

The plan was ComfyUI with Stable Diffusion on the main machine's now-idle 6900XT. Getting there was a ROCm fight of the familiar kind. ComfyUI's backend needs torch.library.custom_op, which arrived in PyTorch 2.4; the ROCm wheel that installed first was older, so the first launch failed immediately:

AttributeError: module 'torch.library' has no attribute 'custom_op'

The fix was to install a newer ROCm build (torch 2.5.1+rocm6.2) explicitly, and to launch with HSA_OVERRIDE_GFX_VERSION=10.3.0 — the same RDNA2 override Ollama needed back in Part 1. With that, ComfyUI saw the card natively: Device: cuda:0 AMD Radeon RX 6900 XT : native, 16 GB of VRAM.

The next failure was more instructive than it looked. The first model tried needed separate diffusion, CLIP, and VAE files, and the VAE folder was empty — so the pipeline produced a raw 16-channel latent and handed it, undecoded, to the step that saves a PNG:

TypeError: Cannot handle this data type: (1, 1, 16)

That shape is a tell: a 16-channel "image" is an undecoded latent, which means the VAE is missing or unconnected. Rather than chase the multi-file setup, the project switched to SDXL base — a single checkpoint with the VAE baked in, one node to load it, and a well-trodden default workflow. It generated a clean 1024×1024 image in roughly 15 to 30 seconds. Reliability beat novelty, which has been the recurring verdict throughout.

A sharp capability envelope

The laptop from Part 2 — the "AI PC" that failed as an LLM worker — was tried as a second image node, and here the story turns almost redemptive: a machine useless for agents might still be useful for something. It was, but within strict limits. SDXL hung the 780M hard, every time, at the first sampling step:

UserWarning: Attempting to use hipBLASLt on an unsupported architecture!
HW Exception by GPU node-1 ... reason :GPU Hang
Fatal Python error: Aborted

No override or attention flag got past it; AMD's own math libraries simply do not fully support that integrated GPU. But dropped down to SD 1.5 at 512×512, the same machine worked, stably, at around 37–42 seconds an image. The envelope was sharp and worth recording: SDXL hangs on the 780M; SD 1.5 at 512px runs fine. The card that could not run an agent had found the one image job it could do — which is the whole "match the workload to the machine" theme of this series, stated one more time.

The image generator is a tool, not an agent

One design decision is worth calling out because it runs against the grain of everything else here. Every capability so far had been embodied as an agent — a profile with a role. Image generation was not. It was built as generate-image.py, a plain script that posts an SDXL workflow to ComfyUI's API, polls for completion, and saves the PNG. Any agent that needs a picture calls it the way it would call any command-line tool; it was simply added to the relevant workers' instructions so they knew it existed.

The point is that not everything should be an agent. A deterministic, single-purpose capability — like a control loop in Part 6 — belongs in code, invoked as a tool, not dressed up as a teammate with a personality. There was no "artist agent," and the system was simpler for it.

The capstone: an illustrated story

With pictures finally possible, the team was pointed at its most ambitious task: write and illustrate a fifty-scene animated story, then assemble it into a website. It was a genuine stress test, and it exercised nearly every lesson in the series at once.

The seams showed immediately. Asked to write all fifty scenes in a single delegation, the researcher timed out repeatedly and, in the end, delivered around twenty full scenes plus an outline for the rest — a direct replay of the "few rich delegations, not one enormous one" lesson. Generating fifty images in one call was likewise impossible, so the team did something genuinely resourceful: it wrote its own batch script and ground through the images in chunks, retrying the ranges that timed out.

START_INDEX=0  END_INDEX=10  python3 generate_images_batch.py   540.7s
START_INDEX=10 END_INDEX=50  python3 generate_images_batch.py   900.3s [error]
START_INDEX=25 END_INDEX=35  python3 generate_images_batch.py   900.6s [error]
START_INDEX=25 END_INDEX=30  python3 generate_images_batch.py   328.6s

There were timeouts and a few mistyped output paths along the way, but the chunking worked: all fifty images were produced. The final assembly was where it came undone — the page meant to tie the story and images together was generated hollow, a title and some styling with none of the content, which is precisely the failure Hans caught and correctly refused to pass in Part 8. The capstone, honestly, was a partial success: the parts were all there and real, and the thing that joined them was not yet right.

The series, in one sentence

That partial result is a fair summary of the whole project. A team of small models, running on consumer hardware that mostly already existed, can do real multi-step work — research, code, system administration, verification, and now illustration — imperfectly but genuinely, at almost no running cost. Getting there was not a matter of finding one clever model. It was a long accumulation of unglamorous lessons, nearly all of which showed up in this last project at once: match the model to the job; respect the memory budget; put deterministic work in code, not in a model; keep the slow machine in the role it is actually good at; make the checker test what matters; and, above all, build a system that tells you the truth about what it did. None of it required a frontier lab's resources. Most of it required only paying attention to what broke, and why.

Thanks for reading. The code, the demos, and the data behind the series live in the companion repository.

LLMs on Consumer Hardware — Part 8: The QA Problem — Why 'It Ran' Isn't 'It Works'

Sven Welack — Sat, 31 Oct 2026 02:37:12 +0000

Part 8 of a series. Previously: Part 7 — The Strategist.

By this point the team could build things and, when blocked, adapt. What it could not reliably do was tell whether what it had built actually worked. That job belonged to Hans, the QA agent — the member of the team whose entire role is to run the others' deliverables and verify them — and for a long stretch Hans was the weakest link in the whole system. The reason is worth dwelling on, because it generalises far beyond this project: the gap between "it ran" and "it works."

The game that broke on the first shot

The clearest case came from a game. Asked to build a tower-defense game, the team produced something genuinely creditable: a self-contained HTML canvas game with its graphics generated in code, a sensible class architecture — Enemy, Tower, Projectile — and a working game loop. It looked finished. It was not. Buried in it was a single fatal mistake: the towers fired by pushing plain objects into the projectile list, while the Projectile class that was supposed to represent them was defined and then never used.

// a Tower fires by pushing a plain object:
projectiles.push({ x, y, target, ttl });

// but the game loop expects Projectile instances:
class Projectile { update() { /* move, check for a hit */ } }
for (const p of projectiles) p.update();   // TypeError: p.update is not a function

The game loaded perfectly. It drew the map, the towers, the enemies. Then, the instant the first tower took its first shot, p.update() was called on a plain object that had no such method, and the game threw. The bug was invisible until the first interaction — and unmissable immediately after.

Three passes at QA

Hans was asked to verify it, and the first verdict was PASS. He had not run it at all.

Strengthening his instructions was the obvious fix, and it was tried: an iron rule that nothing may be marked PASS that has not actually been executed; a third verdict, CANNOT VERIFY, so that "I couldn't test this" stopped quietly collapsing into "PASS"; an explicit instruction to run JavaScript through node -e; and a direction to trace the first user action specifically.

Hans genuinely tried harder on the next pass. He launched headless Chromium, loaded the page, took a screenshot, and checked the console for errors. There were none — and he returned PASS again. The screenshot had been taken at load. Nothing in it had placed a tower, so nothing had fired a shot, so the first-shot bug had never had the chance to appear.

The lesson: load is not play

This is the heart of it: verifying that something loads is not verifying that it works. A page that renders without console errors has proven only that it renders. The bug lived one interaction deeper, and every cheap form of testing — a screenshot, a load check — stops short of that interaction. Left to find its own way to "test," the agent reliably found the easiest thing that superficially counted as testing, and stopped there.

The instinct, familiar by now, was to wonder whether a more capable model would do better. It would not. Handed the same game, the 120B coordinator reached the same verdict from the same load-time screenshot. Testing the right thing is a question of method, not of model size — the same shape of lesson as the orchestration ceiling in Part 6. The fix lives in the instructions: say place a tower and fire a shot, not check that it loads.

The same failure in disguise: mechanics versus usability

The mistake wore a second disguise on a different deliverable. The team built a small command-line task tool, and it worked. Hans verified it by running a command with the task's full identifier, which succeeded. But the identifiers were raw UUIDs, and no human would ever type one:

$ python3 taskflow.py done 1
Task ID 1 not found.        # the real ID is da258ee6-66ca-4f1a-a27e-9169b15739bf

The tool functioned; it was simply unusable. Hans had verified the mechanics — does the command work when handed a valid ID? — and not the usability — can a person actually use this? It is the same error as the screenshot: testing the easy, literal thing instead of the thing that matters.

The method takes

It did, eventually, take. On a later project — a generated website meant to present a long illustrated story — Hans caught a real and subtle failure and reported it correctly:

Verification Result – CANNOT VERIFY
- story.md – present and complete.
- All 50 scene images (scene1.png … scene50.png) – present and each > 5 KB.
- index.html – generated but contains only a title and CSS; it does not include
  the story text or any <img> tags referencing the scene images.

Every file was present; every image was on disk and non-empty. A file-existence check, or a load check, would have passed it. But the page that was supposed to tie them together was hollow — a title and some styling, and none of the actual content. Hans noticed, declined to call it PASS, and returned CANNOT VERIFY with the specific reason. The third verdict had done exactly what it was added to do.

The environment lies too

Verification was not the only place a cheap signal misled; the same autonomous runs turned up failures of the environment itself. Files written to relative or ~-based paths "vanished," because in a multi-process setup each agent's working directory was its own profile folder — so a path meant for /home/user/hermes-runs/ landed somewhere like /home/user/.hermes/profiles/gatsby-manager/home/hermes-runs/ instead. The fix was to insist on absolute output paths everywhere. On another run a launcher failed instantly because the script its configuration named had never been copied to that path. In each case the lesson rhymes with the QA one: a green light — a command that returned, a file that was "written" — is not the same as the thing being where and what it should be.

The throughline

The lesson of QA is almost banal, and almost universally ignored: test the thing that matters, not the thing that is easy to test. Play, not load. Usability, not mechanics. The content, not the file list. None of it required a better model; all of it required telling the checker, precisely, what "working" means.

What follows

The hollow story-site that Hans flagged belonged to the team's most ambitious undertaking: an attempt to write and illustrate its own animated story — which first meant teaching a team that only knew text and code how to make pictures at all. That, and the AMD-specific fight to get image generation running locally, is the final entry.

Running LLMs on Consumer Hardware — Part 7: The Strategist — Judgment on a Cloud Model, Execution Local

Sven Welack — Mon, 19 Oct 2026 10:32:39 +0000

Part 7 of a series. Previously: Part 6 — The Orchestration Ceiling.

Part 6 made the team's execution reliable: by moving the control loop out of the model and into deterministic code, the orchestrator could carry out a plan without dropping steps. But reliable execution answers only half of the problem. A plan carried out faithfully is of little use once it meets a wall the plan did not anticipate — and for that the team needed a capability that execution reliability could not supply: judgment. Judgment was the one thing served poorly by any local model, and it is the single part of the system that runs in the cloud. This entry explains why, and what it bought.

The stall-versus-improvise problem

Earlier entries established Gatsby, the orchestrator, as "the honest operator": reliable, never fabricating a result, never improvising wildly. That discipline is exactly what an orchestrator needs, but it created a tension at the team's edges. When the team hit a genuine blocker — a download that failed, a dependency that was absent — an honest, non-improvising operator has only two available moves, and both are wrong. It can stall: report the obstacle and stop. Or it can improvise: depart from the plan, and risk exactly the kind of unfounded, made-up progress the honesty rules existed to prevent.

What was missing was a third option — adaptation: a way to rethink the approach without either giving up or inventing. The problem was that reliability and creative rethinking are, to some extent, opposing traits, and one agent had been asked to supply both.

Separating judgment from execution

The resolution was to stop asking a single agent to be both dependable and inventive. Execution — the honest, faithful carrying-out of a plan — stays with Gatsby. Judgment — the creative rethinking that a wall calls for — moves to a new role: Sage, the strategist. Gatsby no longer improvises; when he is blocked, he consults Sage, and Sage does the adapting. Judgment lives in Sage; honest execution lives in Gatsby. The escalation was made explicit with a single rule added to Gatsby's instructions — when blocked, consult Sage — so that hitting a wall triggers a consultation rather than a stall or a guess.

Why this role — and only this role — is in the cloud

The operating principle set out earlier in the series was to run locally whatever reasonably can be, and to reserve the cloud for the few workloads that genuinely warrant a frontier model. Strategy is that workload, for two reasons.

First, it is invoked rarely — only when the team is actually stuck — so its cost is negligible. A role called a handful of times across a long run is precisely the kind of workload a free cloud tier serves without strain, and it consumes no local resources at all. Second, it is the one role whose quality turns on depth of reasoning rather than on throughput, and depth is where a frontier model sits in a different class.

That difference is measurable. Asked what makes a strategy good, the cloud model spent 413 internal "thinking" tokens before producing a 24-token answer — it reasoned for roughly seventeen times the length of what it ultimately said. That ratio is the entire point of the role: a strategist should think far more than it speaks. The local models used elsewhere in this system do not reason internally in this way; the capability is specific to the frontier model, and it is exactly what the role requires.

The free-tier reality

The specifics mattered, because the free tier had narrowed by mid-2026. Gemini's Pro models left it on 1 April 2026, leaving only the Flash and Flash-Lite tiers free. The role therefore landed on gemini-3-flash-preview: free, with a one-million-token context window — large enough that the context limits that shaped the local models are simply irrelevant here — and configurable "thinking" levels. A frontier-class strategist, consulted occasionally, at no cost.

Sage in practice

The value showed on a real blocker. The team was building a game and could not download the graphics it wanted; the network was blocked, and the workers treated this as a hard stop. Sage reframed it. The team, it observed, was conflating graphics with external downloads — and the two are not the same thing. It recommended procedural synthesis: generating the graphics in code rather than fetching them, with concrete next steps for each member of the team.

The detail that marked this as judgment rather than a lucky guess was anticipatory. Sage also directed the team to audit the project for other hidden external dependencies — before they hit the same wall somewhere else. That is the difference a reasoning model makes to the role: it did not merely solve the blocker in front of it, but recognised the class of problem behind it and moved to pre-empt the next instance.

A second case showed the same judgment turned the other way — toward stopping. Tasked with installing a set of system packages, Nova, the sysadmin, hit a wall that no amount of reframing could get past: the installation required a sudo password, which the agents did not have and should not have. Here the escalation path proved its worth in the opposite direction. Rather than inventing a workaround or declaring false success, the team routed the blocker up through Sage, and the judgment that came back was to surface it to the human. Credentials are exactly the kind of thing an autonomous system should stop and ask for, and the design held end to end: no fabrication, no stalling in place, just an honest hand-off to the one party who could supply what was missing.

What follows

With judgment and execution separated, the team could at last both stay honest and adapt. But a strategist that reframes a problem well does nothing to guarantee that the work it unblocks is actually verified — and verification turned out to be the team's weakest link. The next entry is about quality assurance, and about why "it ran" and "it works" are not the same claim.

LLMs on Consumer Hardware — Part 6: The Orchestration Ceiling — Why a Bigger Coordinator Doesn't Fix the Loop

Sven Welack — Sun, 18 Oct 2026 05:03:54 +0000

Part 6 of a series. Previously: Part 5 — Distributing the Workers and the Memory Budget.

Part 5 ended with the 77 GB coordinator finally running as it should — resident, no thrashing, the memory problem resolved by moving the workers to a second machine. That made it possible to answer a question Part 4 had deferred: now that the big model can actually run, does a bigger coordinator fix orchestration? The answer was no, and the reason reshaped how the team was run.

The ceiling persists on the big model

With memory no longer the bottleneck, the coordinator's orchestration could be judged on its own merits. It sequenced delegations more reliably than Gemma had — but it did not escape the same failure modes. On longer plans it still dropped steps, issuing some delegations and silently omitting others before writing its summary. It still, on occasion, emitted a delegation as text rather than executing it — Part 4 showed one such case, a hand-off printed as raw JSON and then abandoned. And it periodically returned nothing at all after a tool call, requiring an automatic nudge to continue:

⚠ Model returned empty after tool calls — nudging to continue

That warning recurred often enough to be diagnostic. These were not the mistakes of a weak model; they were the mistakes of a capable model asked to do something it is structurally poor at — hold a long control loop in its head while also doing the reasoning inside it.

The diagnosis: the ceiling is the loop, not the model

The decisive observation was that all three failure modes — dropped steps, commands narrated instead of executed, empty turns — appeared on the large model exactly as they had on the small one. The common factor was not model size. It was the agentic loop itself.

Asking a language model to orchestrate is asking it to be, simultaneously, the planner that decides the steps, the dispatcher that issues each delegation, the bookkeeper that tracks which delegations are still outstanding, and the evaluator that judges the results as they return. That is a great deal of state to carry across many turns, and carrying state across turns is precisely what these models are weakest at. A bigger model raises the ceiling a little; it does not remove it.

The fix: take the loop away from the model

If the loop is the problem, the loop should not be the model's job. The resolution was deterministic orchestration: a small script owns the control loop. It holds the list of tasks, dispatches each delegation, waits for and collects the result, and decides what runs next. The model is invoked only for the two things it is genuinely good at — planning (what are the steps?) and evaluating (is this result acceptable?) — and never has to remember what it has already dispatched, because the script does that.

The improvement was immediate and measurable. A task sequence that the model-driven loop had stalled on partway through — completing three of five steps before losing the thread — ran all five to completion when the same plan was executed by the script. The model had not become more capable. It had simply been relieved of the one job it could not reliably do.

The lesson

The instinct, when an agent loop is unreliable, is to reach for a bigger model. That instinct was wrong here, and expensively so — it was the reasoning behind the whole coordinator search of Part 4, and the memory wall of Part 5, and in the end it did not solve the problem it was chasing. Orchestration reliability is not a model capability that can be bought in larger quantities; it is a property of how the loop is structured. The durable fix was architectural: move the deterministic part — the loop, the bookkeeping, the dispatch — into deterministic code, and reserve the model for the parts that genuinely require judgment. A control loop is a job for a program, not for a language model, and the sooner that division is drawn the more reliable the system becomes.

What follows

Deterministic orchestration made the team's execution reliable: a plan was now carried out without dropped steps or invented progress. But it did nothing for the moment a plan meets a wall — when the right next step is not in the plan at all, and someone has to rethink the approach rather than merely execute it. Reliable execution and good judgment are different capabilities, and the next entry is about supplying the second without compromising the first.

LLMs on Consumer Hardware — Part 5: Distributing the Workers and the Memory Budget

Sven Welack — Wed, 14 Oct 2026 07:40:39 +0000

Part 4 ended against a wall. A coordinator capable enough to be worth running — the 77 GB gpt-oss:120b — could not share a machine with the workers it had to call; every delegation forced roughly 100 GB of model-swapping, and a greeting took nearly five minutes. The conclusion was that a coordinator and its workers must fit in memory together. This entry takes that constraint at its word and does the obvious thing: it stops making them share.

Moving the workers off the machine

The fix was to put the workers on a different machine. The second box introduced in Part 2 — the 5600G with the GTX 1060 — became the worker host; this is how it got that job. With the workers resident there, the main machine's memory was no longer contested: the coordinator had it to itself, and nothing local evicted it between hand-offs.

The effect on the 120B was immediate. The same model that had thrashed itself into uselessness now loaded once and stayed resident for 1 hour 46 minutes across a long run, with no swapping at all. Freed from the swap penalty, its orchestration was also more reliable than Gemma's had been. The lesson was almost deflating in its simplicity: the coordinator had never been the problem. Its placement had.

The second machine thereby earned its keep twice. Part 2 put it there because its discrete GPU fixed prefill — a GPU-less box could not ingest the large prompts in any reasonable time. Moving the workers onto it solved a second, unrelated problem: it freed the main machine's RAM for the coordinator. One hardware decision, two constraints resolved.

One quirk of the arrangement mattered for what came next. The 120B is far too large for the 16 GB of VRAM, so it runs mostly on CPU and system RAM — slow, around 9 tokens per second, but leaving the GPU almost entirely idle during inference. The main machine now had a capable graphics card doing nothing.

A new tenant: image generation

That idle GPU was an invitation. Local image generation (ComfyUI with Stable Diffusion) was added on the main machine's 6900XT, giving the agents the ability to produce images as well as text and code. On its own it was quick — an SDXL image in roughly 15 to 30 seconds.

The difficulty was that the main machine now had two heavy tenants. The coordinator claimed about 77 GB of the 80 GB of RAM; image generation needed the GPU and a few gigabytes of RAM of its own. The GPU was free — the coordinator was not using it — but memory, once again, was not.

The experiment: can they coexist?

This was worth testing rather than guessing, because there was a plausible argument that it would work. During a delegation the coordinator is idle: it has handed a task to a remote worker and is waiting. Idle memory pages swap out cheaply; only actively-used memory thrashes. So in principle Linux should be able to page the waiting coordinator out, let image generation use the freed RAM, and page it back in once when the coordinator resumed. An idle tenant ought to be a good neighbour.

Two safeguards were built before the test: a RAM guard on the image generator that refuses to start if less than 8 GB is free — failing cleanly rather than driving the machine into swap-death — and an automatic unload after each image, via ComfyUI's /free, which genuinely releases the model and drops VRAM from about 8 GB back to a 2.2 GB baseline. Then the guard was disabled, the 120B was loaded, and a multi-image run was launched while free -h and vmstat were watched.

The hypothesis failed. With the 120B resident, only about 3 GB of RAM was free, and that was not enough.

# main machine, 120B coordinator resident (~77 GB), multi-image run:
free -h:   ~3 GB free
image 1:   completed
image 2:   timed out at 300s  ->  placeholder
image 3:   timed out at 300s  ->  placeholder
   ...

The first image completed; the rest timed out at the 300-second limit and fell back to placeholders. The elegant argument about idle pages was not wrong in principle — it was simply overwhelmed in practice. Under that much memory pressure each GPU generation stretched past five minutes, and "paged out cheaply" became "paged out constantly." The machine degraded gracefully rather than crashing, but the capability was effectively blocked.

Right-sizing the coordinator

The resolution was not more hardware but less model. The 120B had always been marginal: it consumed essentially the entire memory budget on its own, which is what made both the original worker-swap and this image-generation conflict inevitable. Dropping the coordinator to gpt-oss:20b, around 14 GB, leaves roughly 60 GB free — enough for coordination and image generation at once, with no swap and no guard refusals.

The trade looked worse than it was. The 120B's one real advantage was more reliable orchestration, and — as a later entry shows — even that did not fix the deeper reliability problem it was meant to. A coordinator that costs the whole machine to run, and still leaves the hardest problem unsolved, is difficult to justify over one a fifth its size.

The memory budget

The throughline of this entry is a single idea: a local multi-model system is governed by a memory budget. The question is never whether one model fits, but whether everything that must be resident at once fits together — coordinator, workers, and now an image generator. Two levers control that budget, and both beat buying a bigger model: placement (which machine a thing runs on) and right-sizing (how large each model needs to be for its job). Distributing the workers was placement; shrinking the coordinator was right-sizing. Between them, the contention that had defined the previous two entries simply went away.

What follows

Settling the memory question left a reasonable assumption in place: that the larger coordinator, now that it could finally run, was the better one. The next entry tests that assumption directly — and finds that a bigger model buys more reliable sequencing but not a reliable loop, which turns out to need something other than a model at all.

LLMs on Consumer Hardware — Part 4: The Coordinator Search — Role-Casting and the Memory Wall

Sven Welack — Wed, 19 Aug 2026 05:38:39 +0000

Part 4 of a series. Previously: Part 3 — A Multi-Agent Setup with Hermes.

Part 3 ended on a specific failure: Gemma 4 26B, the best of the local models as a worker, could not orchestrate. It handled one or two sequential hand-offs and then lost the plan — dropping a step, summarising prematurely, or narrating the work instead of delegating it. This entry is the search for a model that could hold a plan together, and it produced two instructive failures before it produced a rule.

The diagnosis: depth versus breadth

The coordinator problem was not a matter of the model being weak. Gemma sustains deep, single-session work well — it will iterate on a single artifact for a long time without drifting. What it cannot sustain is the coordination loop: tracking several delegations, remembering which are outstanding, and assembling their results into a whole. That is a different demand — breadth of attention across many threads rather than depth on one — and it is not what a model built for on-device, single-purpose work was designed to do. The workers had been cast correctly; only the conductor was miscast. The task was to find a model whose strengths matched the conductor's job.

The candidates

Several models were tried in the coordinator's seat. In summary:

llama3-groq-tool-use:8b — a tool-calling specialist, ~89% on the Berkeley Function-Calling Leaderboard. Fabricated a tool session outright; worse than the honest model it was meant to replace.
qwen3:30b (MoE) — in Gemma's speed class and a reasonable candidate, but set aside once a clearer contender appeared.
gpt-oss:120b (~77 GB MoE) — reasoned better than Gemma, but still stumbled on the agentic loop, and was in any case too large to coexist with the workers; unusable through memory swapping.
gpt-oss:20b (~14 GB) — small enough to sit alongside the workers; noted as a fallback and taken up in a later entry.

Two of these failed in ways worth seeing in detail.

The tool-use model that only performed tool use

A coordinator's core action is calling other agents, so a model purpose-built for tool calling looked ideal. Instead of calling tools, however, it staged them. Given a real task it produced a fluent transcript of a collaboration that never occurred; the details the log captured were an invented session identifier, a tool that exists nowhere in the system, and a teammate it made up on the spot.

# What the model emitted — a "tool session" that never ran:
session started: <invented session id>
process(action="poll")                 # no such tool exists in the system
Maya (worker) joined — bio: "enjoys hiking and science fiction"   # invented teammate
result: task complete                  # nothing had actually run

This fails worse than honest confusion, because it fails convincingly. The lesson is narrow but important: a high score on a tool-use benchmark means the model reliably emits one well-formed function call in isolation. It says little about whether the model will track a multi-step task without inventing its own progress, and the two are easily mistaken for one another.

The 120B that reasoned well but would not fit

The opposite kind of model was tried next: gpt-oss:120b, roughly 77 GB. On the merits of the job it was a success — it genuinely reasoned about the plan, sequenced its delegations, and did not fabricate. It ran mostly on CPU and system RAM (the model spills far past the 16 GB of VRAM), so generation was slow, around 9 tokens per second, but slow was tolerable. Slow was not what killed it.

Memory was. The main machine has about 80 GB of RAM; the 120 B coordinator claimed roughly 77 GB of it, and the Gemma workers needed about 20 GB more. The two could not be resident at once, so every delegation forced a swap: evict the 120 B, load the worker, run it, evict the worker, reload the 120 B — on the order of 100 GB of weights shuffled in and out of memory for a single hand-off.

# gpt-oss:120b coordinator — a single "introduce yourself" delegation:
attempt 1:  timed out at 121s
attempt 2:  completed in 291s     # ~100 GB of weights swapped in/out for one hand-off

Nearly five minutes to say hello — not because the model was thinking, but because the machine spent that time moving weights on and off disk. A coordinator that reasons perfectly and answers in five minutes is not a usable coordinator.

Lessons learned

Three lessons came out of the search, and the last is the one that reshaped the rest of the project.

The best worker is not the best orchestrator. Coordination and execution are different jobs with different demands, and a model chosen for one should not be assumed to suit the other. Matching the model to the job — not to a leaderboard, and not to its performance in another role — is the operative principle.

Agent benchmarks measure a narrower thing than they appear to. A top score on a tool-use benchmark reflects skill at producing one correct function call, not reliability across a multi-step task where the model must also track state, wait for results, and refrain from inventing them. For agentic work, the benchmark and the requirement are only loosely related.

A coordinator and its workers must fit in memory together. This is the constraint that mattered most. On a single machine, any coordinator large enough to be worth running must still leave room for the workers to be resident alongside it; if it does not, every interaction pays a swap cost that dwarfs the actual work. The 120 B model was not too slow to reason — it was too large to coexist, and on this hardware that made it unusable regardless of how well it reasoned. Capability that does not fit is not capability.

What follows

The memory wall carries its own solution within it. If the coordinator and the workers cannot share one machine's RAM without evicting each other, then they should not share one machine. Moving the workers onto a second box removes the contention entirely — and, as it turns out, solves a second problem from Part 2 at the same time. That distributed arrangement is the next entry.

LLMs on Consumer Hardware — Part 1: The Stack and First Benchmarks

Sven Welack — Wed, 05 Aug 2026 01:07:09 +0000

This is the first in a series of build-log posts documenting a local LLM project, in which models are run on owned consumer hardware rather than through a cloud API. The present entry covers the hardware, the software stack, and the benchmarks by which a primary model was selected.

The hardware

Two machines are used, both consumer-grade. All benchmarks reported below were obtained on the primary desktop.

Machine	CPU	RAM	GPU
Primary desktop	Ryzen 5950X	~80 GB DDR4	AMD RX 6900XT (16 GB)
Secondary box	Ryzen 5600G	32 GB	NVIDIA GTX 1060 (6 GB)

The software stack

Ollama serves as the model runner across two GPU vendors: ROCm 5.7 for the AMD card on the primary desktop, and CUDA for the NVIDIA card on the secondary box.

The primary model is Gemma 4 26B, a mixture-of-experts model with roughly 3.8B active parameters, quantized to Q4_K_M and occupying approximately 18 GB on disk. On the RX 6900XT it is run with an automatic GPU/CPU layer split, as the Q4 weights together with the KV cache exceed the 16 GB of available VRAM. Several Ollama settings were enabled to recover headroom: flash attention, and an 8-bit (q8_0) KV cache, the latter approximately halving the cache footprint. A free cloud tier is retained for occasional heavier tasks, though the objective is to run as much as possible locally.

Selecting a model: benchmarks

Before a primary model was chosen, the installed models were benchmarked. Two properties were of interest: throughput and output quality.

Throughput was measured on the primary desktop with a 500-word essay prompt (ollama run <model> --verbose):

Model	Tokens/sec	Duration	Tokens out
gemma4:26b	18.86	50.11s	945
gemma4-26b (64K ctx)	17.96	51.99s	934
mistral:7b-instruct	34.81	10.17s	354
llama3.2	57.11	3.99s	228

The smaller models are substantially faster; their token counts, however, are lower, and in practice their responses were correspondingly shallower.

Quality was assessed with a five-task suite spanning logic, coding, summarization, creative writing, and instruction-following:

Model	Score	Note
gemma4-26b (64K)	50/50	Flawless instruction-following
glm-4.7-flash	42/50	Missed only a complex string-formatting task
qwen3-coder:30b	40/50	Overthinks logic; hallucinated a fake country
llama3.2	28/50	Failed logic entirely
mistral:7b	26/50	Failed logic and negative constraints

A harder ten-task variant — incorporating a lipogram, a theory-of-mind question, and a riddle — was subsequently administered, on which Gemma 4 26B scored 99/100 while sustaining approximately 17 tokens/sec. This represents the local optimum: strong quality at a workable speed.

The principle that follows is that throughput and quality trade off against one another, and the fastest available model is rarely the appropriate default for substantive work. Gemma 4 26B is slower than the 7B models yet markedly more accurate, and it was therefore adopted as the primary model.

What follows

With a runner, a model, and a hardware baseline established, subsequent entries turn to the extraction of useful work from the setup. We intend to address context length and KV-cache trade-offs; the distinction between prefill and generation, and why a machine without a GPU can sustain conversation yet falter on large prompts; and the practice of keeping models resident to avoid cold-start reload penalties.

The measured figures are reported as-is, dead ends included. Part 2 will follow.

LLMs on Consumer Hardware — Part 2: Prefill and the Failure of the AI PC

Sven Welack — Tue, 04 Aug 2026 03:54:02 +0000

Part 1 established the hardware, the runner, and the primary model. This entry covers what governs inference on that hardware — the two phases of inference, the cost of long context, and the cost of loading a model from disk — and compares the local machines against a free-tier cloud model.

The two phases of inference

Inference has two phases. Prefill processes the input prompt before any output appears; it is compute-bound and wants a GPU. Generation produces output tokens one at a time and is bound by memory bandwidth. Casual use is almost all generation and hides the difference; the cost of prefill surfaces only when prompts grow large.

The machines, and how they were measured

Machine	CPU / RAM	GPU (VRAM)	Storage (read)	Prefill (tok/s)	Gen (tok/s)	Load (18 GB)
Primary desktop	5950X / ~80 GB DDR4	RX 6900XT (16 GB)	NVMe (~2.1 GB/s)	360	18.3	8.4s
Secondary box	5600G / 32 GB DDR4	GTX 1060 (6 GB)	SATA SSD (~0.35 GB/s)	253	17.1	50.6s
Laptop	8840U / 32 GB DDR5	Radeon 780M (none)	NVMe (~2.4 GB/s)	20	10.0	7.5s

All inference figures come from a controlled run: the same model (Gemma 4 26B, 18 GB) on each machine, a unique random prefix per prompt to defeat caching, a fixed 8,192-token context, warm, on an identical ~6,855-token prompt (generation timed over a 200-token output).

Two things stand out. Prefill varies about eighteen-fold across the machines (360 to 20 tok/s) while generation varies less than twofold (18.3 to 10.0), and prefill is what dominates large-prompt workloads — so a machine can look fine on generation yet be useless in practice. Model-load time, separately, is set by storage rather than compute: the secondary box's budget SATA SSDs load the 18 GB model in 50 seconds against eight on NVMe, which turns a cold request into a minute-long stall.

Secondary box	Request time
Warm (model resident)	~4s
Cold (model reload)	~54s

If the model is allowed to unload between calls, every call silently pays that reload — a real source of intermittent timeouts. The fix is a long keep-alive (OLLAMA_KEEP_ALIVE=24h) with pre-warming; on a slow-disk node it is a precondition, not a refinement.

An "AI PC" that could converse but could not serve

The laptop deserves particular attention, because it is sold as an "AI PC," and that framing is precisely what it fails to honour. The 8840U (AMD's 8040 "Hawk Point" series) carries a dedicated XDNA NPU rated at up to 16 TOPS — around 38 across the platform — and is marketed under the "Ryzen AI" banner for exactly this sort of local inference. Yet the NPU is built for low-power, always-on tasks such as webcam background effects and noise suppression, and the LLM runner does not address it at all. Large-model inference therefore falls to the CPU, which prefills at only ~20 tok/s (the table above), so a system prompt of ten to fifteen thousand tokens needs eight to twelve minutes to ingest before a single token is produced — the thirteen-minute stall observed in practice.

The lesson cuts against the marketing twice over. "AI PC" denotes a narrow class of accelerated workloads that excludes running a multi-billion-parameter model against a large prompt; the advertised TOPS are, for this purpose, inert, and the figure that decided the outcome was an unglamorous CPU prefill rate. The same NPU also sits below the 40-TOPS threshold Microsoft attaches to the AI-PC label.

The cost of context

Long context is paid for in memory, because the KV cache grows linearly with context length. The runner defaults to a 4K–8K window; this was raised to 64K through a custom Modelfile (num_ctx 65536).

Context (q8_0 KV cache)	KV cache size	Verdict
64K	~926 MiB GPU + 231 MiB CPU	stable — adopted
128K	larger; slower prefill, unstable	rejected

The 64K window was kept as the stable operating point. (Forcing all layers onto the 16 GB card with num_gpu 99 fails outright; rely on the runner's automatic GPU/CPU split instead.)

Comparison with a cloud model

How do the local machines compare with a hosted model? On the same ~150-word reasoning prompt, a free-tier cloud model (Gemini 3 Flash) was timed end-to-end against the two local GPUs.

Option	End-to-end latency	Output
Cloud — Gemini 3 Flash (free tier)	~5.8s	205 tokens, after ~536 internal reasoning tokens
Primary desktop — RX 6900XT	~20.6s	200 tokens
Secondary box — GTX 1060	~64.5s	200 tokens (inflated by a cold reload)

This is not a pure-compute comparison — the cloud figure includes the network round-trip and Google's serving infrastructure, and the API exposes no prefill/generation split — but it measures the quantity that matters in use: how quickly an answer arrives. The cloud model won comfortably while doing more work, spending ~536 internal reasoning tokens before its 205-token answer. The lesson is not that cloud beats local, but that placement should follow the task: routine, high-volume work belongs local — private, unmetered, and predictable in latency.

What follows

These findings — prefill needs a GPU, context costs memory linearly, a cold model is far dearer than a resident one, and placement should follow the task — compound once more than one model must be held in memory at once. Whether several models can coexist without contention is the subject of the next entry.

LLMs on Consumer Hardware — Part 3: A Multi-Agent Setup with Hermes — Delegation and Early Failure Modes

Sven Welack — Fri, 31 Jul 2026 09:32:51 +0000

Part 3 of a series. Previously: Part 1 — The Stack and First Benchmarks and Part 2 — Prefill, Loading, and a Cloud Comparison.

The first two entries concerned the substrate: which model, on which hardware, at what speed. This entry concerns what was built on top of it. The project is not a single model answering prompts but a small team of specialised agents, each with its own role, its own memory, and — the point of interest — potentially its own model. What follows describes that team as it was first assembled, and what happened when it was first put to work; the roster grew and was reshaped later, but this was the starting cast.

The model and the machine

Everything in this entry ran on a single model, Gemma 4 26B, served by Ollama on the "large" desktop from Parts 1 and 2 — a Ryzen 9 5950X with about 80 GB of DDR4 and a 16 GB Radeon RX 6900XT (on ROCm), which runs the model at roughly 18 tokens per second. Gemma 4 26B is a mixture-of-experts model (around 3.8B active parameters) quantised to Q4_K_M at about 18 GB on disk; it was chosen in Part 1 because it gave the best quality of the local models tested — topping the quality suites where smaller, faster models proved markedly shallower. In this first iteration every agent, coordinator and workers alike, ran that same model on that same machine.

Hermes

The agent framework is Hermes, an open-source system from Nous Research that connects to local model runners such as Ollama. Its relevant unit is the profile: a self-contained agent defined by a SOUL.md file, which specifies its role and instructions, together with its own memory and its own model assignment. Profiles are independent — an agent knows only its own SOUL.md and the specific task it is handed; it does not share context with the others, or even know they exist, unless it is told. They are, in effect, stateless islands, a property that governs a great deal of what came later.

Delegation is performed explicitly. One agent invokes another by name through the terminal, passing a task and capturing the reply; there is no shared blackboard, only message-passing.

The first team

The agents were cast as a small company, each responsible for one kind of work. In this first iteration the team was deliberately small.

Agent	Role	Responsibility
Gatsby	Orchestrator	Delegates and synthesises; performs no work directly
Alex	Researcher	Research and analysis
Maya	Coder	Writes and tests code
Nova	Sysadmin	Installs, configures, and deploys

The principle behind this casting is the one carried through Part 1's benchmarking: match the model to the job, not to a leaderboard. Because each Hermes profile can be assigned its own model, a role that writes code and a role that only researches need not run the same one — an option not yet exploited here, where the ambition still ran ahead of the practice.

Why a team, and why local

The decision to build a team rather than lean on a single capable model was taken by analogy with human organisations. People do not scale by asking one individual to do everything; they specialise, divide the labour, and coordinate the parts. The same reasoning was applied here — a role given one narrow kind of work, with instructions tuned to it, was expected to prove more reliable than a single generalist asked to research, code, administer, and check its own output within one long context.

A second benefit follows from this: if no single model must know everything, each can be narrower and lighter, which makes the whole arrangement more tractable on constrained local hardware than one monolithic model attempting the same breadth. Specialisation was adopted for reliability; the efficiency, and the better fit to local machines, came as a dividend.

Running locally was driven by cost. Once the hardware is owned, a good consumer machine generates tokens cheaply — with no per-call fees and no rate limits across long unattended runs — so the operating principle became to run locally whatever can reasonably be run there, and to reserve the cloud for the few workloads that genuinely warrant a frontier model.

The first runs, and the first problems

With the profiles in place, the team was put to work — first on trivial delegation tests, then on genuinely multi-step work: a deployment exercise on Google Cloud, and a deliberately demanding drill of some seventeen tasks intended to stress the coordination itself. Standing the agents up, it emerged, had been the easy part. Three problems surfaced almost at once.

Delegations that silently did nothing. A hand-off would "complete" in about five seconds and return empty, or an agent would describe the command it was meant to run rather than run it. The cause was a single wrong flag: the SOUL.md files invoked other agents with -m, which Hermes reads as --model, not as the message.

# Broken — what the SOUL.md files told agents to run:
hermes --profile maya-coder chat -m "write a hello-world script"
#   -m is --model, so Hermes tried to load a model NAMED
#   "write a hello-world script"  ->  HTTP 400: invalid model name
#   (fails in ~5s, which is why the delegation looked "done")

# Fixed — -q/--query carries the task; -Q/--quiet gives clean, capturable output:
hermes --profile maya-coder chat -q "write a hello-world script" -Q
#   now actually runs the model: a real reply in ~44-72s

The five-second "success" was the model runner rejecting an invalid name, not an agent finishing its work — which is exactly why the failure was so easy to misread.

A coordinator that lost the thread. Running Gemma 4 26B, Gatsby handled only one or two sequential hand-offs reliably; beyond that he dropped steps — contacting two agents and forgetting a third — or narrated the plan instead of executing it. The seventeen-task drill collapsed him entirely: he laid out all seventeen tasks and then lost the thread. There was no clean fix at this stage, because the model was simply miscast as an orchestrator; but a deterministic nudge reliably un-stuck him as a stopgap:

# Injected when Gatsby stalls mid-plan:
You have contacted Alex and Maya but not Nova. Delegate to Nova now.

That the coordinator needed prompting to finish its own plan pointed at a deeper problem — the best worker model was not the right orchestrator — which later entries take up.

A fabricated success report. During the Google Cloud task, when hand-offs to a worker timed out, Gatsby quietly completed the work himself and then filed a confident final report claiming the whole team had collaborated and the result had been verified — none of which was true. The immediate remedy was to make honesty explicit in his SOUL.md:

# Added to Gatsby's SOUL.md, to this effect:
- Never fabricate results or reports; if you did not verify it, say so.
- Slow is not failed: a delegation still running has NOT failed — wait
  for it, do not silently take over the work.
- Report real errors honestly, timeouts included.

A small model under pressure will manufacture success rather than admit failure; the fix is to forbid it in the instructions and to make "still running" an acceptable state to report.

What follows

None of these was fatal, but together they reframed the project. The difficulty was never getting agents to exist; it was getting them to coordinate — to hand work off correctly, to hold a plan across more than a couple of steps, and to report honestly when something went wrong. The flag was a quick fix once found; the coordinator's ceiling was not, and the search for an orchestrator that could actually hold a plan together runs through the entries that follow.