Part 4 ended against a wall. A coordinator capable enough to be worth running — the 77 GB gpt-oss:120b — could not share a machine with the workers it had to call; every delegation forced roughly 100 GB of model-swapping, and a greeting took nearly five minutes. The conclusion was that a coordinator and its workers must fit in memory together. This entry takes that constraint at its word and does the obvious thing: it stops making them share.
Moving the workers off the machine
The fix was to put the workers on a different machine. The second box introduced in Part 2 — the 5600G with the GTX 1060 — became the worker host; this is how it got that job. With the workers resident there, the main machine's memory was no longer contested: the coordinator had it to itself, and nothing local evicted it between hand-offs.
The effect on the 120B was immediate. The same model that had thrashed itself into uselessness now loaded once and stayed resident for 1 hour 46 minutes across a long run, with no swapping at all. Freed from the swap penalty, its orchestration was also more reliable than Gemma's had been. The lesson was almost deflating in its simplicity: the coordinator had never been the problem. Its placement had.
The second machine thereby earned its keep twice. Part 2 put it there because its discrete GPU fixed prefill — a GPU-less box could not ingest the large prompts in any reasonable time. Moving the workers onto it solved a second, unrelated problem: it freed the main machine's RAM for the coordinator. One hardware decision, two constraints resolved.
One quirk of the arrangement mattered for what came next. The 120B is far too large for the 16 GB of VRAM, so it runs mostly on CPU and system RAM — slow, around 9 tokens per second, but leaving the GPU almost entirely idle during inference. The main machine now had a capable graphics card doing nothing.
A new tenant: image generation
That idle GPU was an invitation. Local image generation (ComfyUI with Stable Diffusion) was added on the main machine's 6900XT, giving the agents the ability to produce images as well as text and code. On its own it was quick — an SDXL image in roughly 15 to 30 seconds.
The difficulty was that the main machine now had two heavy tenants. The coordinator claimed about 77 GB of the 80 GB of RAM; image generation needed the GPU and a few gigabytes of RAM of its own. The GPU was free — the coordinator was not using it — but memory, once again, was not.
The experiment: can they coexist?
This was worth testing rather than guessing, because there was a plausible argument that it would work. During a delegation the coordinator is idle: it has handed a task to a remote worker and is waiting. Idle memory pages swap out cheaply; only actively-used memory thrashes. So in principle Linux should be able to page the waiting coordinator out, let image generation use the freed RAM, and page it back in once when the coordinator resumed. An idle tenant ought to be a good neighbour.
Two safeguards were built before the test: a RAM guard on the image generator that refuses to start if less than 8 GB is free — failing cleanly rather than driving the machine into swap-death — and an automatic unload after each image, via ComfyUI's /free, which genuinely releases the model and drops VRAM from about 8 GB back to a 2.2 GB baseline. Then the guard was disabled, the 120B was loaded, and a multi-image run was launched while free -h and vmstat were watched.
The hypothesis failed. With the 120B resident, only about 3 GB of RAM was free, and that was not enough.
# main machine, 120B coordinator resident (~77 GB), multi-image run:
free -h: ~3 GB free
image 1: completed
image 2: timed out at 300s -> placeholder
image 3: timed out at 300s -> placeholder
...
The first image completed; the rest timed out at the 300-second limit and fell back to placeholders. The elegant argument about idle pages was not wrong in principle — it was simply overwhelmed in practice. Under that much memory pressure each GPU generation stretched past five minutes, and "paged out cheaply" became "paged out constantly." The machine degraded gracefully rather than crashing, but the capability was effectively blocked.
Right-sizing the coordinator
The resolution was not more hardware but less model. The 120B had always been marginal: it consumed essentially the entire memory budget on its own, which is what made both the original worker-swap and this image-generation conflict inevitable. Dropping the coordinator to gpt-oss:20b, around 14 GB, leaves roughly 60 GB free — enough for coordination and image generation at once, with no swap and no guard refusals.
The trade looked worse than it was. The 120B's one real advantage was more reliable orchestration, and — as a later entry shows — even that did not fix the deeper reliability problem it was meant to. A coordinator that costs the whole machine to run, and still leaves the hardest problem unsolved, is difficult to justify over one a fifth its size.
The memory budget
The throughline of this entry is a single idea: a local multi-model system is governed by a memory budget. The question is never whether one model fits, but whether everything that must be resident at once fits together — coordinator, workers, and now an image generator. Two levers control that budget, and both beat buying a bigger model: placement (which machine a thing runs on) and right-sizing (how large each model needs to be for its job). Distributing the workers was placement; shrinking the coordinator was right-sizing. Between them, the contention that had defined the previous two entries simply went away.
What follows
Settling the memory question left a reasonable assumption in place: that the larger coordinator, now that it could finally run, was the better one. The next entry tests that assumption directly — and finds that a bigger model buys more reliable sequencing but not a reliable loop, which turns out to need something other than a model at all.
Top comments (0)