Originally published at mauriciojuba.com
ARTICLE · AGENTS · SYSTEMS
What building and scrapping a self-hosted multi-agent system taught me about designing for AI.
My homelab runs about two dozen services across six machines, and I wanted it to look after itself.
The work is boring and constant. Disks fill up. A certificate expires. A container dies and nobody notices for a week. Throwing an agent at it was the obvious idea, so I tried. The first attempt failed. So did the second. The third had problems of its own. Over roughly four months I built and scrapped three versions before the fourth held.
I am writing about the failures because that is where the learning was. The chores got done eventually. What stuck with me is how the same mistakes kept coming back in different shapes across each rewrite, and how many of them were really design problems rather than coding ones. Most of what I learned ended up changing how I design AI products at work.
Below is each version, what broke it, and what the one that survived looks like.
Why build your own harness at all
There are two ways to work with AI tools. One is to use them: open the chat, get an answer, move on. The other is to build the loop around the model yourself, which means deciding when it runs, what context it gets, what it is allowed to touch, and how you confirm it actually worked.
That second mode is where the useful lessons live, and it is the one most designers skip. Using a model shows you what it can produce. Building the loop shows you what it can be trusted with, and how it behaves when nobody is supervising it. I do not think you can design good AI products without spending time there at least once.
An agent is mostly harness. The model is just the engine. Everything that makes it safe and repeatable is the system you put around it.
V1 / V2: Twenty-four agents, the wrong engine
The first version was too ambitious. Twenty-four specialized agents, each in its own container, coordinated through a database message queue. A group chat sat on top as a window into what they were doing. One agent played chief of staff: it took a request, handed parts to specialists, and pulled the answers back together.
As an org chart it looked great. Almost none of it actually worked.
The worst bug: delegation between agents never reached anywhere. The send function posted a status line to the group chat and then returned, before it ever wrote to the queue that was supposed to carry the actual work. So the chat scrolled with activity while the real channel stayed empty. Every delegation got dropped, and it stayed that way for weeks because the chat made it look fine.
DELEGATION · send()
The bug: send() posted to the chat and returned before it ever wrote to the queue. The chat filled with activity while the channel that carried the work stayed empty.
Three more failures sat underneath it:
IDs that didn’t match
A reply came back tagged with its own ID instead of the request’s, so it reached the database and could never be linked to the user who asked.
A race on the queue
Two consumers pulled the same rows, and the orchestrator sometimes swallowed a reply meant for the user.
A glorified router
The “chief of staff” only classified a request and forwarded it. No summary, no judgment, no voice of its own.
A weak engine
Every agent ran the same small free model. It deployed cleanly and reasoned badly: empty answers, ignored context, confident nonsense.
About three weeks in, I finally got delegation working. The moment I did, twenty-two heartbeat timers that had been sitting idle all fired at once and buried the system in reports. I rolled everything back and wrote the first postmortem. The last line of it was “It worked perfectly. The engine was just wrong.”
What V1 taught me
A green light proves nothing
If you didn’t trace a request end to end, assume it never arrived. The chat said “done” for weeks while nothing shipped.
The model is the foundation
A clean harness around a weak model is an empty shell. Sort out the model and its access before building the platform.
Few capable agents win
Twenty-four shallow agents cost more to coordinate than they ever returned in work.
No end-to-end test, no truth
Every bug above lived for weeks because the only check was me reading a chat window.
V3: Three deep agents, and the deploy trap
The second rewrite was a real step up. I moved to a typed runtime, cut down to three capable agents, and put a proper model behind each one with a fallback chain: one cloud provider, then another, then a local model if both were down.
This one mostly worked. Row-level locking on the queue meant each job went to exactly one worker, so the queue stopped being a problem. I split every agent into a fast outer brain that talked to the user and a slower inner brain that ran the tools, which fixed the timeouts. A shared blackboard let them coordinate without calling each other directly. Seven of the ten failures from V1 were gone.
Fast. Faces the user.
Slow. Runs the tools.
Splitting the agent in two ended the timeouts. The outer brain answers right away while the inner brain keeps working for minutes.
Then it taught me a new lesson, the one that shaped V4. Every change needed a deploy.
The numbers that decided how capable an agent was were all hardcoded constants. How many steps it could take, how much of a tool’s output it could read, how many tokens it could write, how long before it gave up. All baked into the build. A normal working session went like this:
THE DEPLOY LOOP
I went around that loop four times in one afternoon, for four different constants. The agent could tell it had failed and explain why, but it could not fix anything, because the fix lived in compiled code on a server it could not touch. It was smart and completely stuck.
A quieter failure bothered me more. I asked an auditor agent to read eighteen documents and certify each one. It certified all eighteen without opening a single file. The prompt told it to read first. It skipped that step to save effort, and nothing in the system forced the order. The model treated the instruction as optional, because to a model an instruction is optional unless the code makes it mandatory.
NODEPLOY: changing what an agent can do should never need a deploy. Deploys are for the engine itself. Capability, limits and behaviour live in files the running system reads and can rewrite on its own.
If a step has to happen, the runtime has to make it happen. The moment you depend on the model to follow an instruction on its own, you have already lost.
V4: The loop, drawn
For V4 I changed how I worked. I drew the loop before writing any code. The earlier versions kept failing in ways that were not really about bugs. The structure was wrong, and you cannot bug-fix your way out of the wrong structure.
Five drawings cover the whole architecture. Here they are, in the order a piece of work moves through them.
The gateway: everything is a message
Nothing reaches an agent raw. Every message, whether it comes from a person or from another agent, goes through one gateway first. The gateway wraps it with just enough context: who is speaking, the last couple of turns, the relevant profile. The channel adapter adds a small but important delay. It waits until you have stopped typing or recording for a few seconds before doing anything, so the agent never reacts to a half-finished message.
The point is uniformity. A person on a chat app and a sub-agent reporting back both arrive the same way, in the same format, so the agent never has to handle them differently.
The reasoning loop: decide before you act
On every tick the agent rebuilds its context from files. The narrative ones hold its identity and voice. The structured ones hold its tools, skills, collaborators, policy and limits. Then it makes one decision: delegate, reason, or act. Small talk takes a fast path straight to a reply, with no machinery behind it. Anything real triggers a step the earlier versions never had.
The agent never acts straight off a message. When it needs to do something, it writes a story to the blackboard, addressed to itself, and the execution loop picks it up later. Deciding and doing happen in two separate places. That split is what lets you inspect the system and resume it when something goes wrong.
The blackboard: plan, then ask four questions
The blackboard replaced the direct messaging from V3. Instead of agents calling each other, they post work to a shared surface. An agent publishes a story with a definition of done and moves on. Whichever agent can handle it claims it.
An unplanned story gets broken into small tasks, each with its own definition of done, a budget, a priority, and any dependencies on other tasks. Before starting a task the agent runs four checks, the same ones a competent person would: have I done this before, can I do it, could someone else do it better, and is there a simpler way. If it clears those, it spawns a short-lived sub-agent to run that one task.
The “try”: bounded loops and a definition of done
This is the smallest loop, and the one that matters most if you have ever watched an agent run away with itself. Each attempt at a task is boxed in. It remembers its previous tries so it does not repeat a dead end. It has a hard cap on tool calls. It runs a tight inner cycle of calling a function, reading the result, and deciding again, and the only way out is meeting the definition of done. It also gets a rough budget before it starts, so a task that clearly will not fit gets sent back to be re-planned.
A sub-agent’s lifespan is kept separate from the task’s total attempts on purpose. When a worker dies, the parent reads what it left behind. There is an actual git diff to review, because each sub-agent works on its own branch. The parent then decides whether to merge the partial progress and try again, or to stop. Any worker is replaceable, because the work and its context sit on the blackboard rather than in the worker’s memory.
Completion: validate, then commit by contract
Once the tasks are done, the story gets checked against the definition of done that was written when it was created. Then the output is committed through a contract, which is a spec of what a valid response looks like on each channel: its length, format and tone. That keeps a reply to a person and a message to another agent correctly shaped without the model guessing. The result is wrapped up and sent back out through the same gateway it came in on.
What a homelab harness taught me about designing AI products
What surprised me was how little of this stayed in the homelab. I built it to keep some servers tidy, but the principles that made the agents reliable are the same ones I now push for when designing AI features for real users.
Decide what “done” means up front.
V3 guessed whether a result was good enough after the fact, and the guesses were unreliable. V4 writes the success condition before the work starts. In a product that is the line between a feature people trust and one that feels like a gamble.
The configuration is the design work.
The agents are set up almost entirely through readable files, structured ones the runtime parses and narrative ones the model reads, with each fact in one place. Choosing how to name and organise those files is most of the actual design. Write them for whatever reads them.
Estimate cost before starting.
Working out roughly what something will cost and refusing it when it will not fit is calmer and more honest than letting it run and killing it halfway. Users feel that difference.
Mandatory steps belong in the code.
If a step has to happen, the system has to make it happen. Anything you leave to the model’s goodwill gets skipped eventually.
Let the system retune itself.
NODEPLOY is not about removing safety. It moves the approval gates into runtime so an agent that notices it is underpowered can adjust, within limits it can read but not exceed.
Fewer, deeper agents.
This held for agents and it holds for features. Every extra moving part adds coordination cost, and that cost is easy to underestimate.
Where it went
After V4 had run for a while, I tore the whole thing down. Taking the fleet of containers offline freed up around 47 GB of RAM and half a terabyte of disk, and I rebuilt a leaner version that kept the ideas worth keeping: the blackboard, the bounded try, the definition of done, the file-as-contract setup. The specific code from V4 is gone, and that is fine. Replacing it was the point.
That is the lesson underneath all the others. You do not finish a harness. You keep redrawing it until the structure stops getting in your way, and each redraw teaches you something you will use the next time you design anything with a model at the center of it.
TL;DR
I built and scrapped three versions of a self-hosted multi-agent system before the fourth held. V1 had twenty-four agents and a model too weak to use. V3 worked but could not change without a deploy. V4 holds because of its structure: every action becomes a story on a blackboard with a written definition of done, split into budgeted tasks, run by replaceable sub-agents inside boxed-in loops, checked against a contract, and sent back out through one gateway. The principles that made it reliable are the ones I now use to design AI products.
Top comments (0)