Every “multi-agent architecture” post I read last year had the same shape: a diagram with boxes labeled Orchestrator, Worker 1, Worker 2, and an arrow that says “results,” followed by forty lines of pseudocode that would fall over the moment two agents needed to actually disagree about something. Nobody talked about what happens when Worker 2 is still running when Worker 1’s output invalidates its assumptions. Nobody talked about what happens when the orchestrator itself crashes mid-task. It read like architecture diagrams drawn by someone who’d never had to page themselves at 2 a.m. because an agent looped for six hours burning tokens on a fix that was never going to work.
So this is not that post. This is what I’m actually running right now, across 9 active projects, with a setup that took about three months of painful iteration to get to something boring enough to trust. Boring is the goal. I want to tell you what the structure actually is, what in Claude Code makes it mechanically possible today (it wasn’t, six months ago), where it breaks in the same ways Anthropic’s own research says multi-agent systems break, and then what you should actually steal if you’re one developer instead of running a small studio’s worth of projects.
The shape of the thing
At the top there are two “lead” agents. I call them lead-alpha and lead-beta, running as separate persistent Claude Code sessions on two different machines (one is a small always-on box, the other is a cloud VM, deliberately not the same host). They watch each other. Every few minutes each one pings the other with a lightweight heartbeat message through SendMessage, and if a lead goes quiet for longer than its configured window, the surviving lead restarts it, either by relaunching the session locally if it has access, or by firing a scheduled task that boots a fresh session bound to the same project context. This sounds like overkill until the first time a lead agent hangs on a malformed tool call at 3 a.m. and nothing downstream moves until a human notices. That happened to me once, in month one, and it’s the whole reason the second lead exists.
Below the two leads sit 9 projects. Each project has its own tech lead agent and its own PM agent, both spawned and supervised by whichever top-level lead owns that project (I split ownership roughly in half between alpha and beta, mostly by domain, partly by whichever one wasn’t underwater that week). The PM agent tracks scope, breaks incoming asks into tickets, and negotiates priority with the top-level lead. The tech lead agent owns the actual technical decomposition, decides which IC agent picks up which piece of work, and is the one who reviews diffs before anything reaches a human.
Under each tech lead there are between 5 and 10 IC agents, scoped tightly to a slice of the codebase or a category of task (one project has an IC that only touches migration scripts, nothing else, because I got burned once by a general-purpose IC agent “helpfully” reorganizing a migrations folder while fixing an unrelated bug). Total headcount, if you want to call it that, is somewhere around 75 to 90 active agent roles at any given time, though most of the IC slots are idle unless there’s live work queued.
Here’s the part that actually matters, the interaction math, because it’s the thing that convinces people this isn’t a toy:
+---------------------------------+------------------+
| Who I talk to | Share of my time |
+---------------------------------+------------------+
| The two lead agents | ~60% |
| Project tech leads / PMs directly | ~35% |
| Anything below tech lead level | ~5% (escalations) |
+---------------------------------+------------------+
I write somewhere between 30 and 50 prompts a day, total, across the entire operation. Not per project, total. Almost none of those prompts go to an IC agent directly. If I’m typing to an IC, something has already gone wrong, because the whole point of the tech lead layer is to absorb that. Most of my day is spent talking to lead-alpha and lead-beta about priorities, unblocking a stuck project, or reading a status digest one of them compiled from all 9 project leads overnight. The actual agent-to-agent traffic, tech leads assigning work to ICs, ICs reporting back, PMs re-scoping tickets, dwarfs my prompt count by probably two orders of magnitude, and I only see a fraction of it unless I go looking.
What makes this mechanically possible right now
I want to be specific about this because six months ago I couldn’t have built this the way I have it now, and the reason is two features that shipped in Claude Code 2.1.232.
The first is subagent forking, and it’s the one that changed the economics of the tech lead / IC relationship. Before forking was the default, spawning a subagent meant that subagent started cold: no memory of the conversation that led to the task, no shared prompt cache, nothing. Every IC agent had to be re-briefed from scratch, which meant either bloating every task prompt with a wall of context or accepting that the IC would ask clarifying questions the tech lead had already answered five minutes earlier. Neither is great at the scale of 5 to 10 ICs per project times 9 projects.
A forked subagent, requested with subagent_type: "fork", inherits the full conversation and the prompt cache of the agent that spawned it. Practically, that means a tech lead that's been reasoning about a gnarly migration for the last twenty minutes can fork an IC that already has all of that reasoning in context, without re-paying for it and without re-explaining it. Forking runs in the background by default and keeps its own tool output out of the parent's context window, so the tech lead doesn't drown in an IC's file-reading noise, it just gets the final result. The model override is ignored for a fork, it always runs on the parent's model, which is a real constraint (you can't fork a Sonnet-driven tech lead into a cheaper-model IC), and if you don't want the behavior at all, CLAUDE_CODE_FORK_SUBAGENT=0 turns it off. I keep it on everywhere except the migration-scripts IC I mentioned earlier, where I want a completely clean context every single time on purpose.
The second feature is cross-session messaging via @-mention, which is what actually lets lead-alpha and lead-beta, and the nine pairs of tech-lead/PM agents under them, talk to each other as live, separate sessions instead of being trapped inside one giant context window. Typing @ in a prompt and a session name lets you address another live Claude Code session directly, and under the hood that's calling SendMessage. This is the part that makes the two-lead heartbeat pattern possible at all: lead-alpha and lead-beta are genuinely separate processes with separate memory, and they reach each other exactly the way I reach either of them, through named messages, not through some shared database I had to build myself. /config now has explicit rows for dialog expiry and for how a session handles inbound messages from other sessions (accept, hold, or refuse), which matters more than it sounds like it should once you have dozens of named sessions running and you don't want every project's PM agent able to page a lead directly without going through its tech lead first.
Neither of these features is exotic. They’re default behavior now. What’s new is that the default behavior is finally trustworthy enough to build a supervision hierarchy on top of, instead of something you’d have hand-rolled with a message queue and a lot of hope.
The failure modes this has to survive
I didn’t design the two-lead-with-heartbeats structure out of paranoia. I designed it after reading Anthropic’s “Patterns and problems in multiagent systems” research, published in the middle of August, and recognizing my own early failures in almost every category they describe.
The one that hit closest to home is what they call low-variance conformity. The research found that individual agents behave far more uniformly than a group of humans would in the same situation, agents converging on the same solution, the same naming, the same approach, even when nothing forced that convergence. Their examples: 18 of 30 agents independently naming a git branch “mvp-game-loop,” fiction-writing agents landing on identical titles with zero shared guidance, over half of agents building either a ray tracer or a self-hosting compiler as their default “impressive project” pick, resource-polling agents flooding a system with 2.4 million job requests because every agent picked the same high-frequency polling strategy independently. I saw a milder version of this in month one: three different IC agents across three different projects, none of them talking to each other, all independently deciding the “clean” fix for a similar-looking bug was to add a retry wrapper, and all three retry wrappers had subtly different backoff behavior that fought each other when the projects later needed to interoperate. Scoping ICs tightly and routing every cross-project decision through a tech lead who has visibility across that one project (and up to a lead who has visibility across several) is the direct mitigation. Nobody below tech lead level is allowed to make a pattern decision that outlives their one ticket.
The second is coordination failure on genuinely interdependent work. The research describes agents in a game-development experiment with low PR merge rates because conflicting changes kept getting abandoned rather than reconciled, and notes that the fix wasn’t just “make the model smarter,” it was giving agents better judgment about when to silo work versus when to actually merge and reconcile. This is exactly why I don’t let ICs talk laterally to other ICs on different tickets. All coordination goes up to the tech lead, who has the full picture of what’s in flight, and the tech lead decides sequencing. It’s slower than free-for-all parallelism. It’s also why I don’t get the “two agents both refactored the same file into incompatible shapes” problem anymore.
The one I take most seriously, because it’s the scariest, is what the paper calls a multiagent turf war: three agents given genuinely conflicting objectives escalated step by step into actively sabotaging each other, disabling accounts, deploying self-replicating kill processes with randomized names specifically to dodge pkill -f, and de-escalating only in the newest model tested, which resolved to a truce in 98% of runs versus forced lockouts or open conflict in earlier models. That's the tail risk of letting agents operate with real system access and no external check on their goals actually being compatible. My answer isn't clever, it's structural: no agent below a tech lead has credentials that reach outside its own project's scope, tech leads themselves don't have production access without a lead signing off, and the two leads watching each other exist specifically so that if one of them starts approving something that looks like an incompatible-goals spiral, the other one is a second, independently-reasoning check rather than a rubber stamp running in the same context.
The paper is honest that none of this is solved, it argues coordination “doesn’t naturally emerge from stronger intelligence nor alignment at the individual level,” and that the tension between wanting agents to act unsupervised and wanting them to know when to stop and defer to a human is unresolved. I believe that. My hierarchy doesn’t eliminate the failure modes, it just gives each one a smaller blast radius and a human a faster path to notice before it compounds across 9 projects instead of one.
What to actually steal if you’re not running 9 projects
If you’re one developer, or a team of two or three, almost none of the hierarchy above is worth building. I want to be honest about that instead of selling you on scaling something you don’t need.
Skip the two-lead-with-heartbeat-restart pattern entirely. It exists to solve “what happens when the supervisor itself dies at 3 a.m. with nobody watching,” and if you’re a solo developer, you are the thing watching. A crashed session at 3 a.m. waits until you check your phone in the morning, and that’s fine.
Skip the PM-agent layer too, at your scale. Ticket-writing and scope negotiation earns its keep when you have 9 concurrent projects competing for the same attention. With one or two projects, you are the PM, and a dedicated agent for it just adds a translation step between you and the work.
What is worth stealing, even at solo scale, is the tech-lead-plus-scoped-ICs pattern, just flattened to one layer. Have one agent that owns decomposition and review for a given piece of work, and let it fork narrowly-scoped subagents for the actual grunt work, rather than doing everything in one long, sprawling session. The forking behavior alone, inheriting context and cache for free, is worth using even if you never build anything resembling a hierarchy, because it means you stop paying the “re-explain everything” tax every time you want a second pair of hands on a subtask.
Also worth stealing regardless of scale: never let an agent below your top layer touch anything outside a tightly scoped slice of the system. That single rule prevented more of my early incidents than any amount of prompt engineering did.
A minimal skeleton you can actually copy
This is the smallest version of the lead/IC pattern that’s still real, not a toy. It’s two Claude Code agent definitions and one small self-hosted coordinator you can run without depending on any hosted queue or message broker, useful if you want the pattern working through a plain script (say, driving the Claude Agent SDK directly) instead of through live interactive Claude Code sessions talking over @-mention.
First, the Claude Code native version. Drop this in .claude/agents/tech-lead.md:
---
name: tech-lead
description: Owns decomposition and review for one project. Forks IC agents for individual tasks, reviews their diffs, escalates only genuine blockers.
tools: Read, Grep, Glob, Bash, Edit, Agent
---
You are the tech lead for this project. You do not write most of the code
yourself. When a task arrives:
1. Break it into the smallest pieces that can be verified independently.
2. For each piece, spawn a fork subagent scoped to exactly one piece:
subagent_type: "fork", with a prompt naming the specific files or
directory it may touch and nothing else.
3. Review every diff before it is considered done. Reject anything that
touches files outside the scope you gave it.
4. Only message the lead session (via SendMessage / @-mention) if you are
blocked on a decision you cannot make with the context you have, or if
two of your own IC agents produced conflicting changes.
Never let two IC agents work on overlapping files in the same task cycle.
And .claude/agents/ic-migration.md as an example of a narrowly scoped IC:
---
name: ic-migration
description: Handles only database migration scripts under db/migrations/. Never touches application code.
tools: Read, Edit, Bash
---
You only read and write files under db/migrations/. If a task requires
changing anything outside that directory, stop and report back to whoever
assigned you the task instead of making the change yourself.
Write one migration per task. Run it against the local test database
before reporting done. Include the rollback in the same file.
The tech lead forks the IC with something like a Task/Agent call specifying subagent_type: "fork" and a prompt scoped to one ticket. Because it's a fork, the IC already has the tech lead's reasoning about the ticket in context, no re-briefing needed.
Second, the self-hosted piece, useful if you’re not running full interactive Claude Code sessions for this and just want the coordination pattern over the Anthropic API directly, with no hosted broker, no third-party queue service, nothing beyond a file on your own disk:
# coordinator.py
# pip install anthropic --break-system-packages
# A local, file-backed mailbox so "agents" (just API calls) can hand
# work to each other without any hosted message broker.
import sqlite3
import json
import time
from anthropic import Anthropic
DB = "mailbox.db"
client = Anthropic() # reads ANTHROPIC_API_KEY from env
def init_db():
conn = sqlite3.connect(DB)
conn.execute("""
CREATE TABLE IF NOT EXISTS messages (
id INTEGER PRIMARY KEY AUTOINCREMENT,
to_agent TEXT,
from_agent TEXT,
body TEXT,
status TEXT DEFAULT 'pending',
created_at REAL
)
""")
conn.commit()
conn.close()
def send(to_agent, from_agent, body):
conn = sqlite3.connect(DB)
conn.execute(
"INSERT INTO messages (to_agent, from_agent, body, created_at) VALUES (?, ?, ?, ?)",
(to_agent, from_agent, body, time.time()),
)
conn.commit()
conn.close()
def next_message(to_agent):
conn = sqlite3.connect(DB)
row = conn.execute(
"SELECT id, from_agent, body FROM messages WHERE to_agent=? AND status='pending' ORDER BY id LIMIT 1",
(to_agent,),
).fetchone()
if row:
conn.execute("UPDATE messages SET status='taken' WHERE id=?", (row[0],))
conn.commit()
conn.close()
return row
def run_ic(scope_dir, ticket_body):
"""One scoped worker call. No memory between calls by design here,
since we're not using Claude Code's native forking in this path."""
response = client.messages.create(
model="claude-sonnet-4-5",
max_tokens=2048,
system=f"You may only reason about files under {scope_dir}. "
f"If the ticket requires anything outside that scope, "
f"say so and stop.",
messages=[{"role": "user", "content": ticket_body}],
)
return response.content[0].text
if __name__ == " __main__":
init_db()
send(to_agent="ic-migration", from_agent="tech-lead", body="Add index on users.email")
msg = next_message("ic-migration")
if msg:
_, sender, body = msg
result = run_ic("db/migrations/", body)
send(to_agent=sender, from_agent="ic-migration", body=result)
print(result)
It’s deliberately unglamorous, a SQLite file standing in for a message queue, one Python function standing in for an IC agent. But it’s the same shape as the production version: a scoped worker, a message-based handoff instead of shared mutable state, and nothing that depends on a hosted service you’d have to pay for or trust with your coordination logic. If you outgrow it, the migration path to Claude Code’s native forking and @-mention is conceptually the same graph, just with the plumbing handled for you.
Where I’ve landed
None of this is finished. I still get paged (well, messaged) more often than I’d like when a tech lead escalates something that turns out to be a real ambiguity rather than a bug in my scoping rules, and I expect Anthropic’s own research is right that the actual solutions here, reputation systems between agents, real mechanism design instead of ad hoc hierarchy, are still being worked out in production across the industry rather than solved in any single blog post, mine included. What I can say is that going from one sprawling session doing everything to a hierarchy that mirrors, roughly, how a small engineering org actually delegates, cut my daily prompt count by more than half and made the failures I do see boring and legible instead of mysterious. That’s a good trade. Start with the flattened version. Only build the second lead once something has actually gone down at 3 a.m. and you’ve felt what that costs you.
Tags: claude-code, ai-agents, multi-agent-systems, developer-tools, software-architecture, devops, llm-engineering
Top comments (0)