I built an AI orchestration system for real feature work, and it broke in a way that took me a while to understand
I'm a backend engineer, mostly .NET, mostly microservices, with Go in the mix too. A few months ago I got tired of babysitting AI coding agents one prompt at a time, so I built a three-layer system to run real feature work end to end — planning, execution, review, integration. Not a toy. I use it on my actual codebase, on actual tickets.
It worked for a while. Then it broke, in a way that took me longer than I'd like to admit to actually name. This post is about that break and what I changed, because I think the fix is more general than my specific stack.
What I built
Three layers, each with one job: Claude Code plans and owns correctness, Hermes executes, and 9Router routes model capacity underneath both.
Claude Code is the brain of the thing. It reads the request, explores the repo, writes the spec, decides architecture, and breaks the feature into tasks. Architecture doesn't get delegated to workers. If a worker hits an unresolved architecture question mid-task, the branch pauses and it goes back to a fresh architecture session. That boundary was deliberate from the start — I didn't want implementation workers quietly making structural decisions on their own.
Hermes is my execution runtime. It owns the Kanban board, dispatches tasks whose dependencies are satisfied, runs independent tasks in parallel, creates isolated Git worktrees per task, retries failures, and — this part matters a lot — keeps "the provider fell over" and "the code is wrong" as two completely different failure classes, because they need to be handled differently.
9Router sits behind Hermes as an OpenAI-compatible endpoint and routes to whatever model capacity is actually available. Tasks get an execution lane and 9Router resolves that to a real provider. I didn't want the planning layer to know or care which specific model was doing the work underneath.
This already had decent properties on its own. Architecture stayed with Claude instead of leaking into worker sessions. Work happened in isolated worktrees, so a dead session didn't mean lost work. Deterministic checks ran before anything semantic. A fresh reviewer looked at finished work without inheriting the implementer's assumptions. A whole-feature integration gate caught cross-task drift before anything got marked done.
I should say clearly: this was never a "throw more agents at it" setup. The planner already refused to split tightly coupled work into microtasks, already avoided maximizing parallelism just because workers were free, already gave each worker bounded context instead of the whole repo, and already pinned shared decisions before workers touched them. I'm mentioning this because it's easy to read what comes next and assume the old design was naive. It wasn't. It just had one gap I couldn't see until I hit it.
Here's roughly what the shape looks like end to end:
The problem wasn't capability, it was shape
Here's the thing nobody really warns you about with multi-agent orchestration: the workflow itself can end up heavier than the change it's delivering. A feature gets split into coupled tasks, each task spins up its own agent context, and the same piece of state ends up duplicated across a few files that can quietly drift apart from each other.
The obvious response is: don't split so much, keep one strong agent alive for the whole feature, less handoff, more continuity.
That doesn't work either, at least not for me. A long-lived session's context just accumulates. Repo exploration, tool output, build logs, old plans, superseded decisions, repeated summaries. A big context window is not the same thing as reliable memory. I watched a session hang onto a stale test result from way earlier in the conversation while it quietly dropped a business rule that had only been stated once, near the start. The information wasn't gone exactly. It was just buried under everything else.
So I kept running into what felt like a false choice — one huge session that slowly forgets things, or several small sessions that don't share enough state to stay consistent. I tried both directions seriously, and both failed, just in different places.
A small example that made me suspicious
I use a discount-code feature as a running test case, because it has enough interlocking rules to expose sloppy boundaries without being a huge feature:
- The code must be active and unexpired.
- The code and the order must belong to the same store.
- The order subtotal must meet the code's minimum amount.
- The discount can't exceed the code's maximum amount.
- Only one code per order.
- Money is
decimal, always.
Split naively into separate tasks for the entity, the calculation service, the API wiring, and tests, I got roughly the failure you'd expect. The database worker stored percent as an int from 1 to 100:
public int Percent { get; set; }
The service worker was given basically the same vague task description and assumed percent was a decimal from 0 to 1:
var discount = order.Subtotal * code.Percent;
Neither worker did anything wrong given what it had in front of it. Both reported local success. It was integration that discovered the contract had never actually been agreed on.
Run the same feature as one long session instead, and I got a different version of the same underlying problem. Late in the session it produced a calculation that correctly checked activity and expiry, but silently dropped the store-match rule and the maximum-discount cap:
public decimal ApplyDiscount(Order order, DiscountCode code)
{
if (!code.IsActive || code.ExpiresAt < DateTime.UtcNow)
return order.Subtotal;
return order.Subtotal * (1 - code.Percent / 100m);
}
Two very different orchestration strategies, and I got the same category of bug from both. That's roughly when I stopped asking "how many agents" and started asking what a task actually is in this system.
The distinction I was missing
I'd basically been treating three things as the same thing:
- A task — durable business work, with scope, acceptance criteria, and an outcome.
- A session — one disposable model context.
- An agent — a model instance acting inside that session.
My system mapped one task to exactly one session, every single time. That mapping is what forced the false choice from earlier. If a task needed more room than one session could hold cleanly, my only two options were "make the session longer" or "make the task smaller," and I'd already seen both of those routes fail.
The option I'd missed was to let a task outlive a session. Keep one coherent unit of work, and continue it across several fresh sessions on the same worktree, without ever pretending it's a different task. The contract stays fixed. The context gets to reset. That's really the whole idea, and I feel a bit dumb that it took me this long to state it that plainly.
This is the part of the whole redesign that actually matters, so here it is on its own:
What I actually changed
I didn't touch the three-layer architecture. Claude still plans and owns final correctness, Hermes still executes, 9Router still routes capacity. What changed is what Hermes treats as the durable unit, and how strictly it protects that unit's identity once execution starts.
A contract hash, pinned on first claim. The moment a task is first picked up, Hermes computes a SHA-256 fingerprint over its ID, title, body, worktree, and branch. Every later session touching this task has to match that hash exactly. If the task description drifted between sessions — planner mistake, stale prompt, whatever — Hermes doesn't quietly keep going under the old worktree with a slightly different task. It refuses and routes the task to a blocker state I added, planning_mismatch, for a human to look at.
Rotation based on real usage, not a guess. I used to have a separate watcher trying to estimate when a session was getting too full. Now Hermes reads its own measured prompt-token usage directly. When it crosses a threshold, it waits for the current tool batch to actually finish — never mid-tool-call — writes a checkpoint, and closes the run as checkpointed. The same task goes back to its retry phase and picks up a fresh session automatically.
Checkpoints are structured, and deliberately boring. Each one is JSON, and it does not try to be clever. It records task and run identity, the contract hash, worktree, branch, context usage at the time it fired, the current diff and test state, next action, and any blockers. It does not ask the model to write a prose "here's where I left off" summary — I decided early on that inventing a semantic progress narrative just to justify rotating context was exactly the kind of thing that could quietly go wrong. The evidence is Git, the diff, and test output. The checkpoint just points the next session at that evidence.
Human approval on exact task IDs before anything runs. Every implementation task has to state its single observable business outcome, which spec requirements it owns, why its boundary was drawn where it was, and whether it's independently shippable and rollback-able. The graph sits at pending_human until a person approves those exact task IDs. My own validation script rejects the graph outright if that evidence is missing.
I also ran a real SQLite dry run afterward, specifically to prove the mechanism rather than just trust the design on paper: two separate runs continued the same task, same worktree, same branch, same contract hash. That part I've actually verified, not just designed.
Running the same feature through both versions
To make the difference concrete, I used the same representative discount-code scenario and ran it through both the old workflow and the new one, start to finish. It's not a real ticket from my codebase — it's a controlled example I keep around for exactly this kind of comparison, small enough to walk through fully but with enough interlocking rules to be honest about the trade-offs. Same rules as above, plus one more: existing orders without a code keep their current behavior.
Because the feature touches money and persistence, my planner correctly refused to parallelize it under the old system. It recognized the three concerns were coupled and made them serial:
PLAN
-> TASK-001: Add discount persistence contract and migration
-> VAL-001: Review persistence and compatibility
-> TASK-002: Implement pricing behavior and focused unit tests
-> VAL-002: Review business rules and edge cases
-> TASK-003: Connect order creation, persistence, and integration coverage
-> VAL-003: Review API/persistence integration and retry behavior
-> INTEGRATION: Full build and relevant full test suite
-> FINALIZE: Launch fresh final review
That's three implementation workers, three independent semantic reviewers, an integration worker, a finalization worker, and a final reviewer, on top of the original planner — up to ten separate model contexts for one coherent piece of behavior, before any retries. Along the way the feature workspace also picked up around ten overlapping files: spec, task list, execution notes, status doc, decisions log, the graph itself, and three per-task YAML files, on top of whatever the Hermes board and run metadata tracked separately. None of these files were wrong individually. They just each held a partial view of the same feature, and keeping them from drifting apart was its own quiet ongoing cost.
Under the new system, the same feature became one durable task, run through three fresh sessions on a single worktree:
PLAN
-> TASK-001: Implement discount-code behavior end to end
-> INTEGRATION: Full build and relevant full suite
-> FINALIZE: Fresh independent final review and knowledge check
The first session does discovery: confirms the money type, the transaction boundary, persistence conventions, which service owns pricing. When it hit my context threshold mid-task, Hermes closed the run with a checkpoint like this:
{
"reason": "context_budget",
"task_id": "TASK-001-runtime-id",
"run_id": 1,
"contract_hash": "sha256:discount-contract",
"workspace_path": "/repo/.worktrees/discount",
"branch_name": "wt/discount",
"context": {"prompt_tokens": 90100, "threshold_tokens": 90000},
"handoff": {
"completed": [],
"remaining": ["Resume from the existing worktree and inspect its diff/test state."],
"tests": [],
"next_action": "Continue the same task from the existing worktree.",
"blockers": []
}
}
Nothing invented, nothing summarized from memory. Task and run identity, the contract hash, worktree and branch, context usage, and whatever the diff and tests already show. The next session reads this and picks up the actual evidence, not a story about the evidence.
A second session picks up the same task, same worktree, same contract hash, and implements the pricing calculation and its unit tests. A third session connects order creation to it, persists the discount and final amount, and covers retry and idempotency behavior. Same three logical concerns as before — persistence, pricing, integration — just handled as three sessions inside one task instead of three separate tasks that each needed their own reviewer and their own merge back to a shared branch.
One fresh adversarial reviewer looks at the whole thing at the end, same as before. The mandatory integration gate still runs, same as before. Nothing about correctness got weaker. What went away was the ceremony between the concerns that were never actually independent — no cross-task merges, no three separate semantic reviews of pieces of one behavior, and the canonical artifact count dropped from around ten files to three: spec, status, and the graph. For this specific example, that's roughly six or seven model contexts doing the job the old flow needed up to ten for.
I want to be careful with these numbers. They're for this one feature, not a benchmark, and a bigger or more genuinely independent feature would look different. But the shape held up when I re-ran similar coupled features — the win isn't doing less work, it's not paying a coordination tax between pieces of work that were never separable in the first place.
Zoomed out past this one feature, here's the general shape of what changed and what didn't:
Laid out side by side, the two designs differ less than you'd think — same guarantees, different unit of ownership:
| Dimension | Old workflow | Redesigned workflow |
|---|---|---|
| Logical unit of work | Technical task per layer (persistence, pricing, API) | One coherent business behavior |
| Execution shape | Serial chain of separate task workers | One durable task, several sequential sessions |
| Worktree ownership | One worktree per task | One worktree for the whole behavior |
| Context lifetime | Effectively tied to the task | Disposable — resets independently of the task |
| Contract lifetime | Pinned once per task boundary | Pinned once, hashed, enforced on every session |
| Handoff mechanism | Merge to base branch + task contract | Structured checkpoint (identity, diff, tests, next action) |
| Semantic review | One reviewer per coupled task | One reviewer for the whole behavior |
| Parallelism | None — coupled work stays serial | None — coupled work stays serial |
| Integration | Mandatory whole-feature gate | Mandatory whole-feature gate, unchanged |
What I'm not going to claim
It's tempting to round "this feels better" up to "this is proven," and I don't think that's earned yet.
Implemented and verified: the coherence gate, the contract hash, checkpoint-based rotation, planning_mismatch, structured telemetry, and the two-run SQLite continuation dry run.
Not yet done: getting this fully live in production for new work, and — the important one — a real comparative benchmark. I don't have numbers yet on token cost, latency, defect rate, or hallucination rate against the old approach. My telemetry can tell me how many sessions a task used, how many checkpoints fired, how many completion-guard trips happened, and I've deliberately labeled that last one a proxy — a guard tripping means a safeguard fired, not that the model actually hallucinated.
The plan is to run a batch of comparable tasks split between the old and new approach and measure it properly: time from request to approved review, first-pass test rate, first-pass integration rate, human interventions per task, retries by cause, defects that survive to post-merge. Until that runs, everything above is "the mechanism works as designed," not "this is faster or better."
The actual takeaway
If you're orchestrating multiple AI agents on real work, the question that mattered most for me wasn't how many agents to use or how much context to give them. It was whether my system's idea of a "session" matched the actual unit of ownership, or whether it was quietly forcing the two to be the same thing.
A durable engineering task may outlive the temporary model session executing it. Once I separated those two lifetimes, and made the system fail loudly instead of quietly when task identity broke, the bugs I'd been chasing stopped feeling random. They were always going to happen once a durable piece of work got tied to a disposable context lifespan.
I still have to prove this is actually better. But I know what I'm measuring now, and I understand why the old version was going to hit this wall eventually.



Top comments (0)