Part 1 of "Multi-Agent Systems in Production: What They Don't Tell You" — a four-part series following the saga of Horcrux Hunt, a multi-agent Harr...
For further actions, you may consider blocking this person and/or reporting abuse
The
Cost = tokens × agents × turns × retries × context_replayframing is the part I'd keep, and I ran it against your own anchor before commenting — because I think the formula understates you rather than overstates.If turn 50 costs 7.5× turn 1 under full replay, that pins the shape: per-turn input = B + D·n, and your 7.5× solves to B = 6.54·D (B = static prompt/state per turn, D = history added per turn). Over 50 turns, per agent:
So
turnsandcontext_replayaren't two independent factors, they're one Θ(N²) term wearing two hats. It shows when you double things:(modelled from your 7.5× anchor, not from your logs; offline, stdlib, no random, two runs same sha256 e5129e6f5f6a9cc4)
That reorders the ladder in one place: agent count is the honest linear term, turn depth is the one that compounds. Sliding-window replay over the same 50 turns, keeping the last k turns of history:
k=10 halves input without touching agent count — which is the opposite of where the "how many agents are too many" instinct sends you first.
Fix 1 is the one I'd defend hardest, though. Pruning 90 actions to 2-4 before the LLM sees them is a gate, not a retry policy, and it's the only one of the four that spends nothing to say no. The retry fixes still discover the cost after the money is committed; "Fail Fast, Fail Free" is really "don't let a decision reach the expensive layer unbounded". Looking forward to part 2.
This is such a thoughtful breakdown, Alexey. You're right that turn depth compounds much faster than most people intuitively expect.
I also love your point that Fix 1 is fundamentally a gate, not just an optimization. That's exactly what I was trying to get at with "Fail Fast, Fail Free"—prevent invalid or unnecessary reasoning from ever crossing into the expensive inference layer. Once an LLM is already thinking about something your system could have rejected deterministically, you've already lost on both cost and latency.
Your sliding-window numbers are a great illustration too. Part 2 actually dives into why I eventually moved beyond simple replay and into state compression, belief tracking, and retrieval strategies. I think you'll enjoy that discussion. Thanks for taking the time to model it so rigorously!
Thanks — and state compression is exactly where I'd expect Part 2 to go, because the window numbers understate what's happening. I extended the same model past your 50 turns:
The cut grows because full replay's cost/turn keeps climbing (32.0 → 407.0 D-units from turn 50 to 800) while the window's flattens at B+k = 16.5 and stays there. So it isn't a percentage discount, it's an exponent change: Θ(N²) → Θ(N). Which is also why the fix looks unimpressive at demo length and decisive at production length. (same sha256 twice, 20e2fedaada46e44)
The thing I'd be curious to see you tackle in Part 2: compression and belief tracking buy you that linearity, but they buy it by deciding what Harry is allowed to forget — and a compressor that silently drops the one signal that mattered fails exactly like your validation retries did, after the money is spent, except now it's also invisible. Replay is dumb but honest; compression is smart and can lie. Curious whether you ended up with a way to catch a bad compression before the turn runs on it, or whether you find out from the win rate.
That's a great observation. you've hit on the core challenge of Part 2. State compression isn't just about making context smaller; it's about deciding what information deserves to survive. A bad compressor can absolutely "lie" by omission, which is arguably more dangerous than naive replay because the failure is silent.
Looking forward to hearing your thoughts once Part 2 is out. I suspect you'll have plenty to challenge there as well. 😊
The belief map fix is the one I keep turning over, because it quietly changes what the model is even for. Once probabilities live outside the LLM, Harry stops being the thing that remembers the game and becomes the thing that picks between two or three ranked options, which is a much smaller job to pay for. Most multi-agent posts add agents to get better decisions. This one takes work away from them and the decisions get better anyway. That inversion is worth more than the 82%.
That's exactly the shift I was hoping to highlight. Once durable state and probabilities move into an external belief map, the LLM no longer has to reconstruct context or "remember" full context every turn. Then, it can focus on reasoning over a much smaller, well-defined decision space. The cost savings were a great outcome, but the bigger win was improved consistency, debuggability, and more predictable agent behavior. Thanks for capturing that so well.
One thing I appreciated is that this doesn't argue against multi-agent systems but it argues against using LLMs where deterministic logic, heuristics, or math are enough. The optimization ladder is a useful mental model to push every decision as far down the stack as possible before paying for inference. That's a principle that applies well beyond games to production AI systems in general.
Thanks for calling this out! Yes, the fight is never to have single or multi-agent, but how you design the system, how you standardize and optimize the workflow between your AI and orchestration layers.
Another good read! My biggest takeaway from this is "does this really need an llm call" and not " how many more Agents should I add"!! If a rule based or mathematical model helps make decision, then why not? Best engineering decision ever :)
I agree Taruna! Most decisions should go down the optimization ladder before your call reaches the expensive LLM layer
Love the breakdown of cost formula. It shows that cost grows, not linearly, but exponentially!
Thank you! Yes, when building multi-agent systems we forget that each agent has a cost associated with every llm action they take.