Anannya Roy Chowdhury

Posted on Jul 16

My Multi-Agent AI Cost $1,847 in One Weekend — Here's the Fix That Cut It 82%

#agents #ai #architecture #programming

Part 1 of "Multi-Agent Systems in Production: What They Don't Tell You" — a four-part series following the saga of Horcrux Hunt, a multi-agent Harry Potter game that taught me everything about production AI the expensive way.

The Weekend My Game Cost More Than My Rent

I built Horcrux Hunt, an interactive Harry Potter-themed game where two AI agents battle each other live in front of an audience. Harry (the protagonist, powered by Claude on Amazon Bedrock via Strands SDK) hunts Horcruxes hidden across 15 locations. Voldemort (the adversary) relocates them, plants decoys, and corrupts Harry's beliefs.

Think of it as adversarial hide-and-seek between two LLMs, with a live audience voting and watching Harry's search unfold in real-time on a Streamlit dashboard.

It was supposed to be a fun weekend demo. The audience loved it. The CloudWatch metrics did not.

The bill: $1,847 for one weekend.

More than my rent.

And the performance was terrible:

12-second latency per turn (audience literally waiting)
23% win rate for Harry (he almost never won)
18% timeout rate (Lambda functions dying mid-thought)
67% of costs from Bedrock LLM calls alone

The audience feedback said things like "game is slow" and "Harry seldom wins." They didn't know it was also bankrupting me.

Why Multi-Agent Systems Cost More Than You Think

We build multi-agents systems. We keep adding agents for every task. But we never ask - "How many agents are too many" ? Here's the formula nobody shows you when they pitch multi-agent architectures:

Cost = tokens × agents × turns × retries × context_replay

Each term multiplies the others. It's not additive, it's multiplicative. Let me break down why, using Horcrux Hunt as the anatomy lesson.

Context Windows Compound Exponentially

Every turn of Horcrux Hunt, the game feeds Harry the entire conversation history like every search, every signal, every ally ability used. By turn 50, that's a LOT:

Every single turn, the LLM re-reads the entire conversation history. It's like Harry reading his complete mission journal from page one every time he makes a decision. By turn 50, you're paying 7.5× what turn 1 cost and that's just one agent.

With two agents (Harry AND Voldemort), you have 2× the token curve. Each agent maintaining its own inflating context.

Output Tokens Are the Hidden Killer

Most people focus on input tokens. But output tokens cost 5× more than input tokens on most models (Claude 3 Sonnet: $0.003/1K input vs $0.015/1K output). And they're sequential costing 12ms per token, and can't be parallelized.

Harry's responses averaged 150-200 output tokens per turn. Voldemort's averaged 100-150. Across 50 turns × 2 agents, output tokens accounted for 39% of wall-clock time. The audience was waiting for token generation, not thinking.

One Turn Is Actually 12 Operations

What I imagined a turn looked like:

Harry thinks → 2. Harry acts → 3. Voldemort responds

What a turn actually required:

Load game state from DynamoDB
Compute valid actions (which locations are on cooldown?)
Build Harry's context (compress game history)
Call Bedrock for Harry's decision
Validate Harry's response (is the action format correct? Is it a legal move?)
Execute Harry's action (update game board, resolve signals)
Compute Voldemort's context
Call Bedrock for Voldemort's decision (or use heuristic)
Validate Voldemort's response
Execute Voldemort's action (relocate Horcrux? Plant decoy?)
Update shared game state (locations, signals, scores)
Persist to DynamoDB + emit CloudWatch metrics

50 turns × 12 operations = 600 operations per game. The sequential nature is the real bottleneck. You can't parallelize Harry and Voldemort because Voldemort's action depends on what Harry just did.

Failing Slow, Failing Expensive

Here's what I noticed when I analyzed my retry rate: 15% of LLM responses failed validation. Harry would produce an invalid action format, or try to search a location on cooldown, or attempt to use an ally ability he'd already spent.

Each failure: 3 seconds of reasoning → rejected → full context replayed → try again. Each retry cost full token price.

The system was discovering failures after spending money. Failing slow (3 seconds into inference). Failing expensive (full token cost per failure). I'd later name the opposite of this pattern.

The Cost Math That Scared Me

Per game: ~$1.95 (naive approach)
1,000 games/day: $1,950/day
Monthly: ~$49,000
Annual: ~$588,000

For a game. A two-agent game where Harry chases Horcruxes.

The cost breakdown:

Bedrock (LLM calls): 67% — $1.31/game
Lambda (compute): 26% — $0.51/game
DynamoDB (state): 7% — $0.13/game

Now imagine deploying this as an always-on interactive experience. Imagine scaling to 1,000 simultaneous games. The numbers get terrifying fast.

The Four Fixes: An Optimization Ladder

I didn't need fewer agents. I needed fewer expensive decisions. Here's the optimization ladder pushing decisions DOWN from the costly LLM layer to cheaper alternatives.

Fix 1: Bound the Problem (Constraint Pruning)

Before: Harry saw 90 possible actions per turn (15 locations × 6 action types: search, attack, use_ally, investigate, fortify, retreat) and the LLM had to reason about which were valid.

After: A constraint solver prunes invalid actions before Harry's LLM sees them:

def get_valid_actions(game_state, agent="harry"):
    actions = []
    for loc in game_state.locations:
        if game_state.cooldown[loc] == 0:         # not on cooldown
            if game_state.usage_count[loc] < MAX_USES:  # not exhausted
                if game_state.budget > 0:           # has action budget
                    actions.append(loc)
    return actions  # typically 2-4 options, not 90

Result: 90 options → 2-4 valid options. Harry's LLM never wastes tokens reasoning about locations on cooldown or abilities already spent. 80% reduction in invalid reasoning tokens.

This is where I coined the principle: "Fail Fast, Fail Free."

The constraint solver catches bad decisions before they touch the LLM. An invalid action rejected by a 0.2ms Python function costs nothing. The same invalid action reasoned about by Claude for 3 seconds costs tokens, latency, and often a retry when post-inference validation fails.

Fail fast = catch it early. Fail free = catch it before the meter starts running.

Fix 2: Replace Tokens with Math (Bayesian Inference)

Before: Harry re-read 50 turns of narrative history (5,000 tokens) to figure out "where is the Horcrux probably located?"

Turn 1: Harry searched Hogwarts → negative signal
Turn 2: Harry searched Diagon Alley → positive signal  
Turn 3: Harry attacked Diagon Alley → decoy!
Turn 4: Voldemort relocated something...
... (50 turns of this = 5,000 tokens)

After: A Bayesian belief map computes probabilities outside the LLM:

class HorcruxBeliefMap:
    def __init__(self, locations):
        n = len(locations)
        self.beliefs = {loc: 1.0/n for loc in locations}  # uniform prior

    def update_on_signal(self, loc, signal):
        if signal == "positive":
            self.beliefs[loc] *= 3.0   # 3x more likely here
        elif signal == "negative":
            self.beliefs[loc] *= 0.1   # 10x less likely
        elif signal == "destroyed":
            self.beliefs[loc] = 0.0    # confirmed eliminated
        self.normalize()

    def top_targets(self, n=3):
        sorted_locs = sorted(self.beliefs.items(), key=lambda x: -x[1])
        return sorted_locs[:n]

What Harry's LLM actually sees: "top_target: Hogwarts (p=0.34), Azkaban (p=0.22)" = 15 tokens, not 5,000.

Result: 97% context reduction. Cost drops from $0.015 to effectively $0 for the belief computation. And Harry makes better decisions because probabilities are more precise than narrative intuition.

Another face of "Fail Fast, Fail Free" is if you can compute the answer with math ($0), why would you pay an LLM ($0.015) to infer it from narrative text?

Fix 3: Skip the LLM Call (Heuristic Decision Trees)

Not every Voldemort decision needs a $200B parameter model. Some game situations have obvious optimal moves:

def voldemort_decide(game_state):
    # If Harry is one move from a real Horcrux → relocate (obvious)
    if game_state.harry_adjacent_to_horcrux():
        return RelocateAction(game_state.threatened_horcrux)

    # If decoy budget available and Harry is confident → disrupt
    if game_state.decoy_budget > 0 and harry_entropy(game_state) < 1.5:
        return PlantDecoyAction(game_state.harry_top_target)

    # If early game with high uncertainty → do nothing (save budget)
    if game_state.turn < 10 and harry_entropy(game_state) > 3.0:
        return WaitAction()

    # Only genuinely complex situations need LLM
    return call_llm_for_strategy(game_state)

Result: 60% of Voldemort's decisions handled with zero LLM cost. Only the genuinely strategic moments where multiple valid strategies compete, justify an inference call.

Fix 4: Isolate the Expensive Layer (Architecture)

The most impactful change was structural. I redesigned Horcrux Hunt so only 2 of 8 modules ever touch the LLM:

FREE MODULES (6):
├── Game Engine         (rules, turn management)
├── Constraint Solver   (valid action computation)
├── Belief Manager      (Bayesian probability updates)
├── State Persistence   (DynamoDB read/write)
├── Validation Layer    (response format checking)
└── Metrics & Logging   (CloudWatch, dashboards)

EXPENSIVE MODULES (2):
├── Harry Strategic Layer    (genuinely uncertain decisions)
└── Voldemort Strategic Layer (only when heuristics can't decide)

The Interface Boundary compresses context at the border between free and expensive:

Input to LLM: 2,000+ tokens of raw game state → compressed to 55 tokens of AgentContext
Output from LLM: Validated immediately, rejected for free if invalid

This is "Fail Fast, Fail Free" as architecture: clear cost boundaries. Validation happens in the free zone. If something is going to fail, it fails in one of the 6 free modules, but never in the 2 expensive ones.

The Results

Same game. Same Harry. Same Voldemort. Same Claude Sonnet model. Radically different architecture:

Metric	Before	After	Change
Cost per game	$1.95	$0.35	-82%
Latency per turn	12s	3s	-75%
Harry win rate	23%	52%	+29pp
Timeout rate	18%	<3%	-83%
Retry rate	15%	<3%	-80%
Annual cost at scale	$588K	$102K	-$486K

Harry wins more, not because the model is smarter, but because he's reasoning over focused, relevant information instead of drowning in 6,000 tokens of noise. The audience sees 3-second turns instead of 12-second waits. The game is actually fun to watch now.

The Lesson

The answer to "How many agents are too many?" isn't a number. It's a question:

"Is this decision worth an LLM call?"

Most of Voldemort's decisions weren't. Most of Harry's probability calculations weren't. Most of the validation logic wasn't. Once I stopped paying the LLM to discover things I already knew, the costs fell off a cliff.

The Optimization Ladder:

Can a rule handle this? (Free — constraints, cooldowns, budgets)
Can a heuristic handle this? (Nearly free — if/then game logic)
Can math handle this? (Cheap — Bayesian updates, entropy)
Does this genuinely need an LLM? (Expensive — but justified for true uncertainty)

Push every decision as far DOWN that ladder as it can go. Fail fast, fail free at every layer.

🚀 What's Next

The bill is under control. But Harry's still losing.

He searches the same location twice. He forgets signals from 3 turns ago. He contradicts his own belief map. The problem isn't cost anymore, it's memory masquerading as reasoning failure.

→ Part 0: Fail Fast, Fail Free

Read this blog to know more about the principle.

→ Part 2: Why Your Agent Forgets where I discover that 68% of Harry's "reasoning failures" are actually retrieval failures, and the fix is entropy math, not bigger models.

Spoiler: I compressed Harry's context from 12,000 tokens to 340. Same LLM. Same prompts. Completely different agent.

💬 Quick diagnostic for your agent:

Run the same input 5 times. Does your agent give the same output every time?

✅ Yes → You probably have a cost or integration problem (Part 1 or 3)
❌ No → You definitely have a memory problem (next post)
🤷 Haven't checked → ...that's the scariest answer of all

Drop your answer below. I'll tell you exactly which post in this series has your fix. 👇

🔖 Follow me to get Part 2 when it drops or keep refreshing your inference bill until you're motivated enough to read it. Your choice. 💸

I am a Gen AI Developer Advocate & Architect. I built a multi-agent AI game to entertain a conference audience and accidentally created the most expensive stress test for multi-agent systems I'd ever seen. So. I adapted the classic "Fail Safe" and came up with "Fail Fast, Fail Free" after that $1,847 weekend taught me that the most expensive failure is one you discover too late.

Top comments (3)

Taruna Biswal • Jul 16

Another good read! My biggest takeaway from this is "does this really need an llm call" and not " how many more Agents should I add"!! If a rule based or mathematical model helps make decision, then why not? Best engineering decision ever :)