DEV Community: Anannya Roy Chowdhury

Have you wasted cost on too many tokens and too many interactions? I did too. The fix wasn't a better model. It was a better architecture. Moving state outside the LLM & simplifying agent interactions cut costs by 82%. Read more here on how I did it.

Anannya Roy Chowdhury — Fri, 17 Jul 2026 08:00:43 +0000

Anannya Roy Chowdhury

Jul 16

My Multi-Agent AI Cost $1,847 in One Weekend — Here's the Fix That Cut It 82%

#ai #agents #programming #architecture

8 min read

My Multi-Agent AI Cost $1,847 in One Weekend — Here's the Fix That Cut It 82%

Anannya Roy Chowdhury — Thu, 16 Jul 2026 11:31:13 +0000

Part 1 of "Multi-Agent Systems in Production: What They Don't Tell You" — a four-part series following the saga of Horcrux Hunt, a multi-agent Harry Potter game that taught me everything about production AI the expensive way.

The Weekend My Game Cost More Than My Rent

I built Horcrux Hunt, an interactive Harry Potter-themed game where two AI agents battle each other live in front of an audience. Harry (the protagonist, powered by Claude on Amazon Bedrock via Strands SDK) hunts Horcruxes hidden across 15 locations. Voldemort (the adversary) relocates them, plants decoys, and corrupts Harry's beliefs.

Think of it as adversarial hide-and-seek between two LLMs, with a live audience voting and watching Harry's search unfold in real-time on a Streamlit dashboard.

It was supposed to be a fun weekend demo. The audience loved it. The CloudWatch metrics did not.

The bill: $1,847 for one weekend.

More than my rent.

And the performance was terrible:

12-second latency per turn (audience literally waiting)
23% win rate for Harry (he almost never won)
18% timeout rate (Lambda functions dying mid-thought)
67% of costs from Bedrock LLM calls alone

The audience feedback said things like "game is slow" and "Harry seldom wins." They didn't know it was also bankrupting me.

Why Multi-Agent Systems Cost More Than You Think

We build multi-agents systems. We keep adding agents for every task. But we never ask - "How many agents are too many" ? Here's the formula nobody shows you when they pitch multi-agent architectures:

Cost = tokens × agents × turns × retries × context_replay

Each term multiplies the others. It's not additive, it's multiplicative. Let me break down why, using Horcrux Hunt as the anatomy lesson.

Context Windows Compound Exponentially

Every turn of Horcrux Hunt, the game feeds Harry the entire conversation history like every search, every signal, every ally ability used. By turn 50, that's a LOT:

Every single turn, the LLM re-reads the entire conversation history. It's like Harry reading his complete mission journal from page one every time he makes a decision. By turn 50, you're paying 7.5× what turn 1 cost and that's just one agent.

With two agents (Harry AND Voldemort), you have 2× the token curve. Each agent maintaining its own inflating context.

Output Tokens Are the Hidden Killer

Most people focus on input tokens. But output tokens cost 5× more than input tokens on most models (Claude 3 Sonnet: $0.003/1K input vs $0.015/1K output). And they're sequential costing 12ms per token, and can't be parallelized.

Harry's responses averaged 150-200 output tokens per turn. Voldemort's averaged 100-150. Across 50 turns × 2 agents, output tokens accounted for 39% of wall-clock time. The audience was waiting for token generation, not thinking.

One Turn Is Actually 12 Operations

What I imagined a turn looked like:

Harry thinks → 2. Harry acts → 3. Voldemort responds

What a turn actually required:

Load game state from DynamoDB
Compute valid actions (which locations are on cooldown?)
Build Harry's context (compress game history)
Call Bedrock for Harry's decision
Validate Harry's response (is the action format correct? Is it a legal move?)
Execute Harry's action (update game board, resolve signals)
Compute Voldemort's context
Call Bedrock for Voldemort's decision (or use heuristic)
Validate Voldemort's response
Execute Voldemort's action (relocate Horcrux? Plant decoy?)
Update shared game state (locations, signals, scores)
Persist to DynamoDB + emit CloudWatch metrics

50 turns × 12 operations = 600 operations per game. The sequential nature is the real bottleneck. You can't parallelize Harry and Voldemort because Voldemort's action depends on what Harry just did.

Failing Slow, Failing Expensive

Here's what I noticed when I analyzed my retry rate: 15% of LLM responses failed validation. Harry would produce an invalid action format, or try to search a location on cooldown, or attempt to use an ally ability he'd already spent.

Each failure: 3 seconds of reasoning → rejected → full context replayed → try again. Each retry cost full token price.

The system was discovering failures after spending money. Failing slow (3 seconds into inference). Failing expensive (full token cost per failure). I'd later name the opposite of this pattern.

The Cost Math That Scared Me

Per game: ~$1.95 (naive approach)
1,000 games/day: $1,950/day
Monthly: ~$49,000
Annual: ~$588,000

For a game. A two-agent game where Harry chases Horcruxes.

The cost breakdown:

Bedrock (LLM calls): 67% — $1.31/game
Lambda (compute): 26% — $0.51/game
DynamoDB (state): 7% — $0.13/game

Now imagine deploying this as an always-on interactive experience. Imagine scaling to 1,000 simultaneous games. The numbers get terrifying fast.

The Four Fixes: An Optimization Ladder

I didn't need fewer agents. I needed fewer expensive decisions. Here's the optimization ladder pushing decisions DOWN from the costly LLM layer to cheaper alternatives.

Fix 1: Bound the Problem (Constraint Pruning)

Before: Harry saw 90 possible actions per turn (15 locations × 6 action types: search, attack, use_ally, investigate, fortify, retreat) and the LLM had to reason about which were valid.

After: A constraint solver prunes invalid actions before Harry's LLM sees them:

def get_valid_actions(game_state, agent="harry"):
    actions = []
    for loc in game_state.locations:
        if game_state.cooldown[loc] == 0:         # not on cooldown
            if game_state.usage_count[loc] < MAX_USES:  # not exhausted
                if game_state.budget > 0:           # has action budget
                    actions.append(loc)
    return actions  # typically 2-4 options, not 90

Result: 90 options → 2-4 valid options. Harry's LLM never wastes tokens reasoning about locations on cooldown or abilities already spent. 80% reduction in invalid reasoning tokens.

This is where I coined the principle: "Fail Fast, Fail Free."

The constraint solver catches bad decisions before they touch the LLM. An invalid action rejected by a 0.2ms Python function costs nothing. The same invalid action reasoned about by Claude for 3 seconds costs tokens, latency, and often a retry when post-inference validation fails.

Fail fast = catch it early. Fail free = catch it before the meter starts running.

Fix 2: Replace Tokens with Math (Bayesian Inference)

Before: Harry re-read 50 turns of narrative history (5,000 tokens) to figure out "where is the Horcrux probably located?"

Turn 1: Harry searched Hogwarts → negative signal
Turn 2: Harry searched Diagon Alley → positive signal  
Turn 3: Harry attacked Diagon Alley → decoy!
Turn 4: Voldemort relocated something...
... (50 turns of this = 5,000 tokens)

After: A Bayesian belief map computes probabilities outside the LLM:

class HorcruxBeliefMap:
    def __init__(self, locations):
        n = len(locations)
        self.beliefs = {loc: 1.0/n for loc in locations}  # uniform prior

    def update_on_signal(self, loc, signal):
        if signal == "positive":
            self.beliefs[loc] *= 3.0   # 3x more likely here
        elif signal == "negative":
            self.beliefs[loc] *= 0.1   # 10x less likely
        elif signal == "destroyed":
            self.beliefs[loc] = 0.0    # confirmed eliminated
        self.normalize()

    def top_targets(self, n=3):
        sorted_locs = sorted(self.beliefs.items(), key=lambda x: -x[1])
        return sorted_locs[:n]

What Harry's LLM actually sees: "top_target: Hogwarts (p=0.34), Azkaban (p=0.22)" = 15 tokens, not 5,000.

Result: 97% context reduction. Cost drops from $0.015 to effectively $0 for the belief computation. And Harry makes better decisions because probabilities are more precise than narrative intuition.

Another face of "Fail Fast, Fail Free" is if you can compute the answer with math ($0), why would you pay an LLM ($0.015) to infer it from narrative text?

Fix 3: Skip the LLM Call (Heuristic Decision Trees)

Not every Voldemort decision needs a $200B parameter model. Some game situations have obvious optimal moves:

def voldemort_decide(game_state):
    # If Harry is one move from a real Horcrux → relocate (obvious)
    if game_state.harry_adjacent_to_horcrux():
        return RelocateAction(game_state.threatened_horcrux)

    # If decoy budget available and Harry is confident → disrupt
    if game_state.decoy_budget > 0 and harry_entropy(game_state) < 1.5:
        return PlantDecoyAction(game_state.harry_top_target)

    # If early game with high uncertainty → do nothing (save budget)
    if game_state.turn < 10 and harry_entropy(game_state) > 3.0:
        return WaitAction()

    # Only genuinely complex situations need LLM
    return call_llm_for_strategy(game_state)

Result: 60% of Voldemort's decisions handled with zero LLM cost. Only the genuinely strategic moments where multiple valid strategies compete, justify an inference call.

Fix 4: Isolate the Expensive Layer (Architecture)

The most impactful change was structural. I redesigned Horcrux Hunt so only 2 of 8 modules ever touch the LLM:

FREE MODULES (6):
├── Game Engine         (rules, turn management)
├── Constraint Solver   (valid action computation)
├── Belief Manager      (Bayesian probability updates)
├── State Persistence   (DynamoDB read/write)
├── Validation Layer    (response format checking)
└── Metrics & Logging   (CloudWatch, dashboards)

EXPENSIVE MODULES (2):
├── Harry Strategic Layer    (genuinely uncertain decisions)
└── Voldemort Strategic Layer (only when heuristics can't decide)

The Interface Boundary compresses context at the border between free and expensive:

Input to LLM: 2,000+ tokens of raw game state → compressed to 55 tokens of AgentContext
Output from LLM: Validated immediately, rejected for free if invalid

This is "Fail Fast, Fail Free" as architecture: clear cost boundaries. Validation happens in the free zone. If something is going to fail, it fails in one of the 6 free modules, but never in the 2 expensive ones.

The Results

Same game. Same Harry. Same Voldemort. Same Claude Sonnet model. Radically different architecture:

Metric	Before	After	Change
Cost per game	$1.95	$0.35	-82%
Latency per turn	12s	3s	-75%
Harry win rate	23%	52%	+29pp
Timeout rate	18%	<3%	-83%
Retry rate	15%	<3%	-80%
Annual cost at scale	$588K	$102K	-$486K

Harry wins more, not because the model is smarter, but because he's reasoning over focused, relevant information instead of drowning in 6,000 tokens of noise. The audience sees 3-second turns instead of 12-second waits. The game is actually fun to watch now.

The Lesson

The answer to "How many agents are too many?" isn't a number. It's a question:

"Is this decision worth an LLM call?"

Most of Voldemort's decisions weren't. Most of Harry's probability calculations weren't. Most of the validation logic wasn't. Once I stopped paying the LLM to discover things I already knew, the costs fell off a cliff.

The Optimization Ladder:

Can a rule handle this? (Free — constraints, cooldowns, budgets)
Can a heuristic handle this? (Nearly free — if/then game logic)
Can math handle this? (Cheap — Bayesian updates, entropy)
Does this genuinely need an LLM? (Expensive — but justified for true uncertainty)

Push every decision as far DOWN that ladder as it can go. Fail fast, fail free at every layer.

🚀 What's Next

The bill is under control. But Harry's still losing.

He searches the same location twice. He forgets signals from 3 turns ago. He contradicts his own belief map. The problem isn't cost anymore, it's memory masquerading as reasoning failure.

→ Part 0: Fail Fast, Fail Free

Read this blog to know more about the principle.

→ Part 2: Why Your Agent Forgets where I discover that 68% of Harry's "reasoning failures" are actually retrieval failures, and the fix is entropy math, not bigger models.

Spoiler: I compressed Harry's context from 12,000 tokens to 340. Same LLM. Same prompts. Completely different agent.

💬 Quick diagnostic for your agent:

Run the same input 5 times. Does your agent give the same output every time?

✅ Yes → You probably have a cost or integration problem (Part 1 or 3)
❌ No → You definitely have a memory problem (next post)
🤷 Haven't checked → ...that's the scariest answer of all

Drop your answer below. I'll tell you exactly which post in this series has your fix. 👇

🔖 Follow me to get Part 2 when it drops or keep refreshing your inference bill until you're motivated enough to read it. Your choice. 💸

I am a Gen AI Developer Advocate & Architect. I built a multi-agent AI game to entertain a conference audience and accidentally created the most expensive stress test for multi-agent systems I'd ever seen. So. I adapted the classic "Fail Safe" and came up with "Fail Fast, Fail Free" after that $1,847 weekend taught me that the most expensive failure is one you discover too late.

Are you building AI Agents? Then you need to look at this design principle!!

Anannya Roy Chowdhury — Mon, 13 Jul 2026 05:53:53 +0000

Anannya Roy Chowdhury for AWS

Jun 30

"Fail Fast, Fail Free : The Design principle my multi-agent game was missing"

#ai #aws #agents #systemdesign

10 min read

I Built a Multi-Agent Game. One Missing Design Principle Broke Everything. Read about this below.

Anannya Roy Chowdhury — Thu, 02 Jul 2026 02:06:58 +0000

Anannya Roy Chowdhury for AWS

Jun 30

"Fail Fast, Fail Free : The Design principle my multi-agent game was missing"

#ai #aws #agents #systemdesign

10 min read

[Boost]

Anannya Roy Chowdhury — Wed, 01 Jul 2026 01:03:46 +0000

Anannya Roy Chowdhury for AWS

Jun 30

"Fail Fast, Fail Free : The Design principle my multi-agent game was missing"

#ai #aws #agents #systemdesign

10 min read

"Fail Fast, Fail Free : The Design principle my multi-agent AI was missing"

Anannya Roy Chowdhury — Tue, 30 Jun 2026 04:18:53 +0000

This is an intro to "Multi-Agent Systems in Production: What They Don't Tell You" — a four-part series based on a game I built for my conference talks at AI Engineer Week, Conf42 LLM, AgentCon Bengaluru, and R/pharma GenAI. This introductory post defines the unifying principle behind everything that follows.

The Most Expensive Bug I Ever Shipped

The bug wasn't in my code. The logic was correct. The prompts were good. The model was state-of-the-art.

The bug was where my system failed.

I built a multi-agent interactive game called "Horcrux Hunt" where two AI agents (Harry and Voldemort) battle live in front of an audience. Harry (Claude on Amazon Bedrock, Strands SDK) hunts Horcruxes hidden across 15 locations. Voldemort (heuristic-first adversary with LLM fallback) relocates them, plants decoys, and corrupts Harry's beliefs. The audience watches on a Streamlit dashboard as the hunt unfolds in real time.

And then we ran it. One weekend event. $1,847 in AWS bills. 12-second latency per turn. Audience waiting. Harry losing 77% of the time.

When I dissected the failure, I found the same pattern everywhere:

The LLM, during Harry's move, reasoned about 90 possible actions. 86 were invalid. It spent 3 seconds and 2,000 tokens discovering what a 0.2ms constraint check could have told it for free.
The Harry agent retrieved 5,000 tokens of history to make a decision. A 55-token probability score contained the same information. But we loaded the full context first and compressed later — paying before checking.
A tool call with invalid parameters hit the API, got a 400 error, retried twice. Client-side validation would have caught it in <1ms, before any call or game action was wasted.
I added Hermione, Ron, and Dumbledore agents** to help Harry. These three agents independently queried the same guidelines, produced conflicting strategies, and Harry's win rate dropped from 61% to 34%. A single priority check before execution would have caught it for free.

Every expensive failure had the same shape: the system knew it would fail, but discovered this too late. After tokens were spent, latency was burned, compute was consumed, and turns were wasted.

I started calling this pattern "failing slow, failing expensive." And its opposite became my design principle:

Fail Fast, Fail Free.

If a decision is going to fail, make it fail before it costs you anything.

That's it. That's the principle.

Fail fast = catch it at the earliest possible checkpoint
Fail free = catch it before the expensive meter starts running

In the Horcrux hunt game, the "meter" is different depending on context:

In cost terms: an LLM call ($0.008-0.015 per failure) vs $0 for a constraint check resolving an invalid action
In latency terms: a 3-second inference call for an action the game rejects anyway vs a 0.2ms validation
In game terms: Harry wasting a turn on a cooldown location vs knowing instantly it's unavailable
In coordination terms: Four agents arguing for 9 seconds vs Harry deciding alone when entropy is low
In reliability terms: a retry loop burning tokens vs a pre-validated clean call

The principle asks one question of every failure in your system: Could this have been caught earlier, cheaper, or both?

Almost always, the answer is yes.

The Anatomy of a Free Failure

What does a "free failure" actually look like? Here's the pattern for the game:

# EXPENSIVE failure (traditional):
# 1. Load full game history and Build full context (500ms, 2000 tokens)
# 2. Call LLM for decision (3000ms, $0.008)
# 3. Parse response (50ms) DETECTED HERE
# 4. Retry from step 1 (another $0.008)
# Total cost of failure: $0.016 + 3.5 seconds

# FREE failure (fail fast, fail free):
# 1. Validate input ← FAILURE DETECTED HERE (0.2ms, $0)
# 2. (never reaches LLM)
# Total cost of failure: $0 + 0.2ms

The key insight: validation is nearly free. Inference is expensive. Move the checkpoint upstream.

This isn't just "input validation" in the traditional software engineering sense. In multi-agent production systems, there are multiple layers where you can catch failures before they become expensive:

Layer 1: Constraint check     →  "Is this action even valid?"     → 0.2ms, $0
Layer 2: Entropy check        →  "Does this need LLM reasoning?"  → 0.5ms, $0
Layer 3: Schema validation    →  "Are these parameters correct?"  → 0.3ms, $0
Layer 4: Safety gate          →  "Is this output safe?"           → 1ms, $0
Layer 5: Priority resolution  →  "Do agents agree?"               → 2ms, $0
─────────────────────────────────────────────────────────────────────────────
Layer 6: LLM inference        →  "What should I do?"              → 3000ms, $0.008
Layer 7: API call             →  "Execute the action"             → 500ms, variable
Layer 8: Retry                →  "Try again"                      → 3500ms, $0.008+

Layers 1-5 are free. Layers 6-8 are expensive. Every failure you catch in Layers 1-5 is a failure that never reaches Layers 6-8. That's "fail fast, fail free."

Why This Matters Specifically for Multi-Agent Systems

In a single-agent system, a failure costs you one LLM call. Annoying but survivable.

In a multi-agent system, failures compound:

1 agent  failing = 1 retry × 1 inference cost
3 agents failing = retries × context replay × coordination overhead × cascading delays

When Harry produces invalid output, Voldemort receives it, reasons about it (paying tokens), produces its own output based on garbage, Executor Agent receives THAT... by the time you detect the failure, you've paid three inference calls, contaminated shared state, and need to rewind everything.

In multi-agent systems, a failure that isn't caught early becomes a failure that multiplies. This is why "fail fast, fail free" isn't just a nice optimization. It's architecturally critical.

The cost of late detection in multi-agent systems:

Where failure is caught	Cost in single-agent	Cost in 3-agent system
Before LLM call (Layer 1-5)	$0	$0
After 1 LLM call (Layer 6)	$0.008	$0.008
After cascading to other agents	$0.008	$0.024 + state rollback
After reaching the user	$0.008	Incalculable

The multiplication factor is why "fail fast, fail free" becomes an architectural principle for my multi-agent game and other production AI systems, not just a coding best practice.

The Four Faces of Fail Fast, Fail Free

This principle shows up differently depending on which failure mode you're facing. Here's a preview of how it manifests across the four parts of this series:

🔥 Cost: Prune Before Reasoning (Part 1)

The LLM doesn't need to reason about invalid options.

# Fail fast: constraint solver runs BEFORE LLM
valid_actions = constraint_solver(game_state)  # 0.2ms, $0
# 90 options → 4 valid actions
# The LLM never sees the 86 invalid ones
# 86 failures caught for free

If 86 of Harry's 90 possible actions are invalid (exhausted location. spent powers), letting the LLM discover this wastes 95% of its reasoning budget. A constraint solver makes those 86 failures free, they never reach the meter.

The mantra: Don't let the LLM think about things you already know the answer to.

🧠 Memory: Gate Before Retrieving (Part 2)

Not every decision deserves full context retrieval, in my case the full 5000 tokens as history for Harry's next move.

# Fail fast: entropy check BEFORE retrieval
entropy = calculate_entropy(belief_map)
if entropy < 1.0:
    # Harry already knows where the Horcrux is
    return heuristic_decision()  # 0 tokens, $0
# Only uncertain decisions justify context retrieval cost

When entropy is low (the agent, using the bayesian belief map, is already confident of a move), sending context of 50 turns to the LLM is waste. The entropy check is a fail-fast gate: "Do I even need to spend tokens on this decision?" 60% of the time, the answer is no. Those decisions become free.

The mantra: Check whether you need to think before you start thinking.

🔌 Integration: Validate Before Calling (Part 3)

Client-side schema validation catches bad parameters for free.

# Fail fast: JSON Schema validation BEFORE API call
errors = jsonschema.validate(params, tool_schema)  # <1ms, $0
if errors:
    return fix_params(errors)  # self-correct without any call
# Only valid calls reach the API

A classic example of my game validation:

# Fail fast: schema validation BEFORE game action executes
errors = validate_tool_call("search_location", {"location": "hogwarts"})
if game_state.cooldown["hogwarts"] > 0:
    return ToolError("Hogwarts on cooldown for 2 turns")  # <1ms, $0
# Only valid, available actions consume game budget

When Harry tries to search a location on cooldown (from Game Theory - a mechanism that restricts immediate retaliation or repeated actions), catching it at validation (free, <1ms) is infinitely better than catching it after an LLM inference + game execution + failure + retry. So what's better than to use MCP here. MCP's killer feature isn't the protocol itself — it's that schema contracts between server and client enable free validation. Every parameter error caught in <1ms is a retry that never happens. At 2.3 retries per request (our pre-MCP baseline), this is massive: 91% reduction in retries, purely by moving the failure checkpoint upstream.

The mantra: The cheapest API call is the one you never make.

🏥 Coordination: Veto Before Executing (Part 4)

In regulated systems, unsafe responses must fail at review, not at the execution step. For example, in my horcrux game, when Hermione and Dumbledore disagree, we need to resolve it before Harry acts.

# Fail fast: priority resolution BEFORE team executes
if hermione.recommends("attack_azkaban") and dumbledore.warns("trap_detected"):
    # Priority: Dumbledore's safety assessment OVERRIDES Hermione's analysis
    return harry_defend()  # resolved in <2ms, no cascading confusion
# Only aligned, conflict-free strategies reach execution

When the Safety analysis vetoes an unsafe action, that "failure" is free and is a <2ms activity. The alternative (delivering an unsafe action using tokens and multiple retries) is infinitely expensive. So, "fail fast, fail free" becomes "validate early, harm never."

The mantra: The safest failure is the one that never reaches the executor.

The Optimization Ladder (Reframed)

Here, I'll introduce the "Optimization Ladder" — a framework for pushing decisions down from expensive layers to cheap ones.

Reframed through "Fail Fast, Fail Free," it becomes a failure checkpoint ladder:

CHEAPEST (try first):
├── Rules & Constraints     → Can I rule this out for free?
├── Heuristics              → Is the answer obvious?
├── Math & Statistics       → Can I compute instead of infer?
├── Compressed Inference    → Can I think with less context?
MOST EXPENSIVE (last resort):
└── Full LLM Reasoning      → Only genuinely uncertain decisions

Each layer is a checkpoint. Each checkpoint catches failures before they cascade to the layer below. The system only pays for inference on decisions that survive every free checkpoint above which turns out to be about 20-40% of turns.

The other 60-80%? Free. In my game, Harry acts on constraints, entropy gates, heuristics, and math. All at zero token cost. And counterintuitively, his win rate improved because less noise = better decisions.

How to Apply This Tomorrow

You don't need to redesign your system. Start with one question:

"Where in my pipeline do I first discover that something is wrong?"

Then ask: "Could I have discovered that one step earlier?"

Repeat until the answer is "no" or "the failure checkpoint is already free."

Practical starting points:

Add input validation before every LLM call. What percentage of your prompts contain information that makes the answer predetermined? What percentage of your agent's reasoning leads to invalid actions? That's your "free failure" opportunity.
Add an entropy/confidence check before retrieval. How often does your agent retrieve context it doesn't need? That's wasted tokens.
Add schema validation before every tool call. What's your retry rate? Each retry = full token cost. Multiply that by your average token cost. That's what free validation saves you.
Add a safety/priority check before every multi-agent output. How often do your agents disagree? Each disagreement caught at orchestration is a contradiction that never reaches the user.

The Series Roadmap

This blog defines the principle. The next four show it in action — all through the lens of building, breaking, and fixing Horcrux Hunt:

Part	Problem	"Fail Fast, Fail Free" Manifestation
Part 1: Cost	$1,847 bill for a weekend game, 12s latency, 23% win rate.	Prune invalid actions BEFORE inference
Part 2: Memory	77% failure rate, perfect reasoning	Gate retrieval by entropy BEFORE loading context
Part 3: Integration	Wrong tool, wrong move	Validate parameters BEFORE making API calls
Part 4: Coordination	Added 3 agents. They Fight. Win rate DROPPED to 34%.	Safety veto BEFORE delivering output

Each part tells a story, shows the failure, explains the fix, and proves the results. But now you know the common thread: every fix is a version of the same principle applied at a different layer.

One More Thing

There's a beautiful symmetry here. "Fail fast, fail free" has existed in software engineering for decades — circuit breakers, input validation, type systems, contract testing. We know this principle.

But somewhere in the excitement of LLMs, we forgot it. We started building systems where the first line of defense is a $200-billion-parameter model. We made inference the validator instead of the validated. We let Harry reason about every possibility instead of telling him which possibilities were already impossible.

Multi-agent systems make this mistake catastrophically expensive because failures compound across agents. But the fix is the same fix we've always known:

Don't let expensive things discover what cheap things already know.

In my Horcrux Hunt game terms:

Don't let Harry reason about locations on cooldown (constraints know this)
Don't let Harry retrieve history when he's already confident (entropy knows this)
Don't let Harry attempt actions with invalid parameters (validation knows this)
Don't let the team argue when priority rules are clear (the mediator knows this)

Check before you call. Validate before you execute. Prune before you reason. Gate before you retrieve. Veto before you deliver.

Fail fast. Fail free.

🚀 What's Next

Harry spent $1,847 learning this lesson in one weekend. You can learn it for free...

→ Part 1: The $1,847 Weekend where the game goes live, the bill arrives, and I discover that 86 of 90 actions Harry reasoned about were already impossible (releasing soon).

If you've ever watched your agent burn tokens on decisions a Python function could have handled, this one's for you.

💬 I'm curious — what's your agent's retry rate right now?

Drop it in the comments. If it's above 5%, you're probably failing slow and failing expensive somewhere in your pipeline. I'll reply with which Part (1-4) has your fix.

🔖 Bookmark this series if you're building agents in production — each post drops one principle that saved me $576K/year in inference costs. Or just watch your monthly bill and you'll know when you need them.* 😏

I am a GenAI Developer Advocate and Architect. I adapted the classic 'Fail safe' principle into what I call 'Fail Fast, Fail Free' after spending too much money on multi-agent systems that discovered their failures too late. I am now on a mission to make every failure in all my systems free or at least cheaper than my rent.

Read how Strands SDK can help you build AI Agents with practically low code!

Anannya Roy Chowdhury — Mon, 01 Jun 2026 11:48:21 +0000

Anannya Roy Chowdhury

May 27

I Let an AI Plan My Vacation (Here's What It Built)

#ai #aws #machinelearning #python

6 min read

I Let an AI Plan My Vacation (Here's What It Built)

Anannya Roy Chowdhury — Wed, 27 May 2026 14:45:00 +0000

I Had 47 Browser Tabs Open. So I Built an AI Travel Agent with Strands SDK + Amazon Bedrock ✈️

Okay, so real talk.

I had 47 browser tabs open.

47.....

Flights on a Flights App. Hotels on a Booking website. Weather on some sketchy website that was 40% ads. A spreadsheet I made at 11 PM that somehow had three separate columns all labeled "maybe hotel??" and one row that just said:

"check vibes"

All I wanted was to plan a week-long trip to Thailand.

And then — because I am a developer, and developers solve problems with code even when they absolutely should not — I closed all 47 tabs, opened Kiro's spec-driven mode, and said:

"Fine. I'll build the travel agent myself with Strands SDK."

Two hours later, I typed one sentence into my chat window and got back:

Flights
Hotels
Weather warnings
A complete 7-day itinerary
Budget breakdowns
Packing recommendations
VISA options

And honestly?

I haven’t touched that spreadsheet since.

So let’s build the thing. 👇

Wait… What Even Is Strands SDK?

Before we dive in:

Strands SDK is AWS’s open-source Python framework for building AI agents.

Think of it like this:

You write Python functions (tools)
Add a @tool decorator
Connect it to a model on Amazon Bedrock
The model decides:
- which tools to call
- in what order
- with what inputs

That’s… basically it.

No YAML facades.
No 400-page orchestration configs.
No dependency graphs that look like a subway map.

Just Python functions and an LLM smart enough to use them.

Here’s the mental model that helped me:

Your Agent = A Smart Intern
Your Tools = What the Intern Can Do
Your Prompt = The Instructions
Bedrock = The Intern's Brain

Except this intern:

never misses deadlines

10/10 would hire again.

What We’re Building

Our AI travel planner should:

✈️ Find flights
🏨 Find hotels
🌦️ Check weather
📋 Check Budgets and Build a full itinerary

One agent.
Four tools.
One prompt.

Done.

Architecture Overview

Stack Used

Strands SDK → Agent framework
Amazon Bedrock → Claude 3.5 Sonnet
AWS Lambda → Tool execution
Secrets Manager → API keys
S3 → Save itineraries
CloudWatch → Logging & observability

Yes, this is production-friendly.

We’ll get there.

Step 0 — Setup

Install dependencies:

pip install strands-agents strands-agents-tools boto3

Enable Claude 3.5 Sonnet inside Amazon Bedrock.

Takes ~2 minutes.

🚨 Please Don’t Hardcode API Keys

I know.

You’re “just testing.”

That API key will end up in a public GitHub repo eventually.

And then you will land up with a bill that burns your wallet.

Use AWS Secrets Manager instead.

import boto3
import json

def get_secret(secret_name: str) -> str:
    """Fetch API keys securely from AWS Secrets Manager."""

    client = boto3.client(
        "secretsmanager",
        region_name="us-east-1"
    )

    response = client.get_secret_value(
        SecretId=secret_name
    )

    return json.loads(
        response["SecretString"]
    )["api_key"]

One function call.
Sleep peacefully.

Tool #1 — Search Flights ✈️

One thing that blew my mind about Strands:

Your docstring is part of the prompt.

Not for developers. For the model.

The LLM reads your tool descriptions to decide:

when to call them
how to call them
what arguments to pass

Good docstrings = smart agents.
Bad docstrings = chaos.

from strands import tool

@tool
def search_flights(
    origin: str,
    destination: str,
    departure_date: str,
    return_date: str,
    num_passengers: int
) -> dict:
    """
    Search for available flights between cities.

    Returns:
    - airline
    - price
    - duration
    - stops
    """

    api_key = get_secret(
        "travel-agent/flight-api-key"
    )

    return {
        "flights": [
            {
                "airline": "Thai Airways",
                "price_per_person": 510,
                "stops": 0,
                "vibe": "Direct. Fancy. Worth it."
            },
            {
                "airline": "Indigo",
                "price_per_person": 440,
                "stops": 1,
                "vibe": "One stop. Still emotionally stable."
            }
        ]
    }

Yes, I added a "vibe" field.

No, the model doesn’t need it.

Yes, it made me happy and my decision making, faster....

Tool #2 — Hotels That Aren’t Secretly Hostels 🏨

You know what I hate?

The hotel listing that says:

"$89/night!"

And then somehow becomes:

$89 base price
taxes
resort fee
“service charge”
“convenience fee”
emotional damage

My tool doesn’t do that.

@tool
def search_hotels(
    destination: str,
    check_in: str,
    check_out: str,
    budget_per_night: float
) -> dict:
    """
    Search hotels within budget.
    """

    return {
        "hotels": [
            {
                "name": "Veranda Resort",
                "price_per_night": $155,
                "rating": 4.8
            },
            {
                "name": "Pacific Club Hotel",
                "price_per_night": $120,
                "rating": 4.5
            }
        ],
        "note": "No hidden fees. Unlike some websites."
    }

Tool #3 — Weather Check 🌧️

Because packing goggles for Northern lights was a character-building experience I never want again.

@tool
def get_weather(
    destination: str,
    travel_month: str
) -> dict:
    """
    Get weather info and packing advice.
    """

    return {
        "avg_temp_celsius": 26,
        "conditions": "Mild with dry heat",
        "packing_must_haves": [
            "Light coloured t-shirts",
            "beach flip flops",
            "Umbrella",
            "sunscreen",
        ]
    }

Tiny tool. Massive usefulness.

Tool #4 — Build the Itinerary 📋

This is where everything comes together.

Flights ✅
Hotels ✅
Weather ✅

Now the agent can finally build a realistic itinerary.

And yes — we save it to S3.

Because my Notes app cannot be trusted.

from datetime import datetime

@tool
def build_itinerary(
    destination: str,
    duration_days: int,
    interests: list,
    budget_remaining: float
) -> dict:
    """
    Build a day-by-day itinerary.
    """

    itinerary = {
        "destination": destination,
        "days": duration_days,
        "interests": interests,
        "daily_plan": [
            {
                "day": 1,
                "theme": "Land and Eat Immediately"
            },
            {
                "day": 2,
                "theme": "Markets and Sunset Beach Walks"
            }
        ]
    }

    s3 = boto3.client("s3")

    s3.put_object(
        Bucket="travel-itineraries",
        Key="thai-trip.json",
        Body=json.dumps(itinerary)
    )

    return itinerary

Assemble the Agent

Now the fun part.

from strands import Agent
from strands.models import BedrockModel

model = BedrockModel(
    model_id="anthropic.claude-3-5-sonnet",
    region_name="us-east-1",
    temperature=0.3
)

SYSTEM_PROMPT = """
You are a smart AI travel planner.

Always:
1. Search flights first
2. Search hotels second
3. Check weather
4. Build itinerary for given budget last
"""

travel_agent = Agent(
    model=model,
    system_prompt=SYSTEM_PROMPT,
    tools=[
        search_flights,
        search_hotels,
        get_weather,
        build_itinerary
    ]
)

response = travel_agent(
    """
    Plan a 5-day Thailand trip for 2 people under $3000.
    """
)

print(response)

Terminal Output

[Tool Call] search_flights
✓ Thai Airways selected

[Tool Call] search_hotels
✓ Veranda Resort selected

[Tool Call] get_weather
✓ Mild weather detected

[Tool Call] build_itinerary
✓ Saved to S3

━━━━━━━━━━━━━━━━━━
YOUR THAILAND TRIP
━━━━━━━━━━━━━━━━━━

Flights:   $1020
Hotel:     $775
Activities: $1200

TOTAL:     $2995 ✅

That ✅ Under budget hit different.

Production Lessons (Learned the Hard Way)

Here’s the honest truth:

Your local demo is not your production system.

Things that will hurt you eventually:

Problem	Fix
Hardcoded keys	Secrets Manager
Infinite tool loops	Max tool-call limits
No observability	CloudWatch logging
Surprise Bedrock bill	`max_tokens` + alarms
Zero error handling	`try/except` everywhere
Unsafe prompts	Bedrock Guardrails

Add Logging with CloudWatch

import logging
from strands.handlers import CallbackHandler

logger = logging.getLogger("travel-agent")

class LoggingHandler(CallbackHandler):

    def on_tool_call(
        self,
        tool_name,
        tool_input
    ):
        logger.info(
            f"Calling {tool_name}"
        )

    def on_error(self, error):
        logger.error(str(error))

This is the difference between:

“cool demo”
and “actual software”

My Biggest Takeaways

1. Your Docstrings Matter More Than You Think

The model uses them as instructions.

Treat them seriously.

2. Tool Order Matters

Without ordering rules, my agent built itineraries before knowing hotel costs.

Beautiful plans. Completely wrong budget.

3. Lower Temperature = Better Planning

High temperature is great for creativity.

Not for budgeting.

4. Persist Everything

S3. DynamoDB. Databases.

Conversation memory disappears eventually.

Your data shouldn’t.

5. The Demo → Production Gap Is Real

Prototypes and demos are magical.

Production systems are where reality shows up with a baseball bat.

Build responsibly.

FAQ

Do I Need a Strands Certification?

Nope.

Just:

Python
Bedrock access
Functions

That’s enough to start.

Can I Use Other Models?

Absolutely.

Swap Claude for:

Llama
Mistral
Titan
Anything available on Bedrock

My Agent Calls Tools in Weird Orders

Fix your system prompt.

Be explicit.

LLMs love explicit instructions.

What’s Next?

You can extend this into:

🧠 Memory with DynamoDB
💬 Slack bots
🤖 Multi-agent orchestration
🏭 Bedrock AgentCore deployments

That’s where things get really interesting.

I started this journey with:

47 browser tabs
one broken spreadsheet
and travel-planning rage

I ended it with:

a functioning AI travel agent
a Thailand itinerary
and a permanently closed spreadsheet

Honestly?

Worth it.

If you enjoyed this post:

❤️ Drop a reaction
🔁 Share it with another developer
👀 Follow for more AI agent tutorials

And if you build something cool with Strands SDK — I genuinely want to see it. Hit the comments box or reach out over LinkedIn. I work for AWS and love building and exchanging views on AI!

Happy building. ✈️

Why your Production Retreival-Augmented-Generation (RAG) is failing and how to fix it?

Anannya Roy Chowdhury — Tue, 17 Mar 2026 05:41:19 +0000

Why Most RAG Systems Fail in Production — and How Developers Can Fix It

Over the past 2-3 years, many developers have built Retrieval-Augmented Generation (RAG) applications.

The typical journey looks something like this:

Step 1 - Connect a Vector Database
Step 2 - Index documents
Step 3 - Send retrieved context to an LLM
Step 4 - Ship a chatbot

At first, everything works. But once the system reaches real users, the issues start appearing.

The assistant retrieves irrelevant documents
Answers sometimes hallucinate
Latency increases as the knowledge base grows
The system becomes expensive to run

If this sounds familiar, you’re not alone. Many RAG systems struggle when they move from prototype to production.

The interesting part is that the problem usually isn’t the language model. It’s the retrieval architecture. Let’s break down what’s actually happening and how you can improve it.

The “Simple” RAG Architecture

Most tutorials introduce RAG using a simple pipeline.

User Query
   ↓
Vector Search
   ↓
Top n Documents
   ↓
LLM Prompt
   ↓
Generated Answer

This approach is great for learning the concept. But production workloads quickly expose some weaknesses.

Problem 1: Vector Search Isn’t Always Enough

Vector similarity works well for semantic matching, but real-world queries are messy. A developer might ask:

“How do I rotate API credentials without downtime?”

A pure vector search might retrieve documents related to:

API authentication
security guidelines
credential policies

All of them sound relevant. But none of them may actually contain the exact steps needed to answer the question.

The result? The LLM tries to generate an answer anyway. This is where hallucinations often begin.

Problem 2: Document Chunking Breaks Meaning

Another hidden challenge is how documents are split before indexing.

Many pipelines use fixed chunk sizes, such as 500 or 1000 tokens. But technical documentation often contains structured sections such as setup instructions, configuration steps, troubleshooting guides, Dos and Don'ts. When these sections are split incorrectly, the retrieval system might return only part of the information needed.

For example:

Chunk A → explains the problem
Chunk B → shows the fix
Chunk C → provides the command

If the model receives only Chunk A, it lacks the context needed to answer correctly.

Problem 3: Real Questions Require Multiple Documents

Users often ask questions that require combining information from several sources.

For example:

“Does this authentication method support multi-region failover?”

The answer may exist across multiple documents - authentication documentation, networking architecture guides, availability recommendations

A simple RAG pipeline retrieves only a few chunks, which may not capture the full picture. This makes it difficult for the LLM to produce a reliable answer.

What Production RAG Systems Do Differently

When teams build reliable RAG systems, they usually add additional layers to the retrieval pipeline. Instead of relying on a single search method, they combine multiple techniques.

A more robust architecture might look like this:

User Query
   ↓
Query Understanding
   ↓
Hybrid Retrieval (Vector + Keyword)
   ↓
Reranking
   ↓
Context Assembly
   ↓
LLM Generation

Each layer helps solve a specific problem.

Query understanding improves recall by rewriting or expanding queries.

Hybrid retrieval combines semantic similarity with keyword matching.

Reranking ensures the most relevant documents appear at the top.

Context assembly structures the prompt so the LLM receives coherent information.

Together, these improvements dramatically increase answer reliability.

How Developers Can Build Better RAG Systems

Once your RAG system starts handling real workloads, infrastructure becomes just as important as model choice. Production applications must handle - large document collections, high query volumes, strict latency requirements.

This is where services from Amazon Web Services can simplify the architecture.

For example:

Amazon OpenSearch Service - Supports hybrid search, allowing you to combine vector similarity with keyword search in the same system.

Amazon Bedrock - Provides access to foundation models without managing infrastructure, making it easier to experiment with different models.

AWS Lambda - Helps orchestrate lightweight retrieval pipelines, enabling query preprocessing and reranking logic.

Amazon S3 - Acts as a scalable document store for large knowledge bases and embedding pipelines.

By combining these services, developers can focus on improving retrieval logic instead of managing infrastructure.

Key Takeaways for Developers

If you’re building a RAG application, here are a few practical lessons that can save time:

Don’t rely on vector search alone — combine semantic and keyword retrieval.
Chunk documents intelligently — align chunks with semantic structure.
Introduce reranking — it often improves answer accuracy significantly.
Treat RAG as a search problem — not just an LLM problem.

Most importantly, remember that a reliable GenAI system is built across multiple layers: retrieval, augmentation, orchestration, and generation.

What We’ll Explore Next

In this post, we looked at why many RAG systems fail when deployed in production.

In the next article, we’ll walk through a step-by-step architecture for building a production-grade RAG system on AWS, including:

hybrid retrieval pipelines
document reranking strategies
latency optimization techniques
cost-efficient GenAI architectures

If you're building GenAI applications today, understanding these patterns can make the difference between a prototype and a system that developers actually trust.