Death by Loop: How One Agent Burned $23,000 While Its Creator Slept Like a Baby

The Number That Woke Me Up

$23,041.67. Down to the cent.

That's what appeared on a client's AWS bill — extra charges generated by a single agent over 7 hours.

It wasn't hacked. No malicious code. No security breach.

It just... looped.

A perfect, elegant, completely-reasonable-looking loop.

And its creator was asleep.

---

You've Seen This Before

If you've deployed AI agents in production, these might sound familiar:

• Your agent says "let me double-check" — and checks 300 times
• The same operation repeats 40+ times in logs, each time with "making progress"
• A 3 AM AWS billing alert, while your agent reports everything as normal
• API usage chart suddenly turns into a vertical line

These aren't bugs. They're symptoms of the same root cause.

I call it: Death by Loop.

---

The Data

Over 3 months, the ARK team helped 12 teams diagnose agent deployment failures. Here's what we found:

Failure Mode	Frequency	Avg Loss	Detected By
Death by Loop	38%	$8,400/incident	Billing shock
Death by Hallucination	24%	Data corruption	User reports
Death by Deadlock	16%	System freeze	Monitoring
Death by Amnesia	12%	Context loss	User complaints
Escalation/Poisoning/Silence	10%	Varies	Security audit

Loops are the most common, most expensive, and hardest to catch.

Why? Because looping agents don't crash. They look busy.

---

The Four Subtypes of Death by Loop

Type A: Self-Correction Spiral

Agent: generate code → run → error → "I'll fix it" → modify → run → new error → "one more fix"
...× N iterations...
Agent: "Making progress..."

The most common type. Self-correction becomes self-amplification. Each "fix" introduces a new bug, which needs another fix.

The $23,000 case: An agent was asked to fix an API response format issue. Should have taken 5 minutes. Instead, it alternated between generating, validating, and fixing with GPT-4 — 6,847 API calls. At ~$3.36 per call (GPT-4 32k context): 6,847 × $3.36 = $23,005.92.

Type B: Goal Collapse

Original goal: "Optimize database query performance"
Agent: analyze → finds index issue → adds index → writes got slower → optimize writes
→ memory pressure → adjust buffers → concurrency issues → modify connection pool...
...3 hours later...
Agent: "I've discovered a deeper issue..."

The agent loses track of what "done" means. The original goal fragments into infinite sub-tasks. Every step seems reasonable in isolation.

Type C: Tool Contention

Agent A: modifies file X →
Agent B: sees X changed, reverts →
Agent A: sees revert, modifies again →
Agent B: reverts again...

A multi-agent deadlock variant. Both agents are individually correct. Together, they're a disaster.

Type D: Validation Spiral

Agent: write code → write tests → tests fail → "need more test cases" → write more tests
→ "edge cases not covered" → more tests → "let me verify the verification logic..."

The agent keeps raising the bar for "tested enough" until it becomes unreachable.

---

Solutions: Three Layers of Defense

Layer 1: Hard Limit Circuit Breaker (Non-Negotiable)

class LoopGuard:
    """
    Three-tier protection: step limit → cost limit → time limit
    Part of ARK Trust/CostGuardian
    """

    def __init__(self, max_steps=50, max_cost_usd=10, max_duration_seconds=600):
        self.steps = 0
        self.cost = 0.0
        self.start_time = None
        self.max_steps = max_steps
        self.max_cost = max_cost_usd
        self.max_duration = max_duration_seconds

    def check_step(self, action_name: str) -> bool:
        if self.start_time is None:
            self.start_time = time.time()

        self.steps += 1

        if self.steps > self.max_steps:
            raise LoopDetected(
                f"Step limit exceeded ({self.steps}/{self.max_steps})",
                guard_type="step_limit"
            )

        if self.cost > self.max_cost:
            raise LoopDetected(
                f"Cost limit exceeded (${self.cost:.2f}/${self.max_cost})",
                guard_type="cost_limit"
            )

        elapsed = time.time() - self.start_time
        if elapsed > self.max_duration:
            raise LoopDetected(
                f"Duration exceeded ({elapsed:.0f}s/{self.max_duration}s)",
                guard_type="time_limit"
            )

        return True

Layer 2: Pattern Detector (Catch It Before It Explodes)

class PatternDetector:
    """
    Detects repeating action patterns using sliding-window Jaccard similarity.
    Catches loops before they hit the hard limit.
    """

    def __init__(self, window_size=10, similarity_threshold=0.7):
        self.window = deque(maxlen=window_size)
        self.threshold = similarity_threshold

    def add_action(self, action: str) -> Optional[str]:
        self.window.append(action)

        if len(self.window) < self.window.maxlen:
            return None

        recent = list(self.window)
        mid = len(recent) // 2

        similarity = len(set(recent[:mid]) & set(recent[mid:])) / \
                     len(set(recent[:mid]) | set(recent[mid:]))

        if similarity > self.threshold:
            return "loop"
        return None

Layer 3: Goal Decay Monitor (When Good Agents Go Off-Rails)

class GoalDecayDetector:
    """
    Tracks whether the agent is drifting away from the original goal.
    Uses cosine similarity between initial goal embedding and current actions.
    """

    def __init__(self, embedding_fn, decay_threshold=0.4):
        self.embed = embedding_fn
        self.initial_goal_embedding = None
        self.decay_threshold = decay_threshold

    def set_goal(self, goal: str):
        self.initial_goal_embedding = self.embed(goal)

    def check_decay(self, current_action: str) -> bool:
        if self.initial_goal_embedding is None:
            return False

        similarity = cosine_similarity(
            self.initial_goal_embedding,
            self.embed(current_action)
        )

        return similarity < self.decay_threshold

---

What You Can Do Today

Three things, right now:

1. Add hard limits. Steps, cost, time — all three. Don't trust your agent's self-judgment. It doesn't know when to stop.
2. Deploy loop detection. A sliding-window pattern detector in your agent's execution log. Loops don't happen suddenly — the signal is there by step 20.
3. Set cost alerts. Daily and per-task API spend thresholds. A $23,000 bill was abnormal by call #200, not call #6,847.

---

About ARK Trust

We're building ARK Trust — an open-source agent safety infrastructure. The CostGuardian module is exactly the LoopGuard you saw above, production-ready: hard limits, pattern detection, goal decay monitoring, and multi-agent contention detection.

This isn't "yet another agent framework."
It's the braking system your agents are missing.

🔗 ARK Trust on GitHub
📧 Stay updated: [e-mail]

---

Coming next: "Death by Hallucination" — when your agent wraps fabricated answers in 98% confidence scores