Postmortem: Your Agent Can Mistake Courtesy for Progress

You notice staging first as a Slack ping, not as a stack trace, which already tells you the failure is cultural. The CPU graph is a flat roof, and the request log repeats one tool name like a stuck record. You open the agent transcript and every turn looks earnest because the model saw an error and called search_docs again. Nothing in that file looks like a crash, which is why it took forty minutes to treat it as an incident.

This write-up is not a sermon about vibe coding, and it is not a ranking of chat vendors either. You asked an assistant to wire a retrieval tool into a small internal chatbot and then shipped the glue after one good demo. The failure hid in the unhappy path, where a 429 became English and returned as the next user turn. The model did what models do with unfinished work, and it tried again politely until the worker had no file descriptors left.

Timeline you can defend in a review

The chatbot received a product question at 14:02 that required a keyword search against a flaky staging index. The first tool call returned HTTP 429, and your wrapper serialized that status into a calm sentence that asked the model to retry. That sentence became the next turn, which emitted the same search_docs call about two hundred milliseconds later. No stack trace appeared, because from the runtime's point of view the agent was still making progress through another completion.

By 14:04 the worker had issued several thousand duplicate searches and the connection pool sat on its ceiling like a clogged drain. Public traffic stayed tiny, so the load balancer graphs looked almost bored while the damage stayed inside one process. The /healthz route still returned 200 because it never entered the agent loop or touched the search client. You were watching the wrong door while smoke rose from a kettle the model would not take off.

A teammate finally pasted a log line that showed tool=search_docs with no request id, no attempt counter, and no fingerprint. That missing identity is what turned a retry into a new task every time the model apologized. Killing the process stopped the storm, and the transcript on disk remained a long hallway of courteous repeats. You should keep that hallway, because a looping agent is much easier to teach with evidence than with memory.

Contributing factors that stacked

The model was not a villain, and the HTTP client was not uniquely cursed on that afternoon either. Tool errors had been coerced into chat text, which erased status codes, retry-after headers, and the identity of the failed call. The orchestrator had a max-tokens knob and no max-steps knob, so a long apology looked like work that never ended. The health check lived on a different code path, so the process could look alive while it hammered the index without pause.

Think of a junior on-call who hears a locked door and knocks every two hundred milliseconds because nobody allowed a stop. Generated glue often encodes that junior with painful accuracy, since it is fluent, eager, and incapable of boredom. Courtesy in a transcript is not a control system, even when it reads like a careful engineer narrating a plan. You needed a door closer the model could not sweet-talk, not a firmer paragraph in the system prompt.

A supervisor the model cannot vote away

You do not repair this incident with a scolding prompt, because prompts are still inside the loop that failed you. You add a supervisor with a wall-clock budget, a hard cap on tool rounds, and a fingerprint that makes duplicate calls visible. Errors stay structured until the supervisor decides the turn is finished, which means a 429 never becomes a plea in natural language. The harness below is a local template for an OpenAI-style tool loop, not a framework to drop into production unreviewed.

# replay_guard.py — supervisor for tool-calling loops (template, not a product)

from dataclasses import dataclass, field
from hashlib import sha256
from time import monotonic

@dataclass
class LoopGuard:
    max_steps: int = 8
    max_seconds: float = 20.0
    max_same_fingerprint: int = 2
    started: float = field(default_factory=monotonic)
    steps: int = 0
    seen: dict = field(default_factory=dict)

    def check(self, tool_name: str, arguments: str) -> None:
        self.steps += 1
        elapsed = monotonic() - self.started
        digest = sha256(f"{tool_name}:{arguments}".encode()).hexdigest()[:12]
        self.seen[digest] = self.seen.get(digest, 0) + 1
        if self.steps > self.max_steps:
            raise RuntimeError(f"step budget exceeded at {self.steps}")
        if elapsed > self.max_seconds:
            raise RuntimeError(f"wall clock exceeded at {elapsed:.1f}s")
        if self.seen[digest] > self.max_same_fingerprint:
            raise RuntimeError(f"duplicate tool call {tool_name} fp={digest}")

def classify_tool_error(status: int, retry_after: str | None) -> dict:
    """Keep machine errors machine-readable. Do not narrate them to the model yet."""
    retryable = status in {408, 429, 500, 502, 503, 504}
    return {
        "ok": False,
        "status": status,
        "retryable": retryable,
        "retry_after": retry_after,
        "next": "backoff" if status == 429 else "fail",
    }

You call LoopGuard.check before every tool dispatch, and you never stringify a 429 into a sentence that begs for another attempt. If the error is retryable, the supervisor sleeps on Retry-After or fails the turn without asking the model to invent a synonym. After the incident you store JSONL with request_id, step, tool, and fingerprint, because a chat panel hides repetition. A fixture made from the 14:02 transcript then becomes a regression test instead of a war story you retell after stand-up.

# test_replay_guard.py
import json
from replay_guard import LoopGuard
import pytest

def test_duplicate_search_is_killed():
    guard = LoopGuard(max_steps=6, max_seconds=5, max_same_fingerprint=2)
    query = '{"q": "refund window"}'
    with pytest.raises(RuntimeError, match="duplicate tool call"):
        for _ in range(3):
            guard.check("search_docs", query)

def test_incident_jsonl_trips_the_same_guard(tmp_path):
    path = tmp_path / "incident.jsonl"
    row = json.dumps({"tool": "search_docs", "arguments": '{"q": "refund"}'})
    path.write_text((row + "\n") * 5)
    guard = LoopGuard(max_same_fingerprint=2)
    with pytest.raises(RuntimeError):
        for raw in path.read_text().splitlines():
            item = json.loads(raw)
            guard.check(item["tool"], item["arguments"])

You run the tests in a throwaway venv so the durable fix is a command, not a memory of a meeting. That command is boring on purpose, because boring is what staging lacked at 14:02. If pytest is silent, you have not replayed the incident; you have only admired a guard that never saw the hallway. Keep the fixture next to the orchestrator so the next refactor cannot delete the door closer by accident.

python3 -m venv .venv
source .venv/bin/activate
pip install pytest
pytest -q test_replay_guard.py

Replay the failure without turning it into a bill

You still want a live model when the tool fails in the middle of a streamed sentence rather than in a frozen fixture. Paid APIs make that experiment painful, because a broken loop is exactly the traffic pattern that turns a debug session into an invoice. A free-model scratch environment is the right isolation box, since this failure mode is repetition and missing brakes rather than rare chain-of-thought.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option that can host this replay harness while the transcript fails in view. Treat that as a quiet room for an embarrassing loop, not as a capacity plan, a leaderboard, or a guarantee about limits. If you already run a local runtime, use that instead, because isolation is the requirement and the brand is optional.

When you replay, inject faults on purpose instead of waiting for staging to flap under someone else's demo. Return 429 on the first two search_docs calls, then a 200, and confirm the guard stops before a fourth identical fingerprint. If your wrapper still turns the 429 into chat, the model will recast the query, so hash canonical arguments. Watch the JSONL, not the pretty transcript, because pretty text is how this incident stayed invisible.

Who should ignore these defaults

A step cap will not save you from a correct tool that is slow and holds a lock in the index. It will not save you from prompt injection that changes the tool name every round so fingerprints never collide either. It will not replace tracing, and if you skip spans around tool dispatch you will debug with novels again after the next release. People building multi-hour research agents should not copy eight steps and twenty seconds, because those numbers are incident brakes for a small chatbot.

You should not add this supervisor if your so-called agent is a single function call with no tools and no loop. Ceremony around a one-shot completion wastes attention you could spend on timeouts in the HTTP client you already have. You also should not stretch a free debugging box into a production plan, because this article claims no hardware, quotas, or permanence. The older lesson still holds: health checks must share fate with the work, and errors need types the model cannot rewrite.

Once pytest is red on the old transcript and green on a bounded retry, you can let the chatbot speak again. If you need a scratch pad to rerun that fixture on a live free model, MonkeyCode's free server option keeps it off staging.