Postmortem: The Agent That Would Not Stop Retrying a Locked API

You notice the laptop fan first, then the invoice email, and only then the terminal that never returned to a prompt. The agent had been fixing a flaky test since Thursday evening, and every failed tool call spawned another paid completion. By Saturday morning you had a stack of identical 401 traces, a warmer room, and no merged pull request. This write-up is a postmortem of that retry storm, not a victory lap for yet another autonomous coding demo.

What actually broke

You pointed a coding agent at a private repository and a cloud completion endpoint that required a rotating key. The key expired at 18:12 on Thursday, a boring event until an unbounded loop treats every 401 as weather. The agent kept proposing the same three-line change, calling the same test command, and paying for another interpretation of the same error. Nothing in the harness treated authentication failure as a stop, so the loop looked healthy while already on fire.

Think of the agent as a junior engineer locked out of the office who still bills overtime for every tug on the handle. You would not praise persistence in that hallway, and you should not praise it in a tool loop either. The durable lesson is classification: some errors are weather, and some errors are a locked door. Once you name the locked door as a class of failure, you can stop knocking and change the plan.

Timeline you can replay

Thursday 18:12 is when the cloud key stopped authenticating, which later matches the provider audit log if you still have access. Thursday 18:14 is when the agent received the first 401 and immediately scheduled another completion with a longer system prompt. Thursday 18:19 through Friday 02:40 is a flatline of identical status codes, interrupted only by the laptop sleeping and waking. Saturday 09:03 is when you killed the process, and Saturday 09:11 is when the invoice mail arrived for a flaky test.

If you want to reproduce the shape of this failure without touching a real bill, stand up a local door that never opens. The snippet below is a labeled example, not a forensic trace from a named company production outage. You should run it only against localhost, and you should keep the paid SDK out of this rehearsal entirely. Save the handler as locked_door.py so the later client and the pytest file can share the same port.

# labeled example: a locked door for local rehearsal
from http.server import BaseHTTPRequestHandler, HTTPServer

class LockedDoor(BaseHTTPRequestHandler):
    def do_POST(self):
        self.send_response(401)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(b'{"error":"invalid_api_key"}')

    def log_message(self, fmt, *args):
        return

if __name__ == "__main__":
    HTTPServer(("127.0.0.1", 8787), LockedDoor).serve_forever()

You start that locked door with a plain Python command, then you point a tiny client at it from a second terminal. The client is deliberately naive, because naive retry logic is still what most agent loops ship with. Watch the status column, not the prompt text, because the body will look like a model apology while the code remains 401.

python locked_door.py

# labeled example: the naive retry that recreates the incident
import json, time, urllib.request

url = "http://127.0.0.1:8787/v1/completions"
payload = json.dumps({"prompt": "fix the flaky test"}).encode()
for attempt in range(1, 50):
    req = urllib.request.Request(url, data=payload, method="POST")
    req.add_header("Content-Type", "application/json")
    try:
        with urllib.request.urlopen(req, timeout=5) as resp:
            print(attempt, resp.status)
    except Exception as exc:
        print(attempt, type(exc).__name__, getattr(exc, "code", None))
        time.sleep(0.2)

You will watch dozens of attempts fail the same way, which is the entire outage compressed into a short coffee break. The contributing factor is not the 401 itself; it is the missing name for that 401. If your harness prints only the model's next thought, you will keep funding a conversation with a locked door. That is the moment a postmortem should start, long before anyone asks which model sounded more confident.

Contributing factors

The first factor is error blindness, because the harness stored the HTTP body as model commentary instead of a typed failure. The second factor is retry symmetry, because network blips and auth failures shared one backoff as if politeness could unlock the door. The third factor is budget silence, because token counters lived in a dashboard you were not watching at two in the morning. The fourth factor is fallback absence, because the only configured lane was the paid endpoint, so the loop had nowhere cheaper to go.

You can add a fifth factor: the prompt said keep going until tests pass, which is a goal, not a circuit. Goals without stop classes are how polite hallway knocking quietly becomes a whole billed weekend of retries. A free fallback does not repair a missing classifier, but it does change the blast radius once you finally have one. That spare lane only helps after the paid client is forbidden from sending another billed request.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option as that spare lane. You should verify those availability claims in the project itself, because this article does not invent model names, quotas, hardware, or duration. When the classifier reports auth, quota, or outage, you stop the paid client and continue only in a sandbox, or you wake a human.

The durable fix

The fix is a small state machine you own, not a longer prompt that begs the model to be careful. You name three failure classes, trip a circuit after a short streak, and refuse paid requests until a human resets the breaker. The code below is a proposed guard you can drop beside an agent runner until you run the tests. This snippet does not claim load-test numbers, company adoption, or any comparison against unnamed competing products.

# proposed guard: classify, trip, then optionally fall back
from dataclasses import dataclass
from enum import Enum
from time import monotonic

class FailureClass(Enum):
    WEATHER = "weather"      # timeouts, 429 with retry-after
    LOCKED_DOOR = "locked"   # 401, 403, invalid_api_key
    BUDGET = "budget"        # local cap, provider credit errors
    UNKNOWN = "unknown"

class CircuitState(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF = "half"

@dataclass
class Decision:
    action: str
    reason: str
    use_paid: bool
    use_fallback: bool

class AgentCircuit:
    def __init__(self, lock_streak=2, weather_streak=5, budget_cap=0):
        self.lock_streak = lock_streak
        self.weather_streak = weather_streak
        self.budget_cap = budget_cap
        self.paid_tokens = 0
        self._lock_hits = 0
        self._weather_hits = 0
        self.state = CircuitState.CLOSED
        self.opened_at = 0.0

    def classify(self, status: int, body: str) -> FailureClass:
        text = (body or "").lower()
        if status in (401, 403) or "invalid_api_key" in text:
            return FailureClass.LOCKED_DOOR
        if status == 429 or "timeout" in text:
            return FailureClass.WEATHER
        if "insufficient" in text or "credit" in text:
            return FailureClass.BUDGET
        if self.budget_cap and self.paid_tokens >= self.budget_cap:
            return FailureClass.BUDGET
        return FailureClass.UNKNOWN

    def observe(self, status: int, body: str, tokens=0) -> Decision:
        self.paid_tokens += tokens
        kind = self.classify(status, body)
        if kind is FailureClass.LOCKED_DOOR:
            self._lock_hits += 1
            if self._lock_hits >= self.lock_streak:
                self.state = CircuitState.OPEN
                self.opened_at = monotonic()
                return Decision("stop_paid", "locked door", False, True)
            return Decision("retry_once", "first lock signal", False, True)
        if kind is FailureClass.BUDGET:
            self.state = CircuitState.OPEN
            return Decision("stop_paid", "budget cap", False, True)
        if kind is FailureClass.WEATHER:
            self._weather_hits += 1
            if self._weather_hits >= self.weather_streak:
                return Decision("backoff", "weather streak", False, True)
            return Decision("retry_paid", "transient weather", True, False)
        return Decision("continue", "unclassified", True, False)

    def reset_with_new_key(self):
        self._lock_hits = 0
        self._weather_hits = 0
        self.state = CircuitState.CLOSED

You wire observe after every provider response, including the exceptions your HTTP client raises on a 401. A status code is not a string in a chat transcript; it is an event that must change the next action. The fallback can be a local mock or that free server option, but it must use a different client object. Separate clients keep expired keys from hitchhiking into the sandbox and starting a second retry storm.

Here is a compact pytest file that locks the postmortem into source control before you trust the agent overnight. These tests are proposed examples; they pass against the class above, and they do not represent a production incident score. Run them in the same pull request as the agent config so a prompt edit cannot silently delete the breaker.

# proposed tests: these encode the postmortem, not a marketing claim
from agent_circuit import AgentCircuit, CircuitState

def test_second_401_opens_circuit_and_blocks_paid_lane():
    c = AgentCircuit(lock_streak=2)
    first = c.observe(401, '{"error":"invalid_api_key"}')
    second = c.observe(401, '{"error":"invalid_api_key"}')
    assert first.use_paid is False
    assert second.use_paid is False
    assert second.use_fallback is True
    assert c.state is CircuitState.OPEN

def test_single_timeout_does_not_open_the_door_circuit():
    c = AgentCircuit()
    d = c.observe(429, "timeout")
    assert d.action == "retry_paid"
    assert c.state is CircuitState.CLOSED

pytest -q test_agent_circuit.py

The decision table is the rest of the runbook, written so a tired you can follow it without rereading the class. You keep the table next to the agent config so the system prompt cannot override it at two in the morning. Prompts will argue about confidence and tone, while tables referee where the money is allowed to go. Print this table in the on-call doc, not only in a blog post you will forget by next Thursday.

Signal	Class	Paid lane	Fallback lane	Human
401/403 or invalid key	locked door	stop	optional sandbox	rotate key
429 / timeout	weather	retry with backoff	after streak	no
credit / local cap	budget	stop	optional sandbox	raise cap
tests still failing, HTTP 200	unknown	continue once	no	inspect patch

What this does not fix

A circuit breaker will not make a weak patch pass tests, and a free fallback will not inherit paid-endpoint latency. You should not treat free model access or a free server option as a high-availability contract that cannot change. If your agent can push branches or apply migrations, the fallback lane belongs in a disposable clone. Logging every tool call is part of the fix, because without that log you are guessing which door you knocked on.

You should not use this approach if you need deterministic vendor SLAs, or if your compliance team forbids mixed providers on one task. You should not use it if you cannot log every tool call, because the breaker is only as honest as its events. Skip this pattern if your agent is already a cron job, because then you need a job timeout. Do not treat the breaker as permission to leave a loop running unattended; it is a seatbelt, not a chauffeur.

If you need a spare lane after a locked paid door, try MonkeyCode's free model access and free server option on a throwaway repo. Keep the circuit tests in the same pull request as the agent config, and rotate the original key before you reopen the paid lane. That is the entire invitation: a rehearsal, not a promise that the weekend will bill itself politely next time.