Postmortem: The Retry Patch That Ate Staging

You were not even sitting at the keyboard when staging stopped answering checkout requests late on a quiet Thursday night. A Slack ping at 22:14 reported a wall of 429s, and the on-call bot blamed a branch an agent had opened earlier. You assumed a human had reviewed that diff because the title sounded calm and the CI badge was still green. The agent had been told to reduce timeouts, and it obeyed by deleting jitter and widening the client retry budget.

This write-up is a postmortem, not a victory recap, because the damage lived in the tool-call chain rather than in a flashy dashboard screenshot. You will walk a reconstructed timeline, name the contributing factors, and leave with a replay harness that fails closed on a mutating edit. Treat the story as a worked incident, not as a claim about one company’s production metrics. The useful part is the durable fix you can run twice, not a lecture about whether agents are good.

At 20:07 the agent received a ticket that said checkout felt slow and that timeouts should come down without naming a retry ceiling. It searched the client, patched two files, and ran unit tests that stubbed HTTP so thoroughly that a 429 could never appear. You would have seen the timeout fall from 2500ms to 800ms, retries climb from two to six, and the jitter helper collapse into a constant ten milliseconds. Each tool call looked locally reasonable, which is exactly how a cascade hides inside a tidy chat transcript.

At 20:41 the agent opened a pull request titled “reduce checkout latency,” and an auto-approver merged it after a green unit job. Staging rolled the pods at 21:18, while traffic was still thin, so the new retry budget had nothing painful to amplify yet. A synthetic canary at 21:55 finally struck a rate-limited dependency, and every checkout client turned one 429 into six tight retries. By 22:14 the dependency had shed load, the pods were busy retrying, and you were reading Slack in a dark kitchen.

None of those steps required a malicious model, a leaked key, or a novel exploit in the HTTP library you already trust. Tool calling simply turned a vague symptom into file edits that nobody replayed against a live 429. If you have been watching the current wave of in-browser agent demos, you already know a model can invoke tools; the missing muscle is incident discipline around those invocations. An agent that never leaves a laptop can still melt shared staging when mutating tools have no budget and no ledger.

The first contributing factor was a ticket that described a feeling instead of a constraint, so latency won and retry ceilings lost. The second factor was a suite that mocked happy paths and never injected 429, 503, or a slow Retry-After header. The third factor was an auto-merge rule that treated a green unit job as evidence a live dependency would stay calm. The fourth factor was the absent tool-call ledger, which left you scrolling a chat to guess which edit landed in which order.

Picture the agent as a very fast junior engineer who has never been paged for last quarter’s retry storm and does not fear a 429. Speed is not the defect in that picture; missing brakes are the defect, and comments would not have saved you. The generated patch was tidy, the names were clear, and the commit message was grammatical enough to lull a reviewer. Clear code that deletes jitter is still an incident waiting for the first burst of real traffic.

You need three artifacts that survive the next enthusiastic agent session: a ledger, a mutating-tool gate, and a replay that speaks 429. The ledger records tool name, path, and digest so the next postmortem does not depend on a vendor chat window. The gate refuses writes under app/, infra/, or clients/ unless a human token is present in the environment. The replay drives your HTTP client against a tiny local server that returns 429 three times and 200 after that.

Save the following worked example as agent_guard.py. It is labeled as unexecuted sample code for this article, not as a file recovered from the Thursday night cluster.

#!/usr/bin/env python3
"""Worked example: tool-call ledger, mutating-tool gate, and 429 replay."""
from __future__ import annotations

import hashlib
import json
import os
import random
import time
import urllib.error
import urllib.request
from http.server import BaseHTTPRequestHandler, HTTPServer
from pathlib import Path
from threading import Thread

LEDGER = Path(".tool_ledger.jsonl")
PROTECTED = ("app/", "infra/", "clients/")


def record(tool: str, payload: dict) -> None:
    line = {
        "ts": time.time(),
        "tool": tool,
        "payload": payload,
        "digest": hashlib.sha256(
            json.dumps(payload, sort_keys=True).encode()
        ).hexdigest()[:16],
    }
    with LEDGER.open("a", encoding="utf-8") as handle:
        handle.write(json.dumps(line) + "\n")


def allow_mutation(path: str) -> bool:
    if not any(path.startswith(prefix) for prefix in PROTECTED):
        return True
    return os.environ.get("AGENT_MUTATE") == "signed-off"


def apply_edit(path: str, content: str) -> str:
    record("edit_file", {"path": path, "bytes": len(content)})
    if not allow_mutation(path):
        raise PermissionError(
            f"blocked edit to {path}; export AGENT_MUTATE=signed-off"
        )
    Path(path).parent.mkdir(parents=True, exist_ok=True)
    Path(path).write_text(content, encoding="utf-8")
    return "ok"


class FlakyHandler(BaseHTTPRequestHandler):
    hits = 0

    def do_GET(self):
        FlakyHandler.hits += 1
        if FlakyHandler.hits <= 3:
            self.send_response(429)
            self.send_header("Retry-After", "1")
            self.end_headers()
            self.wfile.write(b"rate limited")
            return
        self.send_response(200)
        self.end_headers()
        self.wfile.write(b"ok")

    def log_message(self, *_args):
        return


def start_flaky() -> HTTPServer:
    httpd = HTTPServer(("127.0.0.1", 8765), FlakyHandler)
    Thread(target=httpd.serve_forever, daemon=True).start()
    return httpd


def naive_client(timeout_s: float, retries: int, jitter: bool) -> dict:
    errors = 0
    for attempt in range(retries + 1):
        try:
            urllib.request.urlopen(
                "http://127.0.0.1:8765/checkout", timeout=timeout_s
            )
            return {"ok": True, "attempts": attempt + 1, "errors": errors}
        except (urllib.error.URLError, TimeoutError, OSError):
            errors += 1
            time.sleep(0.05 + random.random() * 0.2 if jitter else 0.01)
    return {"ok": False, "attempts": retries + 1, "errors": errors}


if __name__ == "__main__":
    start_flaky()
    time.sleep(0.1)
    storm = naive_client(timeout_s=0.8, retries=6, jitter=False)
    FlakyHandler.hits = 0
    calm = naive_client(timeout_s=2.5, retries=2, jitter=True)
    print(json.dumps({"storm": storm, "calm": calm}, indent=2))

You run the comparison with one command that prints whether the “optimized” client recovered or simply hammered the fake dependency until the process gave up.

python3 agent_guard.py

On a typical laptop the storm path records a fistful of tight failures, while the calmer client survives because jitter and a smaller retry budget let a later request land as 200. You now have a failing-then-passing story you can paste into the pull request instead of trusting a mocked unit test. If you want the gate itself to fail closed during an agent session, you call apply_edit from the tool layer and leave AGENT_MUTATE unset until a human exports it.

Prove the gate before you argue about model quality, because a blocked write is cheaper than another staging melt. The next snippet is also a worked example, and it should raise PermissionError until you deliberately sign off.

python3 - <<'PY'
from agent_guard import apply_edit
try:
    apply_edit("clients/checkout.py", "RETRY = 6\n")
    print("unexpected allow")
except PermissionError as exc:
    print(exc)
PY

A second command rebuilds the postmortem timeline after the agent has been playing with files, which is the part most chat UIs throw away.

python3 - <<'PY'
import json
from pathlib import Path
ledger = Path(".tool_ledger.jsonl")
if not ledger.exists():
    raise SystemExit("no ledger; the next incident will be a chat screenshot")
for line in ledger.read_text(encoding="utf-8").splitlines():
    row = json.loads(line)
    print(f"{row['ts']:.0f} {row['tool']} {row['digest']} {row['payload']}")
PY

That loop is the difference between “the model did something” and an incident timeline you can defend in writing next week. You should require the ledger file as a CI artifact whenever an agent is allowed to open a pull request against client code. Without it, next Thursday looks exactly like this Thursday, except the Slack thread is longer and the blame is fuzzier.

You should not rehearse mutating agents against the same staging cluster that product managers use for Friday demos. A disposable server is the right blast radius, because you can let the retry storm happen, keep the ledger, and then destroy the machine. MonkeyCode is an open-source coding environment with free model access and a free server option, which is enough to run the harness without pointing the agent at real checkout.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use that box as a flight simulator, not as a second production, and throw the disk away when the ledger has taught you something. Feed the original ticket to the agent there, watch which tools it selects, and only promote a patch after the 429 replay stays green. If the free models available to you cannot honor the gate, that negative result is still useful, because you keep the human token unset and you do not merge. Do not treat free capacity as a promise about uptime, model names, quotas, or how long the offer lasts.

This harness will not replace an observability stack, a feature flag, or a real load test against your payment provider. It will not stop a determined operator from exporting AGENT_MUTATE and merging anyway, and it will not rank models. If you work in a regulated environment that forbids sending code to a shared server, run the same files on a machine you already control and skip the hosted option. You should also skip this approach if your agents only draft comments and never touch runtime clients, because a retry storm is not your incident class.

The durable fix is boring on purpose: constrain the ticket, record the tools, fail closed on protected paths, and replay 429s before auto-merge. You do not need a frontier model to learn that lesson, and you do not need staging as a sacrificial altar. If you want a throwaway rehearsal of the script, a free server you can delete after the ledger is written is enough for the exercise.