48-Hour Field Notes: The POST That Fired Twelve Times

The first clue was not an exception. It was a quiet access log: the same POST /hooks/run-complete twelve times in 38 seconds, each with a fresh TCP connection and the same JSON body. The planner summary still read “delivered.” The worker had not crashed. The endpoint had answered 429 on the first eleven tries and 200 on the twelfth, and the generated client treated that sequence as a clean win.

This is a 48-hour field note, not a launch post. I wanted a small, boring webhook client that an agent could maintain. I got a retry storm that looked like throughput. The rest of the weekend was spent teaching the loop to sit still.

Hour 0: a green bar that hid a queue

The task was ordinary. A test runner finishes, then notifies an internal hook so a dashboard can flip a row from running to done. I asked an agent to write the client, a tiny mock server, and a pytest file. The unit test passed because it stubbed requests.post to return 200 on the first call. Nothing in that test ever saw a throttle.

On a real socket the picture changed. I pointed the client at a local server that answered 429 three times, then 200. The process did not fail. It also did not wait. It hammered.

python -m http.server 8765 &
# not this — a static file server cannot speak 429 with Retry-After

A file server is the wrong stand-in. HTTP retries are a protocol conversation, not a file read. I replaced it with a 40-line mock that records every arrival and honors Retry-After.

# save as mock_hook.py — proposed harness, not production code
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json, threading, time

HITS = []
LOCK = threading.Lock()
FAILS_BEFORE_OK = 3

class Hook(BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers.get("Content-Length", "0"))
        body = self.rfile.read(length)
        with LOCK:
            HITS.append((time.time(), self.headers.get("Idempotency-Key"), body))
            n = len(HITS)
        if n <= FAILS_BEFORE_OK:
            retry_after = "1"
            self.send_response(429)
            self.send_header("Retry-After", retry_after)
            self.send_header("Content-Type", "application/json")
            self.end_headers()
            self.wfile.write(b'{"error":"throttled"}')
            return
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(b'{"ok":true}')

    def log_message(self, fmt, *args):
        return

if __name__ == "__main__":
    ThreadingHTTPServer(("127.0.0.1", 8765), Hook).serve_forever()

The first agent-written client ignored every header except status. Retry-After: 1 sat in the response like an unread sticky note on a locked door. The loop slept 50 milliseconds, or not at all, and tried again. Twelve arrivals, one success, eleven wasted turns. The dashboard still flipped to done, which is how a storm disguises itself as a feature.

What I tried that did not hold

I first added a counter. max_attempts=5 stopped the worst spin, but it still retried 400 and 401 as if a second POST would fix a bad payload. That is a category error. A 401 is a key. A 400 is a schema. Neither yields to enthusiasm.

I then asked the agent to “add exponential backoff.” It multiplied a delay by two and capped it at two seconds. There was no jitter. Two workers started on the same clock and collided on the same second, which is how a polite retry becomes a chorus. The mock’s hit list showed pairs 12 milliseconds apart. Backoff without jitter is a metronome, not a cushion.

The third miss was identity. Each retry minted a new body timestamp and no Idempotency-Key. The server, if it had been real, would have recorded twelve completions for one run. The mock only counted. Counting is not the same as recognizing.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode’s free model access and free server option as a scratch box so the mock and the client could fight each other without borrowing a laptop fan. Those two availability facts are the only product claims here. I am not attaching model names, token ceilings, or hardware specs I cannot verify.

The artifact: classify, then budget, then key

The fix that survived the second night was not a bigger model prompt. It was a status classifier the client must consult before it sleeps. I wrote it as a pure function so pytest could pin it without opening a socket.

# retry_policy.py
from dataclasses import dataclass

@dataclass(frozen=True)
class Decision:
    action: str          # "ok" | "retry" | "fail"
    sleep_s: float       # 0 if not retry
    reason: str

RETRYABLE = {408, 425, 429, 500, 502, 503, 504}
FATAL_CLIENT = {400, 401, 403, 404, 405, 409, 413, 415, 422}

def decide(status: int, retry_after: str | None, attempt: int, budget: int) -> Decision:
    if status < 300:
        return Decision("ok", 0.0, "2xx")
    if status in FATAL_CLIENT:
        return Decision("fail", 0.0, f"non-retryable {status}")
    if attempt >= budget:
        return Decision("fail", 0.0, "budget exhausted")
    if status not in RETRYABLE:
        return Decision("fail", 0.0, f"unlisted {status}")
    sleep = 1.0
    if retry_after:
        try:
            sleep = min(max(float(retry_after), 0.0), 30.0)
        except ValueError:
            sleep = 1.0
    # deterministic jitter for tests: 0.5x .. 1.5x
    jitter = 0.5 + ((attempt * 17) % 100) / 100.0
    return Decision("retry", round(sleep * jitter, 3), f"retryable {status}")

The tests are the protocol. They do not stub the world into 200. They name the storm.

# test_retry_policy.py
from retry_policy import decide

def test_200_is_terminal():
    d = decide(200, None, attempt=0, budget=5)
    assert d.action == "ok"

def test_400_is_never_retried():
    d = decide(400, "2", attempt=0, budget=5)
    assert d.action == "fail"
    assert d.sleep_s == 0.0

def test_429_honors_retry_after_and_stays_under_cap():
    d = decide(429, "2", attempt=1, budget=5)
    assert d.action == "retry"
    assert 1.0 <= d.sleep_s <= 3.0

def test_budget_beats_retryable_status():
    d = decide(503, "1", attempt=5, budget=5)
    assert d.action == "fail"
    assert "budget" in d.reason

Run them without the mock first. Policy bugs should fail in under a second. Only then start the server and a client that actually sleeps.

# client.py — proposed; sleeps on Decision.sleep_s
import os, uuid, json, urllib.request, urllib.error
from retry_policy import decide

HOOK = os.environ.get("HOOK_URL", "http://127.0.0.1:8765")
BUDGET = 5

def post_once(payload: dict, key: str) -> tuple[int, str | None]:
    data = json.dumps(payload).encode()
    req = urllib.request.Request(
        HOOK, data=data, method="POST",
        headers={
            "Content-Type": "application/json",
            "Idempotency-Key": key,
        },
    )
    try:
        with urllib.request.urlopen(req, timeout=5) as resp:
            return resp.status, resp.headers.get("Retry-After")
    except urllib.error.HTTPError as e:
        return e.code, e.headers.get("Retry-After")

def deliver(payload: dict) -> None:
    key = payload.get("run_id") or str(uuid.uuid4())
    for attempt in range(BUDGET + 1):
        status, retry_after = post_once(payload, key)
        d = decide(status, retry_after, attempt, BUDGET)
        if d.action == "ok":
            return
        if d.action == "fail":
            raise RuntimeError(f"hook failed: {d.reason} status={status}")
        time_sleep = __import__("time")
        time_sleep.sleep(d.sleep_s)

The integration check is a wall clock, not a vibe. Start the mock, POST once, then assert len(HITS) == FAILS_BEFORE_OK + 1 and that every recorded key is identical. If the length is twelve, the client is still deaf. If the keys differ, the server cannot tell a retry from a new run.

python mock_hook.py &
HOOK_URL=http://127.0.0.1:8765 python -c "from client import deliver; deliver({'run_id':'run-9','ok':True})"
# then inspect HITS in a /debug route, or print them on SIGUSR1

I added a /debug/hits GET to the mock on Sunday morning. Reading it was faster than grepping stdout. Agents like to log; a structured hit list is harder to misread than a novel of timestamps.

What broke after the classifier landed

Retry-After is allowed to be an HTTP-date, not only a delta-seconds integer. The first parser called float("Wed, 14 Sep 2026 12:00:00 GMT") and fell back to one second. That is legal Python and illegal patience. I narrowed the parser: digits become seconds; anything else becomes a hard fail into the default sleep, logged as retry-after-unparsed. Unparsed is a first-class outcome. Silent fallback is how the twelve-POST bug returned wearing a new coat.

Timeouts were the other crack. urlopen(..., timeout=5) raises URLError, which is not an HTTP status. The classifier never saw it. I mapped connect/read timeouts to a synthetic 408 at the adapter boundary, not inside decide. Mixing transport failure with application status in one function makes the tests lie.

Clock skew showed up when I tried Retry-After as a date on a box whose NTP had drifted. I stopped accepting dates in this client. A webhook notifier does not need calendar math. Delta-seconds or default. That is a limitation, and it is cheaper than debugging timezones in a retry loop.

What I would repeat

I would write the classifier and its four tests before the agent touches sockets. I would give the mock a fixed FAILS_BEFORE_OK and an idempotency map, then refuse any client that cannot survive that script. I would treat 429 as a yellow light with a posted wait, not as a soft 500. I would cap sleep, cap attempts, and cap uniqueness: one key per run_id.

I would not let a planner summarize “success after retries” without printing attempt count and the last non-2xx status. A summary that omits the eleven 429s is how this bug shipped into a log that looked healthy. The number twelve is the story. Hide it and you will debug the dashboard instead of the client.

MonkeyCode’s free server was enough to keep the mock and the pytest file in one place while I iterated. If you already have a local interpreter, you do not need it. The harness above is ordinary CPython.

Limitations, and who should not copy this

This policy is for a single webhook with a known owner. It is the wrong shape for payments, emails, or any call with side effects you cannot make idempotent. It is also wrong for APIs that throttle by user and expect you to stop, not to jitter. A 404 on a hook path is a deploy bug; retrying it only multiplies noise.

Do not use unbounded parallelism against a free or shared box. A retry storm is a small denial of service you inflict on yourself. Do not parse Retry-After dates unless you own the clock. Do not claim the loop is “resilient” because a twelfth try landed. Resilience is the budget, the key, and the classifier agreeing in a test you can rerun cold.

The 48 hours ended with four tests, a mock that can say no, and a client that sleeps when told. That is the whole artifact. The planner can keep its adjectives. The hit list is the record.