Replay One Failure Before a Retry Patch Ships

A checkout webhook began returning timeouts just after midnight, and the on-call engineer opened an assistant thread before the kettle finished boiling. The suggested patch widened the retry loop, stretched the backoff, and described the edit as obviously safe for a transient partner error. The diff touched fewer than thirty lines, yet that service already shared a partner rate limit that billed every additional outbound attempt. A hopeful merge would have converted a short outage into a quiet invoice and a harder incident review.

The useful response was not another broad opinion about artificial intelligence replacing careful operational judgment overnight. It was a time-boxed spike with one hypothesis and a kill rule written down before any candidate code ran. The hypothesis said the wider retry would recover the captured timeout without exceeding four outbound attempts or moving the idempotency key. Ninety minutes covered setup, replay, notes, and the ship-or-discard decision, so unproven work stayed unmerged when the window closed.

Freeze the clock before judging the patch

The artifact is a small replay harness that freezes time and counts outbound attempts against one redacted failure. It is proposed spike code, not a report of a named production trace already executed on a public host. A captured failure is stored as JSON with the idempotency key and the ordered partner responses inside it. The harness drives the candidate retry function through that sequence while a fake clock advances only when the test requests sleep.

If the function sleeps, the fake clock jumps forward, and the test records that jump instead of waiting on a wall clock. That choice keeps the ninety-minute budget from being consumed by real backoff delays during a local replay. Reviewers inspect attempt counts without confusing laptop time with partner time, and the harness below is what the spike trusts or kills. The listing that follows is an unexecuted example meant to be copied into the spike directory and run locally.

import json
import sys
from dataclasses import dataclass, field

@dataclass
class FrozenClock:
    now_ms: int = 0
    sleeps: list = field(default_factory=list)

    def sleep(self, ms: int) -> None:
        self.sleeps.append(ms)
        self.now_ms += ms

@dataclass
class AttemptLog:
    calls: int = 0
    keys: set = field(default_factory=set)

def replay(retry_fn, trace: dict, clock: FrozenClock) -> AttemptLog:
    log = AttemptLog()
    responses = list(trace["responses"])

    def partner_call(key: str) -> dict:
        log.calls += 1
        log.keys.add(key)
        if not responses:
            return {"status": 599, "body": "trace exhausted"}
        return responses.pop(0)

    retry_fn(
        key=trace["idempotency_key"],
        call=partner_call,
        sleep=clock.sleep,
        budget=trace["max_attempts"],
    )
    return log

def judge(log: AttemptLog, trace: dict) -> None:
    if log.calls > trace["max_attempts"]:
        raise SystemExit("kill: attempt budget exceeded")
    if log.keys != {trace["idempotency_key"]}:
        raise SystemExit("kill: idempotency key drifted")
    if not log.calls:
        raise SystemExit("kill: patch made no attempt")

# Replace candidate_retry with the assistant patch under trial.
def candidate_retry(key, call, sleep, budget):
    delay = 200
    for _ in range(budget):
        result = call(key)
        if result["status"] < 500:
            return result
        sleep(delay)
        delay = min(delay * 2, 1600)
    return {"status": 504, "body": "budget spent"}

def main() -> None:
    trace_path = sys.argv[1] if len(sys.argv) > 1 else "trace.json"
    with open(trace_path, encoding="utf-8") as handle:
        trace = json.load(handle)
    clock = FrozenClock()
    log = replay(candidate_retry, trace, clock)
    judge(log, trace)
    print(f"ship: calls={log.calls} clock_ms={clock.now_ms}")

if __name__ == "__main__":
    main()

The candidate function under test should accept the same four arguments that the harness passes into each replay. A careful patch retries only while the captured status is retriable, sleeps through the frozen clock, and stops once the budget is spent. A reckless patch often treats every non-200 status as temporary, including a conflict that means the partner already accepted the charge. The spike cares about that distinction more than it cares about the confident prose wrapped around the diff.

A trace file keeps the evidence inspectable by a reviewer who never opened the assistant thread during the incident. The sample holds one timeout, one overload, and one success, which is enough to exercise backoff without imitating a full capture. Teams should replace that sample with a redacted trace from their own incident archive before trusting a ship decision. They should also refuse any file that still contains secrets, card numbers, partner credentials, or raw customer payloads.

{
  "idempotency_key": "evt_frozen_1042",
  "max_attempts": 4,
  "responses": [
    {"status": 504, "body": "timeout"},
    {"status": 503, "body": "overload"},
    {"status": 200, "body": "accepted"}
  ]
}

The commands belong in the spike notes so another engineer can rerun the decision without reconstructing the chat. The first command creates a disposable directory, and the later commands run the harness and record a hash of the trace. A changed fixture hash means the evidence moved, so the previous exit code no longer supports a ship decision. None of these commands should be pointed at a live partner endpoint, and none of them requires production credentials.

mkdir -p spike/retry-replay && cd spike/retry-replay
python replay_harness.py trace.json
sha256sum trace.json > trace.sha256 2>/dev/null || shasum -a 256 trace.json > trace.sha256

Read the exit code, then stop

Reading the result should stay mechanical rather than rhetorical, because the spike promised evidence instead of another assistant summary. A zero exit, a stable idempotency key, and an attempt count inside the budget support a ship for this trace only. A non-zero exit, a drifted key, or an exhausted response list supports a kill, even when the comment calls the miss harmless. If the ninety minutes expire before a clean run, the spike ends as a kill by timeout and the patch waits.

This approach is a poor fit for several kinds of work, and saying so matters as much as the harness itself. It should not be used as a load test, a security review, or proof that a disposable server matches production capacity. It should not be used on traces that cannot be redacted, or on flows where a retry can move money without an idempotency key. Regulated teams that require validated vendors and signed change control should keep the replay in a local lab until those controls are satisfied.

The limitations are ordinary, and they belong beside the exit code rather than in a separate apology. A frozen clock hides real jitter, resolver failures, and pools that misbehave only when many calls overlap. One trace cannot stand for a week of partner behavior, and a green replay does not license a wider policy on unrelated routes. Free model access can draft the harness quickly, yet that draft stays untrusted until kill rules fire in code.

A disposable host can also disappear, differ from production, or reject the workload before the replay finishes. The note should record that host only as the place the replay ran, not as a performance trophy or a capacity claim. If the trace is older than the current partner contract, the spike should be killed and recaptured instead of treated as fresh evidence. Readers with a redacted trace can run this harness on a disposable server and keep the patch only if the replay holds.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A free server option is enough to reproduce the setup.