Postmortem: 429 Retry Wrote Into a Recycled Agent Slot

A retry after HTTP 429 replayed a destructive apply. The second call landed in a recycled workspace slot. The durable fix is an idempotency key plus a retry classifier.

Incident summary

This postmortem reconstructs a lab failure on a shared agent server. Two jobs overlapped after a client rate-limit retry. The later apply mutated files owned by the earlier job.

The failure was not a model hallucination at all. It was a transport retry with a non-idempotent tool. Shared slots made the replay visible and harmful.

Lab boundary

This reconstructed incident used MonkeyCode free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The product matters only as a pooled worker that can recycle slots. Any shared agent runner can hit the same retry class.

Timeline

The clock below is a lab clock, not production telemetry. Each step was reproduced with the fixture in the next section.

T+0s — accept

The runner accepted Job A and assigned slot S7. Slot S7 mounted a clean worktree for repository demo-ledger. The apply tool pointed at that worktree path.

T+4s — first apply

Job A emitted a tool call named apply_patch. Arguments named pkg/lock.py and a unified diff. The worker wrote the file and returned HTTP 200 with receipt r-a1.

T+9s — rate limit

Job A then called run_tests against the same slot. The free server returned HTTP 429 with Retry-After: 2. The client default retry replayed the last in-flight request.

T+11s — wrong buffer

The HTTP client retried apply_patch, not run_tests. Connection pooling had left the apply request as the retry buffer. Slot S7 had already been marked idle after the 200.

T+12s — recycle

Job B was waiting on the free server queue. The scheduler assigned Job B to slot S7. Job B's worktree replaced Job A's files under the same path.

T+13s — replay

The replayed apply_patch ran against slot S7. The diff for pkg/lock.py landed in Job B's tree. Job B's later tests passed on a file it never authored.

What broke

Three isolation invariants failed at the same moment. Each failure was necessary for the cross-job write. Removing any one check would have stopped the replay.

• The retry layer assumed every tool call was idempotent.
• The worker reused slot paths without a generation number.
• The HTTP client retried the wrong buffered request.

Green tests on Job B hid the corruption. The test suite never asserted file provenance per job. A passing run is not a proof of isolation.

Contributing factors

Several ordinary engineering choices combined into the incident. No single choice would have shipped as an incident review finding. The interaction among those choices was the defect.

• HTTP 429 handling used a generic library retry.
• apply_patch had no idempotency key and no receipt check.
• Slot identity was a reused directory, not a unique generation.
• Tool JSON and HTTP were retried as one transport.
• The lab treated free-server idle as a safe recycle signal.
• No test asserted that a worktree belongs to one job id.

None of these choices look reckless in isolation. Together they turn a rate limit into a cross-job write.

Why the HTTP buffer selected apply_patch

Many clients retry the last written bytes on 429. Streaming tool calls keep apply_patch in that buffer longer. run_tests had not flushed when the 429 arrived.

The status code arrived on a reused keep-alive socket. The retry library did not know tool names. It only knew bytes, status, and Retry-After.

Artifact: reproduce the race

The fixture below is a local race simulator. It does not call a hosted model at all. It shows slot recycle plus a buffered retry.

# retry_slot_race.py
# Label: local simulator, not production telemetry.

from dataclasses import dataclass, field
from pathlib import Path
import json
import threading

@dataclass
class Slot:
    slot_id: str
    generation: int = 0
    job_id: str | None = None
    root: Path = field(default_factory=lambda: Path("/tmp/agent-slots"))

    def path(self) -> Path:
        return self.root / self.slot_id

class WorkerPool:
    def __init__(self) -> None:
        self.slots = {"S7": Slot("S7")}
        self.lock = threading.Lock()

    def assign(self, job_id: str, slot_id: str = "S7") -> Slot:
        with self.lock:
            slot = self.slots[slot_id]
            slot.generation += 1
            slot.job_id = job_id
            p = slot.path()
            p.mkdir(parents=True, exist_ok=True)
            (p / ".job").write_text(json.dumps({
                "job_id": job_id,
                "generation": slot.generation,
            }))
            return Slot(slot.slot_id, slot.generation, job_id, slot.root)

    def apply_patch(self, slot: Slot, relpath: str, body: str) -> dict:
        target = slot.path() / relpath
        target.parent.mkdir(parents=True, exist_ok=True)
        meta = json.loads((slot.path() / ".job").read_text())
        if meta["job_id"] != slot.job_id:
            raise RuntimeError("slot_job_mismatch")
        if meta["generation"] != slot.generation:
            raise RuntimeError("stale_generation")
        target.write_text(body)
        return {"ok": True, "generation": slot.generation, "job_id": slot.job_id}

def simulate_race() -> None:
    pool = WorkerPool()
    slot_a = pool.assign("job-A")
    pool.apply_patch(slot_a, "pkg/lock.py", "held_by=A\n")
    buffered = ("pkg/lock.py", "held_by=A\n")
    slot_b = pool.assign("job-B")
    try:
        pool.apply_patch(slot_a, *buffered)
        print("FAIL: replay wrote across jobs")
    except RuntimeError as exc:
        print(f"PASS: replay blocked ({exc})")

if __name__ == "__main__":
    simulate_race()

Run the simulator locally with a plain interpreter.

python3 retry_slot_race.py

The first version without the generation check prints the FAIL line. The durable version must print PASS and refuse the write.

Artifact: retry classifier

Transport retries need an explicit tool-level policy. The table is encoded as code, not as comments in a prompt.

# retry_policy.py

IDEMPOTENT = {"read_file", "list_dir", "git_status", "run_tests_readonly"}
NEVER_RETRY = {"apply_patch", "git_commit", "rm", "mv", "write_file"}
CONDITIONAL = {"run_tests", "install_deps"}

def classify(tool_name: str, http_status: int) -> str:
    if tool_name in NEVER_RETRY:
        return "do_not_retry"
    if http_status == 429 and tool_name in IDEMPOTENT:
        return "retry_after_header"
    if http_status == 429 and tool_name in CONDITIONAL:
        return "retry_only_with_receipt"
    if http_status >= 500 and tool_name in IDEMPOTENT:
        return "retry_once"
    return "do_not_retry"

A unit test pins the destructive apply case.

# test_retry_policy.py

from retry_policy import classify

def test_apply_patch_never_retries_on_429() -> None:
    assert classify("apply_patch", 429) == "do_not_retry"

def test_read_file_may_retry_on_429() -> None:
    assert classify("read_file", 429) == "retry_after_header"

python3 -m pytest test_retry_policy.py -q

Artifact: apply receipts

Idempotency keys belong at the apply boundary itself. The worker stores one receipt per idempotency key. A replay with the same key returns the original receipt.

A replay with a stale generation is rejected. That closed check is the isolation boundary. Directory wipes without a generation number still lose.

# apply_receipts.py

from dataclasses import dataclass
import hashlib
import json
from pathlib import Path

@dataclass(frozen=True)
class ApplyRequest:
    job_id: str
    generation: int
    idempotency_key: str
    relpath: str
    body: str

class ReceiptStore:
    def __init__(self, path: Path) -> None:
        self.path = path
        self.path.parent.mkdir(parents=True, exist_ok=True)
        if not self.path.exists():
            self.path.write_text("{}")

    def _load(self) -> dict:
        return json.loads(self.path.read_text())

    def _save(self, data: dict) -> None:
        self.path.write_text(json.dumps(data, indent=2, sort_keys=True))

    def commit(self, req: ApplyRequest, worktree: Path) -> dict:
        data = self._load()
        prior = data.get(req.idempotency_key)
        if prior:
            if prior["job_id"] != req.job_id:
                raise RuntimeError("key_bound_to_other_job")
            return prior
        digest = hashlib.sha256(req.body.encode()).hexdigest()[:12]
        stamp = json.loads((worktree / ".job").read_text())
        if stamp["generation"] != req.generation:
            raise RuntimeError("stale_generation")
        if stamp["job_id"] != req.job_id:
            raise RuntimeError("slot_job_mismatch")
        target = worktree / req.relpath
        target.parent.mkdir(parents=True, exist_ok=True)
        target.write_text(req.body)
        receipt = {
            "job_id": req.job_id,
            "generation": req.generation,
            "relpath": req.relpath,
            "digest": digest,
            "status": "applied",
        }
        data[req.idempotency_key] = receipt
        self._save(data)
        return receipt

Decision table

Use this table before enabling client retries on an agent worker. The unsafe cells are more important than the safe ones. Copy the unsafe cells into tests first.

Tool	429	5xx	Timeout	Replay safe
apply_patch	no retry	no retry	no retry	same key and generation only
write_file	no retry	no retry	no retry	same key and generation only
read_file	retry after header	retry once	retry once	yes
run_tests	retry with receipt	no retry	no retry	no, tests can mutate caches
git_commit	no retry	no retry	no retry	never

Timeouts are the dangerous sibling of HTTP 429. A timeout after a successful write looks identical to a lost response. The client must not replay apply on timeout.

The client must read the receipt store instead. Unknown outcome is not a license to replay. Receipts exist to resolve that unknown apply outcome.

Durable fix

The durable fix requires four mandatory checks together. Skipping any one check reopens the race.

1. Give every slot a monotonic generation number.
2. Bind every apply to job_id + generation + idempotency_key.
3. Classify tools before any HTTP retry library runs.
4. Treat timeout as unknown, not as a missed request.

The HTTP retry layer stays dumb by design. It retries only when the classifier returns retry_after_header or retry_once. Destructive tools must never enter that retry path.

Slot recycle on the shared server may continue. Recycle is how a free shared server stays cheap. Recycle is safe only when generation checks fail closed.

Detecting the incident in logs

Operators should log slot generation on every tool result. A detector greps for apply receipts whose job_id mismatches the slot stamp. That query is the paging alert for this class.

jq -r 'select(.tool=="apply_patch") | [.job_id,.slot_generation,.receipt_job] | @tsv' agent.jsonl \
  | awk -F'\t' '$1 != $3 { print }'

A nonempty printout means a cross-job apply occurred. Stop the worker pool before more slots recycle. Then restore each worktree from its per-job snapshot.

Rollback

Rollback is a worktree restore from Job B's original snapshot. Do not reverse the diff from Job A's patch blindly. Job B may have edited the same path after the replay.

Keep Job A's receipt in the store for audit. Do not reuse that idempotency key on Job B. Issue a new key only after the generation bump.

Commands for a local gate

Wire the checks into a pre-apply hook. The hook runs on the worker, not in prompt text. The command form below is the worker hook.

python3 check_apply_envelope.py --slot /tmp/agent-slots/S7 --request apply.json

# check_apply_envelope.py
import argparse
import json
from pathlib import Path
import sys

def main() -> int:
    p = argparse.ArgumentParser()
    p.add_argument("--slot", required=True)
    p.add_argument("--request", required=True)
    args = p.parse_args()
    stamp = json.loads((Path(args.slot) / ".job").read_text())
    req = json.loads(Path(args.request).read_text())
    if stamp["job_id"] != req["job_id"]:
        print("slot_job_mismatch", file=sys.stderr)
        return 2
    if stamp["generation"] != req["generation"]:
        print("stale_generation", file=sys.stderr)
        return 3
    if req["tool"] in {"apply_patch", "write_file", "git_commit"}:
        if not req.get("idempotency_key"):
            print("missing_idempotency_key", file=sys.stderr)
            return 4
    print("ok")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

The sample request file looks like this.

{
  "job_id": "job-A",
  "generation": 1,
  "tool": "apply_patch",
  "idempotency_key": "job-A:apply:pkg/lock.py:1",
  "relpath": "pkg/lock.py"
}

Limitations

This design does not stop a model from issuing two different keys. Two different keys still mean two applies. Operators still need a separate patch review gate.

The receipt store is local to the worker disk. A crash before fdatasync can lose a receipt. A lost receipt plus a client retry can still double-apply inside one generation.

The classifier is a denylist plus a small allowlist. Unknown tools default to do_not_retry on purpose. That default can stall a job under load.

It still will not cross-write a neighbor job. The lab did not measure throughput, latency, or model quality. Those numbers are omitted on purpose here.

Who should not use this approach

Skip this design on a dedicated single-tenant runner with one worktree. Slot recycle is not present on that runner. A simpler apply log may suffice in that case.

Skip it when every tool is a pure read. Generic HTTP retry libraries are acceptable in that case. Do not copy the receipt machinery for read_file only.

Skip it for interactive local agents that never queue. The race needs a recycled slot and a buffered retry. Laptops that block one job at a time do not hit this path.

What remains open

Prompt-level instructions do not fix transport retries at all. The classifier must sit in front of the HTTP client. That placement is the actual lesson from the incident.

Shared free servers remain useful for small labs. They require isolation that looks like boring infrastructure. Generation numbers and receipts provide that isolation.

Readers who already run a pooled worker can add the envelope check first. The rest of the harness can follow after the first red test.