Retry, Repair, or Replay: A Failure-Triage Tree for Agent Steps

Step 6 of a 9-step agent job times out. The run is holding a checkpoint, an open HTTP call that may or may not have landed, and a retry counter that nobody wired up. The next decision is the expensive one — and most codebases make it with a bare except: retry().

That default is wrong in two directions at once. It hammers dead endpoints, and it re-fires steps that already wrote something into the world.

This post is a glossary, a decision tree, and four worked leaves, plus a dependency-free classifier you can run in under a second. The scenario above is illustrative, not a customer story; the code is the artifact you can actually check.

Terms worth pinning down first

Failure class. Whether the fault is in the transport (timeout, 429, 5xx, connection reset) or in the task (wrong output, wrong field, wrong decision). These need different budgets, not the same for loop.

Side-effect boundary. The first line in a step after which the outside world has changed: a row written, an email sent, a payment authorized. Everything before it is safe to throw away.

Checkpoint. A stable identifier for the last known-good state, granular enough that replay resumes at a step rather than at the job.

Retry. Same input, same prompt, same parameters. Cheap to reason about, only valid before the side-effect boundary.

Repair. Changed input: a narrowed instruction, an added schema, one failing example. Expensive in tokens, appropriate when the output parses but is wrong.

Replay. Re-run from a checkpoint behind an idempotency key, so a duplicate request returns the original result instead of creating a second one.

Poison step. A step whose failure fingerprint is unchanged after the repair budget is spent. Retrying it is a way to convert a bug into an outage.

Reconciliation. The out-of-band read that proves what actually happened downstream. Triage decides what to try; reconciliation decides what is true.

The three branches

Every recovery decision in an agent step reduces to three questions, asked in this order.

1. Is the fingerprint stable and non-transient after repair? If yes, stop. This is a poison step and no branch below improves it.
2. Are there side effects? If yes, the only safe action is replay from a checkpoint with an idempotency key — or abort if no checkpoint exists.
3. Is the transport healthy? Transient and under budget means retry. Transient and over budget means abort: a dead endpoint is not a prompt problem. Non-transient means repair, once or twice, then abort.

Order matters more than the rules themselves. Side effects must be checked before the retry branch, and poison must be checked before replay, or you will replay a step whose model output was garbage in the first place.

Decision table

transient	side effects	checkpoint	repair attempts	action
yes	no	—	0	retry (budget open)
yes	yes	ckpt-17	0	replay behind idempotency key
yes	yes	none	0	abort, reconcile by hand
no	no	—	1	repair once, record prompt hash
no	no	—	2	abort, quarantine
no	yes	ckpt-17	2	abort — poison beats replay

The classifier, runnable

Save as triage.py. Standard library only, CPython 3.10+ for the union syntax.

#!/usr/bin/env python3
"""triage.py - pick a recovery action for a failed agent step."""
from __future__ import annotations

from dataclasses import dataclass
from enum import Enum


class Action(str, Enum):
    RETRY = "retry"
    REPAIR = "repair"
    REPLAY = "replay"
    ABORT = "abort"


RETRY_BUDGET = 3
REPAIR_BUDGET = 2


@dataclass(frozen=True)
class Failure:
    step: str
    transient: bool = False        # timeout, 429, 5xx, reset
    schema_ok: bool = True         # did output match the contract?
    side_effects: bool = False     # did this attempt already write outside?
    checkpoint: str | None = None  # last known-good state id
    attempts: int = 1
    repair_attempts: int = 0
    signature: str = ""            # normalized error fingerprint


def triage(f: Failure) -> tuple[Action, str]:
    if not f.transient and f.repair_attempts >= REPAIR_BUDGET:
        return Action.ABORT, "poison step: fingerprint survived the repair budget"
    if f.side_effects:
        if f.checkpoint:
            return Action.REPLAY, f"re-run from {f.checkpoint!r} behind an idempotency key"
        return Action.ABORT, "side effects with no checkpoint: reconcile by hand"
    if f.transient and f.attempts < RETRY_BUDGET:
        return Action.RETRY, "transport fault, retry budget open"
    if f.transient:
        return Action.ABORT, "transport budget spent; a prompt edit will not fix a dead endpoint"
    if f.repair_attempts < REPAIR_BUDGET:
        return Action.REPAIR, "transport fine, output wrong: narrow the instruction once"
    return Action.ABORT, "no leaf matched: escalate with the trace"


CASES = [
    ("transient read", Failure("fetch", transient=True, attempts=1), Action.RETRY),
    ("transport exhausted", Failure("fetch", transient=True, attempts=3), Action.ABORT),
    ("write with checkpoint",
     Failure("charge", transient=True, side_effects=True, checkpoint="ckpt-17"), Action.REPLAY),
    ("write without checkpoint", Failure("charge", side_effects=True), Action.ABORT),
    ("contract violation", Failure("extract", schema_ok=False), Action.REPAIR),
    ("repair exhausted", Failure("extract", schema_ok=False, repair_attempts=2), Action.ABORT),
    ("poison beats replay",
     Failure("charge", side_effects=True, checkpoint="ckpt-17", repair_attempts=2), Action.ABORT),
]


def main() -> int:
    failures = 0
    for name, failure, want in CASES:
        got, why = triage(failure)
        ok = got is want
        failures += 0 if ok else 1
        print(f"{'ok  ' if ok else 'FAIL'} {name:<24} -> {got.value:<6} {why}")
    return 1 if failures else 0


if __name__ == "__main__":
    raise SystemExit(main())

Run python3 triage.py. Expected output, with the last line being the one that catches the classic mistake — replaying a step whose model never produced valid output:

ok   transient read           -> retry  transport fault, retry budget open
ok   transport exhausted      -> abort  transport budget spent; a prompt edit will not fix a dead endpoint
ok   write with checkpoint    -> replay re-run from 'ckpt-17' behind an idempotency key
ok   write without checkpoint -> abort  side effects with no checkpoint: reconcile by hand
ok   contract violation       -> repair transport fine, output wrong: narrow the instruction once
ok   repair exhausted         -> abort  poison step: fingerprint survived the repair budget
ok   poison beats replay      -> abort  poison step: fingerprint survived the repair budget

If a line differs, the test fails and the tree disagrees with the table above. Fix the tree, not the table.

Four leaves, one worked example each

Leaf 1 — Retry

A geocoding lookup times out on attempt 1 of 3. transient=True, no side effects, no writes. Action: retry with backoff (200ms, 800ms, 3200ms), log the attempt counter, and let the budget run out. The thing you must not do: reuse this leaf for the enrichment step that posts a normalized address to an internal service. That step has a side-effect boundary.

Leaf 2 — Replay

The charge step times out after the provider accepted the request. Whether the charge landed is unknown, which is exactly the state replay exists for. You have ckpt-17, so you re-submit the same request with the same idempotency key and the provider returns the original response instead of a second charge.

class ReplayLedger:
    """Append-only claim file: first writer wins, replayers get False."""

    def __init__(self, path: str) -> None:
        self.path = path

    @staticmethod
    def key(job_id: str, step: str, checkpoint: str) -> str:
        return f"{job_id}:{step}:{checkpoint}"

    def claim(self, k: str) -> bool:
        with open(self.path, "a+", encoding="utf-8") as fh:
            fh.seek(0)
            if k in {line.strip() for line in fh if line.strip()}:
                return False
            fh.write(k + "\n")
            return True

The checkpoint is not optional here. With side_effects=True and checkpoint=None, the classifier aborts — deliberately, because the honest action is a human reading the provider's list endpoint, not a script guessing.

Leaf 3 — Repair

The extract step returns valid JSON that is missing due_date. The transport is fine; the contract is not. Repair means the input changes: append the required schema, add the one failing example, bump repair_attempts, and record a hash of the new prompt so the next failure is comparable to the last. Two repairs maximum. After that the fingerprint is stable and the leaf flips to abort.

Leaf 4 — Abort

The charge step returns 402 three times, transient=False, repair_attempts=2, and it has side effects. Every branch below the poison check would do something plausible and wrong. Action: quarantine the job, keep the trace, page a human, and run reconciliation against the provider before any resubmission. The cost of this leaf is one interrupted job. The cost of skipping it is a duplicate-charge incident.

Where free capacity actually lands in this tree

The repair leaf is the speculative one. Two repair attempts per failing step, across a nine-step job, means inference on inputs that mostly fail — and every repair changes the prompt, which makes the cost hard to amortize with caching.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

That is the honest place to try MonkeyCode's free model access: point only the repair branch at it, keep retry and replay on your existing infrastructure, and compare the abort rate after a week. The other fit is the replay ledger — a tiny append-only process that benefits from the free server option because it needs uptime, not compute. Free-tier quotas, model lists, and server terms change; read the project's current documentation rather than any number quoted in a blog post, including this one, and check the limits before you put a real queue behind it.

Limitations, and who should skip this

• The ledger above is not concurrency-safe. Single writer only; multi-worker setups need a real store with an atomic conditional write.
• Triage classifies failures, not outcomes. A step that returns HTTP 200 with semantically wrong data lands in repair, and only reconciliation catches it.
• The tree assumes idempotent downstream operations or a written compensation action. If neither exists, replay is a duplicate generator and this whole approach is unsafe.
• If your steps cannot checkpoint — long streaming writes, non-resumable file uploads — the side-effects branch collapses to abort and you gain little beyond a vocabulary.
• Do not put financial, medical, or deletion paths behind an automated replay without a reconciliation step that a human reads.

Test plan before production

1. Run python3 triage.py and require exit code 0. All seven assertions are regression tests, especially the last one.
2. Add a fixture for every step in your job, recording real observed values for transient, schema_ok, side_effects, and checkpoint.
3. Point the job at a sandbox downstream that records every write. Inject a timeout after the write and confirm exactly one write appears after replay.
4. Force checkpoint=None on a side-effecting step and confirm the job aborts instead of retrying. This is the single most valuable test in the list.
5. Run 50 jobs with a 5% injected failure rate and compare duplicate-write count against your current except: retry() baseline. Zero is the target; anything else means the ledger is leaking.

Start with the selftest, not the architecture diagram. If the repair leaf turns out to be where your tokens go, running those speculative attempts on MonkeyCode's free model access for a week is a cheap way to get evidence before you change anything that is currently working.