Failure Injection for Agent Loops: A Replayable Cassette Workshop

Most agent loops do not have a degradation plan; they have a retry wrapper and an optimistic assumption. This workshop replaces that assumption with rehearsal: students build a cassette-backed fault injector that replays quota exhaustion, timeouts, and malformed tool calls entirely offline. The whole 90-minute session runs on a laptop with no network access after the first recording pass. Below is the schedule, the reference harness, and the assertions that define a passing submission.

Why rehearsal beats monitoring for quota-limited endpoints

Monitoring tells you that a loop failed at 03:00; a cassette harness tells you what your loop does before it ever runs in production. When many exercises share one rate-limited endpoint, failures stop being rare events and become ordinary traffic. Three patterns dominate in that setting: a hard quota rejection, a transient timeout, and a schema-invalid tool call that arrives as a successful HTTP response.

This is where the free-tier framing becomes practical rather than promotional. MonkeyCode's operator describes a free tier that includes model access (the outreach brief cites 10 million free tokens) and a free server option, and the project is described as open source. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Treat every availability number as operator-supplied and re-check current terms before you depend on them, because the workshop method below does not change if the quota changes.

The free server option matters here for exactly one narrow reason. If your runner process can be recycled between exercises, then process lifetime is a fault you must model, not an assumption you may keep. That single reframing is the difference between a harness that survives a mid-session restart and one that loses partial work silently.

The artifact: a clock-injected harness in three files

The starter repository has three files. Students receive the first two, and they write the third themselves, because the tests are the deliverable.

1. harness.py — the loop under test

# harness.py
import time

class QuotaExhausted(RuntimeError):
    """Provider said stop. Carries Retry-After when the header was present."""
    def __init__(self, retry_after=None):
        super().__init__("quota_exhausted")
        self.retry_after = retry_after

class TransientError(RuntimeError):
    """Timeouts and 5xx responses: retryable by default."""

class BadToolCall(RuntimeError):
    """Response arrived, but the tool-call JSON did not fit the schema."""

def fault_transport(cassette, faults):
    """cassette: dict op -> response. faults: consumed one item per call."""
    plan = list(faults)
    calls = {"n": 0}

    def transport(payload):
        calls["n"] += 1
        fault = plan.pop(0) if plan else None
        if fault is not None:
            raise fault
        return cassette[payload["op"]]

    transport.calls = calls
    return transport

def run_steps(transport, ops, policy, sleep=time.sleep):
    outcomes = []
    for op in ops:
        attempt = 0
        while True:
            attempt += 1
            try:
                outcomes.append({"op": op, "status": "ok",
                                 "value": transport({"op": op}),
                                 "attempts": attempt})
                break
            except BadToolCall:
                outcomes.append({"op": op, "status": "degraded",
                                 "reason": "schema", "attempts": attempt})
                break
            except TransientError:
                if attempt >= policy["max_attempts"]:
                    outcomes.append({"op": op, "status": "degraded",
                                     "reason": "timeout", "attempts": attempt})
                    break
                sleep(policy["base_delay"] * 2 ** (attempt - 1))
            except QuotaExhausted as exc:
                outcomes.append({"op": op, "status": "stopped",
                                 "reason": "quota", "retry_after": exc.retry_after})
                return outcomes
    return outcomes

2. cassettes.py — record once, replay forever

# cassettes.py
import hashlib, json, pathlib

class CassetteMiss(Exception):
    pass

def fingerprint(payload: dict) -> str:
    canonical = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()[:16]

class CassetteStore:
    def __init__(self, root="cassettes", mode="replay"):
        assert mode in {"record", "replay"}
        self.root = pathlib.Path(root)
        self.root.mkdir(exist_ok=True)
        self.mode = mode

    def call(self, transport, payload: dict) -> dict:
        path = self.root / f"{fingerprint(payload)}.json"
        if self.mode == "replay":
            if not path.exists():
                raise CassetteMiss(f"no cassette for {payload.get('op')}")
            return json.loads(path.read_text())["response"]
        response = transport(payload)
        path.write_text(json.dumps({"request": payload, "response": response}, indent=2))
        return response

3. test_policy.py — the graded deliverable

# test_policy.py
from harness import (BadToolCall, QuotaExhausted, TransientError,
                     fault_transport, run_steps)

POLICY = {"max_attempts": 3, "base_delay": 0.01}
CASSETTE = {"plan": {"steps": ["read", "patch"]}, "patch": {"ok": True}}

def test_transient_is_retried_once_then_succeeds():
    transport = fault_transport(CASSETTE, [TransientError, None])
    out = run_steps(transport, ["plan"], POLICY, sleep=lambda _: None)
    assert [o["status"] for o in out] == ["ok"]
    assert out[0]["attempts"] == 2

def test_quota_stops_the_loop_and_keeps_partial_work():
    transport = fault_transport(CASSETTE, [None, QuotaExhausted(retry_after="2")])
    out = run_steps(transport, ["plan", "patch"], POLICY, sleep=lambda _: None)
    assert [o["status"] for o in out] == ["ok", "stopped"]
    assert out[1]["retry_after"] == "2"

def test_bad_tool_call_degrades_without_retrying():
    transport = fault_transport(CASSETTE, [BadToolCall])
    out = run_steps(transport, ["plan"], POLICY, sleep=lambda _: None)
    assert out[0]["status"] == "degraded"
    assert transport.calls["n"] == 1

The full suite finishes in well under a second because the sleep function is injected, which is the point of the exercise.

The 90-minute schedule

Minutes	Segment	Student output
0-10	Framing: three failure classes and why monitoring arrives late	Written list of their own loop's failure modes
10-25	Exercise 1: record one golden transcript to cassettes/	Five cassette files committed
25-40	Exercise 2: replay offline and diff two runs	Proof that replay is byte-identical after stripping volatile fields
40-65	Exercise 3: inject quota, timeout, and bad schema	Three failing tests, then three passing ones
65-80	Exercise 4: write the degradation policy and encode it	The policy table below, expressed as assertions
80-90	Debrief: red-team a neighbour's policy	One adversarial case that breaks their loop

Exercise notes that keep the room moving

• In Exercise 1, strip timestamps and request IDs before committing, or replay determinism fails for reasons unrelated to your loop.
• In Exercise 2, assert equality on the outcome list, not on raw response bodies, because provider payloads drift more often than your control flow.
• In Exercise 3, keep one fault list per test; mixing a quota fault with a timeout fault in one list teaches the wrong lesson about causality.
• In Exercise 4, require a numeric bound on total wait time, otherwise a student can pass every assertion with an unbounded backoff.

The degradation policy, before and after

Failure	Detection	Policy	Budget effect	Required log field
Quota rejection with Retry-After	Header present	One retry after the stated delay, then stop the run	At most one extra call	retry_after
Quota rejection without a header	Header absent	Stop immediately, no retry	Zero extra calls	reason=quota
Timeout	Transport exception	Up to max_attempts, then degrade	Bounded by policy	attempts
Schema-invalid tool call	Validator rejects JSON	Repair once with a stricter instruction, then mark degraded	One repair call	reason=schema
Slow first token	Elapsed time over threshold	No retry; reduce concurrency instead	Unchanged call count	elapsed_ms

Four invariants every submission must hold

1. Call budget: for a scripted op list, the transport is invoked no more times than the policy allows.
2. Replay determinism: the same cassette and policy produce an identical outcome list, ignoring declared volatile fields.
3. Partial work survival: a quota stop returns all completed steps, not an empty list.
4. Bounded waiting: total injected clock time stays under the policy's declared maximum.

Limitations, and who should skip this workshop

The method has real edges, and pretending otherwise would waste your time. A cassette records responses, not latency distributions, so throughput questions stay unanswered unless you record timings separately. Replay also cannot detect that a provider changed its behaviour last week, because the whole point is to serve the old bytes. The free-tier figures quoted above are operator-supplied and can change without notice, so treat them as the workshop's starting conditions rather than a permanent guarantee.

If your endpoint has a paid SLA, existing chaos tooling, and cheap idempotent calls, this harness is ceremony without benefit. If your correctness depends on live streaming semantics such as token timing, prompt caching behaviour, or log probabilities, replay-only testing is insufficient by construction. Finally, never commit real prompts, customer data, or credentials into cassettes/; redact first, and assume the directory will be published someday.

What to take back to your own repository

The transferable lesson is not the file layout, but the habit of treating availability as a tested input rather than a background hope. Teams that rehearse quota exhaustion tend to discover two things quickly: their retry loop spends budget it never needed, and their partial results are lost in a code path nobody had read. Both are cheap to fix once a cassette makes them reproducible in under a second.

If you want somewhere low-volume to record that first golden transcript before you buy a key, the free tier described earlier is designed for exactly this kind of rehearsal; record once, replay forever, and bring assertions sharp enough to break your own loop.

MonkeyCode provides free models that can run this workflow.