Retry Amplification: A Circuit Breaker for Batch Model Jobs

A nightly job walked a folder of 4,000 JSON files and called a model once per file. At 02:14 the upstream started returning 429s. By 02:40 the job had issued roughly 61,000 calls and written zero results. The queue was not slow. It was multiplying itself.

That shape of incident is common in batch model work, and it rarely comes from a broken model. It comes from a retry loop that treats every error as temporary, including errors that mean "stop". The fix is not a bigger retry budget. It is a breaker that refuses to make the next call.

Retry amplification, in three numbers

Amplification is easy to measure after the fact and easy to predict before it. Three numbers tell you whether your job can melt itself:

• Calls per item. One call per item is healthy. Eight calls per item means your loop is the load generator.
• Wall-clock per item. If p95 time per item doubles while p50 stays flat, a subset of items is stuck retrying.
• Identical error signatures. Twenty different errors are noise. One error repeated 4,000 times is a hard stop signal.

The third number is the one most teams never collect, because logs store messages, not normalized signatures.

Why free access changes your math but not your failure modes

Using a free tier, like the free model access and free server option MonkeyCode describes, removes the invoice from the loop. It does not remove wall-clock limits, rate limits, or the cost of your own wasted runs. A runaway loop against free capacity still burns an afternoon.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The breaker below is provider-neutral on purpose. It counts failures, normalizes them, and stops. Whether the call behind it is billed or free only changes how loudly you notice a bad night.

The state machine, in one table

A batch breaker has three states. Write them down before you write code, because half-open logic is where most hand-rolled breakers go wrong.

State	allow()	On success	On failure
CLOSED	always true	reset counters, stay CLOSED	increment; trip at threshold
OPEN	false until cooldown expires	n/a	extend cooldown, capped
HALF-OPEN	true exactly once	close and reset cooldown	reopen immediately

The subtle rule is the last row: a half-open breaker grants one probe, not unlimited probes with a timer attached. If that one probe fails, the breaker reopens without waiting for the threshold again.

A breaker with an injectable clock

Persist state to a file so a restart cannot silently reset a tripped breaker. Inject the clock so tests never sleep.

# batch/breaker.py
"""Stop making calls before the retry loop stops you."""
from __future__ import annotations

import hashlib
import json
import pathlib
import re
import time

_DIGITS = re.compile(r"\d+")


def signature(exc: BaseException) -> str:
    """Normalize an error so identical failures share one fingerprint."""
    text = f"{type(exc).__name__}: {exc}".lower()
    return hashlib.sha256(_DIGITS.sub("n", text).encode()).hexdigest()[:12]


class Breaker:
    def __init__(self, path, *, threshold=3, base_cooldown=30.0,
                 max_cooldown=600.0, repeat_trip=2, clock=time.monotonic):
        self.path = pathlib.Path(path)
        self.threshold = threshold
        self.base_cooldown = base_cooldown
        self.max_cooldown = max_cooldown
        self.repeat_trip = repeat_trip
        self.clock = clock
        self.state = self._load()

    def _default(self) -> dict:
        return {"failures": 0, "opened_at": None, "cooldown": self.base_cooldown,
                "half_open": False, "last_signature": None,
                "signature_repeats": 0, "trips": 0}

    def _load(self) -> dict:
        if self.path.exists():
            return json.loads(self.path.read_text())
        return self._default()

    def _save(self) -> None:
        self.path.write_text(json.dumps(self.state, indent=2, sort_keys=True))

    def allow(self) -> bool:
        s = self.state
        if s["opened_at"] is None:
            return True
        if s["half_open"]:
            return False                      # one probe only
        if self.clock() - s["opened_at"] < s["cooldown"]:
            return False
        s["half_open"] = True
        self._save()
        return True

    def record_success(self) -> None:
        self.state = self._default()
        self._save()

    def record_failure(self, exc: BaseException) -> None:
        s = self.state
        sig = signature(exc)
        s["signature_repeats"] = (
            s["signature_repeats"] + 1 if sig == s["last_signature"] else 1
        )
        s["last_signature"] = sig
        s["failures"] += 1

        storm = s["signature_repeats"] >= self.repeat_trip
        if s["half_open"] or storm or s["failures"] >= self.threshold:
            if s["trips"] > 0:
                s["cooldown"] = min(s["cooldown"] * 2, self.max_cooldown)
            s["opened_at"] = self.clock()
            s["trips"] += 1
            s["half_open"] = False
        self._save()

Signatures, not messages

signature() strips digits before hashing, so attempt 7 failed and attempt 8 failed collapse into one fingerprint. That is what lets repeat_trip catch an error storm long before the plain failure count reaches its threshold. Pick the normalization you can defend: identifiers and UUIDs usually belong in the stripped set, status codes usually do not.

Tests that never sleep

A fake clock keeps the suite fast and makes cooldown behavior exact.

# tests/test_breaker.py
from batch.breaker import Breaker


class Clock:
    def __init__(self): self.now = 0.0
    def __call__(self): return self.now
    def advance(self, seconds): self.now += seconds


def make(tmp_path, clock, **kw):
    return Breaker(tmp_path / "breaker.json", clock=clock, **kw)


def test_trips_after_threshold(tmp_path):
    clock = Clock()
    b = make(tmp_path, clock, threshold=3)
    for _ in range(3):
        assert b.allow()
        b.record_failure(RuntimeError("upstream 429"))
    assert b.allow() is False


def test_half_open_grants_exactly_one_probe(tmp_path):
    clock = Clock()
    b = make(tmp_path, clock, threshold=1, base_cooldown=10.0)
    assert b.allow()
    b.record_failure(RuntimeError("boom"))
    clock.advance(9)
    assert b.allow() is False
    clock.advance(1)
    assert b.allow() is True
    assert b.allow() is False


def test_success_closes_and_resets(tmp_path):
    clock = Clock()
    b = make(tmp_path, clock, threshold=2)
    b.allow(); b.record_failure(RuntimeError("boom"))
    b.allow(); b.record_failure(RuntimeError("boom"))
    clock.advance(30)
    assert b.allow()
    b.record_success()
    assert b.state["trips"] == 0


def test_error_storm_trips_before_threshold(tmp_path):
    clock = Clock()
    b = make(tmp_path, clock, threshold=50, repeat_trip=2)
    b.allow(); b.record_failure(TimeoutError("attempt 7 failed"))
    b.allow(); b.record_failure(TimeoutError("attempt 8 failed"))
    assert b.allow() is False

python -m pytest -q tests/test_breaker.py

Wiring it into a batch runner

The runner holds one rule: check allow() before every call, and never let an exception escape into the next item.

# batch/run.py
import pathlib
from batch.breaker import Breaker

breaker = Breaker(".breaker.json", threshold=3, repeat_trip=2)


def handle(item: pathlib.Path) -> None:
    ...  # your model call, then write the result atomically


def main() -> int:
    done = calls = 0
    for item in sorted(pathlib.Path("inbox").glob("*.json")):
        if not breaker.allow():
            print(f"open: {calls} calls, {done} items done")
            return 2
        calls += 1
        try:
            handle(item)
        except Exception as exc:
            breaker.record_failure(exc)
            continue
        done += 1
        breaker.record_success()
    print(f"closed: {calls} calls, {done} items done")
    return 0

Exit code 2 and the calls/done pair are the whole postmortem. A cron wrapper can page on that code and archive .breaker.json with the run logs.

Two operational details

1. Delete .breaker.json deliberately. A job that resumes six hours later should not inherit a stale cooldown, so make resetting it an explicit step in your runbook.
2. Probe with the cheapest item. When half-open fires, feed it a tiny input first. A probe that OOMs tells you nothing about the upstream.

Health metrics worth paging on

• calls_per_completed_item above 1.2 over 15 minutes.
• trips_total increasing after the job was previously stable for a week.
• signature_repeats for the current run reaching repeat_trip - 1.

If a breaker never trips across months of runs, the threshold is decoration. Lower it in a staging job and confirm the open path still works before you trust it in production.

Decide in ten seconds

Situation	Use a breaker?	Why
Batch job, idempotent items, one upstream	Yes	Safe to stop and resume later
Interactive request path	No	The user needs an answer, not a stopped service
Items that mutate shared state mid-run	Not yet	Make them idempotent first
One-shot script, under a minute	No	Nothing to protect
Multiple upstreams with different limits	Per upstream	One breaker per dependency

Limits and who should skip this

A breaker suppresses the symptom, not the cause. If the upstream is genuinely down for an hour, you get a quiet job instead of a loud one, and quiet jobs get forgotten. Pair the breaker with an alert, or you have built a silencer.

It also hides partial progress. A run that stops at 900 of 4,000 items looks successful unless you report done explicitly, which is why the runner prints both numbers.

Skip this approach if your items are not idempotent, if retries are the only way you recover from a known flaky dependency, or if a human is waiting on each result. In those cases a per-request timeout and a retry budget are the simpler tools.

What to run this week

Add breaker.py and the four tests above to your smallest batch job, then point a staging run at an upstream you can break on purpose. Run the job on disposable infrastructure if you do not want a tripped breaker in your working tree; the free server option in MonkeyCode is one place to try that without touching your laptop's state.

Confirm one thing before you walk away: when the breaker opens mid-run, does your job resume cleanly from the items it already finished? If the answer is no, fix that first. A breaker on top of non-idempotent work just makes the outage smaller and harder to see.