My AI Agent Hit a Rate Limit at Midnight and Retried 11,000 Times. My "Smart" Retry Logic Was the Dumbest Code I've Shipped.

The email from my API provider was polite, which somehow made it worse.

"We've detected unusual traffic from your account and applied a temporary restriction while we review."

I'd been asleep. My agent hadn't. Sometime after midnight, one downstream API started returning 429s, and the retry logic I'd written in twenty minutes — the logic I was genuinely proud of — turned a small transient hiccup into an 11,000-request denial-of-service attack on my own vendor.

This is the post-mortem. It's also the article I wish someone had written before I shipped an agent to production, because "add retries" is advice that sounds responsible and is actively dangerous without the rest of the story.

The setup: a modest agent doing a modest job

Nothing exotic. A Python agent running on a Raspberry Pi 5 that pulls in new support tickets, classifies them, drafts responses, and files everything into a spreadsheet. It calls an LLM API for the classification and drafting, plus a ticketing API for reads and writes. Maybe 200–400 API calls a day under normal conditions.

The retry logic looked like every tutorial I'd ever read:

def call_with_retry(fn, max_attempts=5):
    for attempt in range(max_attempts):
        try:
            return fn()
        except RateLimitError:
            time.sleep(2 ** attempt)  # exponential backoff!
            continue
    raise RuntimeError("gave up")

Exponential backoff. I felt sophisticated. Do you see the four bugs in those six lines? I couldn't see any of them for three months.

What actually happened at midnight

At 00:14, the ticketing API had a bad deploy and started rate-limiting aggressively. My agent was mid-batch processing 60 tickets. Here's the cascade:

1. Each ticket's API call failed with 429, backed off, retried — five attempts each, per call.
2. But a single "ticket" involves multiple API calls (fetch, classify, draft, update, log). Each had its own independent retry budget. One ticket could generate 15+ requests.
3. The batch loop had no global awareness. Ticket 1 exhausting its retries didn't slow down ticket 2 — if anything, all those failed retries increased the request rate against an already-throttled endpoint, guaranteeing tickets 2–60 also failed.
4. And the killer: my cron job had restart-on-failure semantics via systemd. When the batch finally crashed, systemd restarted it. The agent, having no memory that it had just been rate-limited into oblivion, happily started the batch again.

By 6 AM: ~11,000 requests, account flagged, three hours of my morning spent on a support call proving I wasn't a crypto scraper.

The four bugs, named properly

Bug 1: Backoff without jitter. Every retry used exactly 2^attempt seconds. When you have parallel calls (and even sequential calls across a batch), synchronized retries pile up in lockstep — the thundering herd problem. The fix is one line: time.sleep(2 ** attempt * random.uniform(0.5, 1.5)). Everyone knows about jitter. I knew about jitter. I still didn't add it, because in my tests, everything was sequential and it "worked."

Bug 2: No retry budget. Each call site got its own five attempts, and nothing tracked total retries across the process. The right mental model: retries are a budget for the whole system, not a per-call allowance. I now keep a global counter; once the process has done, say, 20 retries in a rolling minute, everything fails fast instead of retrying.

Bug 3: No circuit breaker. After the 50th consecutive 429, my code was still politely backing off and trying again. A circuit breaker — trip after N consecutive failures, stop calling entirely for M seconds, then probe with a single request — would have capped the damage at dozens of requests instead of thousands. It's ~30 lines of Python and it's the single highest-leverage reliability pattern I've added to my agent stack.

Bug 4: Retries + automatic restarts = infinite loop. This is the one nobody warns you about. systemd's Restart=on-failure is also good advice, and combining two pieces of good advice gave me a machine that could never stop failing. The fix: the agent now writes a "cooling off" state file when it detects sustained rate limiting, and exits with a success code plus a scheduled delay — or, on the systemd side, I set StartLimitIntervalSec / StartLimitBurst so rapid restart loops are refused. If you take one thing from this article: your process supervisor doesn't know your API is angry. Make sure either your supervisor rate-limits restarts, or your agent does.

What the fixed version looks like

The whole retry layer is now one small module every call goes through:

class APICaller:
    def __init__(self):
        self.failures = 0          # consecutive failures
        self.open_until = 0        # circuit breaker
        self.retry_times = deque() # rolling retry budget

    def call(self, fn, max_attempts=3):
        if time.time() < self.open_until:
            raise CircuitOpenError(f"breaker open until {self.open_until}")
        for attempt in range(max_attempts):
            if len(self.retry_times) > 20:
                self.retry_times.popleft()
            if len(self.retry_times) >= 20 and time.time() - self.retry_times[0] < 60:
                raise RetryBudgetExhausted()
            try:
                result = fn()
                self.failures = 0
                return result
            except RateLimitError as e:
                self.failures += 1
                self.retry_times.append(time.time())
                if self.failures >= 10:
                    self.open_until = time.time() + 300  # 5 min cooldown
                    raise CircuitOpenError("tripped")
                retry_after = getattr(e, "retry_after", None)
                delay = retry_after or (2 ** attempt) * random.uniform(0.5, 1.5)
                time.sleep(min(delay, 60))
        raise GaveUpError()

Details that matter:

• Honor Retry-After. If the API tells you when to come back, that number beats any backoff formula you invented. This alone would have prevented most of the incident — the 429 responses included the header, and I was ignoring it.
• Cap the max delay. 2^attempt grows fast; nothing should ever sleep more than ~60s inside a call, or you get agents hanging for minutes per request.
• Distinguish retryable from fatal. A 429 or 503 is retryable. A 401 (bad key) or 400 (malformed request) will fail identically forever — retrying those is pure waste. My original code retried everything.
• Log every retry. I had no idea this was happening for six hours because retries were invisible. Every retry now emits a structured log line, and my watchdog alerts me if retry volume crosses a threshold.

The part I still got wrong afterward

Honesty section: after fixing all four bugs, I got cocky and removed the batch size limit, reasoning that the circuit breaker made it safe. Two weeks later a different API (the LLM one) had an outage, the breaker tripped correctly, and my agent spent the cooldown doing nothing — then processed the entire 300-ticket backlog the instant the breaker closed, spiking costs and nearly tripping the breaker again. Circuit breakers control failure; they don't control recovery. I now ramp back in after a breaker closes (10% of normal batch, then 50%, then full) and I keep the batch cap. Resilience patterns interact, and "I added the fix" is not the same as "the system is fixed."

The checklist I wish I'd had

1. Jitter on every backoff, always.
2. A global retry budget, not per-call retries.
3. A circuit breaker with a cooldown, plus a ramp-up on recovery.
4. Honor Retry-After headers.
5. Never retry 4xx errors except 429.
6. Constrain your process supervisor's restart rate.
7. Log retries as first-class events and alert on volume.
8. Test it: point your agent at a mock that returns 429 forever and watch what it does. I finally did this, and it took nine minutes to find a fifth bug (a retry inside a retry inside a helper).

Agents are different from normal scripts because they run unattended, make decisions, and fail in ways that compound. Your retry logic isn't a detail — it's the difference between a rough night and a suspended account.

I write up the specific playbooks in The Solo Operator's AI Agent Playbook — code LAUNCH90 at checkout makes it $1.90. If it doesn't save you 5 hours in week one, reply to the receipt for a refund.