The Retry Loop Passed Locally. On the Free Server It Became a Rate-Limit Storm.

The worker had been running for six hours before the first 429 appeared in the logs. My first instinct was to blame the API provider, because the code had passed every local test and the diff looked completely reasonable. Then I noticed something worse: the retry loop was retrying, but the timestamps between attempts were almost exactly one second apart, every single time. That gap was the clue I almost scrolled past.

The setup

Last week I asked a free model in MonkeyCode to write a small worker that pulls events from an external API and stores them in a local queue. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The model returned a compact sync loop, and I deployed it to MonkeyCode's free server option without reviewing the failure path carefully. Local runs against a mock endpoint looked perfect, so the worker went straight into the scheduled job.

The symptom

The first sign was a pile of 429 responses from the API provider, and the second sign was that the whole server started timing out on unrelated requests. My worker was sharing an egress IP with every other tenant on that free server, and my retry loop was hammering the API from all of them at once. The provider did not just throttle my worker; it throttled the entire IP range, which meant innocent neighbors got blocked too. How do you debug a loop that never fails and never succeeds? You start with the timestamps.

Step 1: Read the timestamps, not the messages

The log said "retrying in 1s" over and over, and I had skimmed it as healthy backoff. When I actually computed the gaps between consecutive log lines, they were all 1.0 seconds, which is not exponential backoff at all. A tiny Python script made the pattern obvious:

import re
from datetime import datetime

pattern = re.compile(r"\[(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})\] retrying")
last = None
for line in open("worker.log"):
    match = pattern.search(line)
    if not match:
        continue
    now = datetime.strptime(match.group(1), "%Y-%m-%d %H:%M:%S")
    if last:
        print(f"gap: {(now - last).total_seconds():.1f}s")
    last = now

The output was a long column of gap: 1.0s, repeated thousands of times. The message said "retrying," and the data said "spinning."

Step 2: Reproduce in isolation

I extracted the retry loop into a standalone script and pointed it at a tiny mock server that always returns 429. The bug became obvious within seconds: the loop had no attempt cap and a fixed one-second sleep, and it never read the Retry-After header. Here is the shape of what the model generated:

import time
import requests

def pull_events(url):
    while True:  # retry forever
        try:
            response = requests.get(url, timeout=10)
            response.raise_for_status()
            return response.json()
        except requests.HTTPError as error:
            if error.response.status_code == 429:
                time.sleep(1)  # fixed delay, ignores Retry-After
                continue
            raise

The code was simple, readable, and wrong in a way that only shows up under load. Locally, one worker retrying once per second is invisible. On a shared egress IP, a few dozen tenants doing the same thing creates a synchronized thundering herd, and the API provider responds by blocking the whole range.

Step 3: Check the environment, not just the code

I compared the egress IP from my laptop and from the free server, and the difference explained everything:

curl -s https://api.ipify.org   # laptop: 203.0.113.42
# free server: 198.51.100.17

A single retry loop is a minor nuisance on a dedicated IP. On a shared IP, it is a public nuisance, because every tenant's traffic looks identical to the API provider. The fix had to be polite not just for my worker, but for everyone behind the same address.

The fix

import random
import time
import requests

MAX_ATTEMPTS = 5

def pull_events(url):
    for attempt in range(MAX_ATTEMPTS):
        try:
            response = requests.get(url, timeout=10)
            response.raise_for_status()
            return response.json()
        except requests.HTTPError as error:
            if error.response.status_code != 429:
                raise
            delay = _retry_delay(attempt, error.response)
            print(f"attempt {attempt + 1} hit 429; waiting {delay:.1f}s")
            time.sleep(delay)
    raise RuntimeError("API kept returning 429 after 5 attempts")

def _retry_delay(attempt, response):
    retry_after = response.headers.get("Retry-After")
    if retry_after:
        return float(retry_after)
    base = min(2 ** attempt, 30)
    return base + random.uniform(0, base * 0.1)

Three changes mattered: a hard cap on attempts, respect for Retry-After, and jitter so retries do not synchronize across tenants. The jitter is the part that feels optional until you watch thirty workers retry on the same second.

The artifact: a test that locks the behavior

I wrote a small test so the next model-generated retry loop cannot silently regress:

from unittest.mock import Mock, patch

def test_backoff_respects_retry_after():
    response = Mock()
    response.headers = {"Retry-After": "7"}
    assert _retry_delay(0, response) == 7.0

def test_backoff_grows_with_jitter():
    response = Mock()
    response.headers = {}
    with patch("random.uniform", return_value=0):
        delays = [_retry_delay(i, response) for i in range(5)]
    assert delays == [1.0, 2.0, 4.0, 8.0, 16.0]

Now the retry policy is part of the test suite, not a vibe in the review. If a future model rewrites the worker and drops the jitter, the test fails before the API provider does.

The reusable checklist

• Compute the gaps between log lines before reading the messages.
• Reproduce the failure against a mock that always returns 429.
• Compare the egress IP and rate limits between local and server.
• Read Retry-After and the X-RateLimit-* headers before choosing a delay.
• Add jitter and a cap, then write a test that proves both.

Limitations

This approach is overkill if your API is internal and your retries are rare, because a fixed one-second sleep will never hurt anyone. It is also not a substitute for reading the provider's actual rate-limit documentation, because Retry-After is a hint, not a contract, and some APIs use sliding windows that no amount of jitter can fix. And if your workload is genuinely bursty, the right answer is a queue with a rate limiter, not a cleverer sleep.

The takeaway

The retry loop looked correct in the diff, and that is exactly why it survived review. The free server did not break my code; it exposed the difference between code that works for one user and code that works for a hundred users sharing one address. Next time you generate a worker with a free model, ask it to show you the failure path, then run the reproduction before you deploy. The model will write the happy path for free; the debugging is still yours.