I Blamed the Rate Limit for 48 Hours. The Retry Storm Was Coming From Inside My Code.

I blamed the rate limit first because blaming the platform is the comfortable move. For 48 hours I probed a free model endpoint from a free server, and only when the 429s stopped did I count my own retries. The storm was coming from inside my code, and that realization changed how I review retry logic forever. How much of your outage is the upstream, and how much is the loop you wrote around it?

This week someone on DEV argued that we benchmark models but rarely benchmark the harness around them, and my 48 hours agree with that thesis completely. Disclosure: This article was prepared as part of MonkeyCode's product outreach. My probe ran against MonkeyCode's free model access, and the probe itself was deployed on MonkeyCode's free server option, so the experiment was real even if the relationship deserves that label up front. I am deliberately not quoting quotas, model names, or latency numbers, because I spent most of the run measuring my client, not their platform.

The setup: one probe, one naive loop

The first version of the script was the kind we all ship on a Friday afternoon. Every 60 seconds it sends one tiny completion request, records the status code and elapsed time, and retries up to three times with a fixed one-second pause.

# probe_v1.py — the innocent-looking version
import time
import requests

FREE_MODEL_ENDPOINT = "<your-free-model-endpoint>"  # substitute your own

payload = {"prompt": "ping", "max_tokens": 8, "temperature": 0}

for attempt in range(3):
    try:
        response = requests.post(FREE_MODEL_ENDPOINT, json=payload, timeout=30)
        if response.status_code == 200:
            print("ok", response.elapsed.total_seconds())
            break
        print("status", response.status_code)
    except requests.exceptions.Timeout:
        print("timeout, trying again")
    time.sleep(1)

Nothing about this loop looks dangerous, which is exactly how a storm begins. There is no cap on total time, no jitter, and no respect for whatever the server is trying to tell me. To make the failure visible, I need to show you what it turned into.

What broke: eleven failures became 340 requests

On hour 31 the endpoint started answering with 429s and the occasional timeout. Because several probe workers shared the same failure window, every hiccup was multiplied across retries, and the whole pipeline went quiet for fourteen minutes.

What came back	What my v1 client did	What I'd do instead
429 Too Many Requests	Retried immediately after 1s	Respect Retry-After or back off hard
503 Service Unavailable	Fixed 1s pause, 3 attempts	Exponential backoff with full jitter
Timeout after 30s	Waited 1s and re-sent everything	Re-send for reads; queue writes elsewhere
Connection reset	Retried instantly	Back off, because the server may be mid cold-start

How many of those 340 requests were necessary? Roughly eleven, one per upstream failure. The other 329 were my client politely proving that it had not read the error message, and that gap is the real story. Would I have noticed without a counter on retries? Probably not for another week.

The fix: treat retries as a budget, not an emotion

The second version introduced three rules: full jitter, a hard attempt cap, and Retry-After as a specification rather than a suggestion. Jitter breaks the synchronization that makes parallel workers retry in lockstep, while the cap bounds the damage when the upstream is genuinely gone.

# probe_v2.py — jitter, caps, and listening to the error
import random
import time
import requests

FREE_MODEL_ENDPOINT = "<your-free-model-endpoint>"
MAX_ATTEMPTS = 4
BASE_DELAY = 0.5   # seconds
MAX_DELAY = 8.0    # seconds

def delay_before(attempt: int) -> float:
    cap = min(BASE_DELAY * (2 ** attempt), MAX_DELAY)
    return random.uniform(0, cap)

payload = {"prompt": "ping", "max_tokens": 8, "temperature": 0}

response = None
for attempt in range(MAX_ATTEMPTS):
    try:
        response = requests.post(FREE_MODEL_ENDPOINT, json=payload, timeout=30)
    except requests.exceptions.Timeout:
        time.sleep(delay_before(attempt))
        continue
    if response.status_code == 429:
        retry_after = response.headers.get("Retry-After")
        wait = float(retry_after) if retry_after else delay_before(attempt)
        time.sleep(min(wait, 30))
        continue
    if response.status_code >= 500:
        time.sleep(delay_before(attempt))
        continue
    break

if response is None or response.status_code != 200:
    print("gave up after", MAX_ATTEMPTS, "attempts")

Inside the same change I added a metric I had never tracked before: amplification, which is outbound requests divided by successful responses. A calm run sits near 1.0, and my storm window hit roughly 31. If you monitor latency but not amplification, you are watching the symptom while the disease retries politely in the background.

Reproduce the storm locally, before touching a real model

The trick that made this experiment repeatable was a fake upstream that returns 429 thirty times in a row. Point your client at 127.0.0.1:8765, and you can validate the retry logic in seconds instead of waiting for hour 31.

# fake_upstream.py — pretend to be an overloaded model endpoint
from http.server import BaseHTTPRequestHandler, HTTPServer

class UpsetUpstream(BaseHTTPRequestHandler):
    count = 0

    def do_POST(self):
        UpsetUpstream.count += 1
        if UpsetUpstream.count <= 30:
            self.send_response(429)
            self.send_header("Retry-After", "1")
            self.end_headers()
            return
        self.send_response(200)
        self.end_headers()
        self.wfile.write(b'{"ok":true}')

if __name__ == "__main__":
    HTTPServer(("127.0.0.1", 8765), UpsetUpstream).serve_forever()

The same pattern works for timeouts, resets, and slow responses, and it should live in your repo as a test fixture. This is the test that would have caught my v1 in ten minutes, and I wish I had written it before hour 31. Cheap simulation beats expensive production surprises every time.

What I'd repeat, and what I'd skip

• Start every new client with full jitter and a hard attempt cap, even if the upstream looks healthy.
• Log amplification as a first-class metric, grouped by failure window and by status code.
• Test retry behavior with a scripted fake upstream before pointing anything at a shared endpoint.
• Respect Retry-After whenever the header exists; the server is telling you its recovery time.

What I would skip: tuning the numbers to match my exact 48-hour window. Your payload size, your timeout, and your concurrency all change the right parameters, so copy the rules and not the constants.

Honest limitations

A good backoff hides outages as easily as it prevents them, so this approach is not a monitoring replacement. A genuinely dead endpoint, a drifting model, or a server that sleeps between requests will still break your pipeline, and you still need a dead-letter queue plus human eyes on the trend lines.

Who should not copy this posture: teams with a hard response-time SLO, because a free server with a sleeping scheduler is the wrong substrate for customer-facing guarantees. If you already have paid concurrency under contract, a retry loop tuned for a free tier will feel needlessly slow, and your amplification dashboard will look embarrassing.

The takeaway

Models will drift, servers will sleep, and platform policies will change; the only layer you fully control is what your client does in the gap. My next experiment starts with amplification as the headline metric and a fake upstream in the test suite, and I will blame the platform only after I have counted my own requests. What is the worst amplification number you have accidentally shipped? I would genuinely love to hear that I am not alone in the three-hundred-request club.

Comments

[Kane Lim]

This is a great example of why retry behavior should be treated as part of system reliability, not just error handling.

The amplification metric is particularly interesting. Most teams monitor request rate, latency, error rate, and availability, but rarely ask: “How many downstream requests did one logical operation actually generate?” A rising request-to-success ratio can expose retry storms before they become an obvious upstream outage.

I’d take the pattern one step further and make the retry policy explicit at the architectural level:

Retryable vs. non-retryable errors should be classified rather than relying on status-code ranges alone.
Retry budgets should be enforced across workers, not only per request.
Exponential backoff + full jitter should be the default for transient failures.
Retry-After should have bounded validation rather than blindly trusting an upstream value.
Circuit breakers/bulkheads can prevent a failing dependency from consuming the entire worker pool.
For LLM workloads, idempotency and request semantics deserve special attention before retrying expensive operations.

I also really like the fake-upstream approach. Reliability logic should be tested against deterministic failure modes 429 bursts, 5xx responses, connection resets, slow responses, and partial failures rather than waiting for production to provide the test case.

One metric I'd add alongside amplification is retry contribution to latency: how much of the end-to-end latency/SLO budget is being consumed by retries versus the original request. That makes the operational impact much easier to quantify.

The bigger lesson is excellent: don't benchmark only the model or API; benchmark the control loop around it. The client, queue, retry policy, concurrency, timeout, and observability layer can completely change the behavior of an otherwise healthy dependency.

This is exactly the kind of engineering problem I enjoy working on AI infrastructure, automation, reliability, and observability. I’m always interested in connecting with developers thinking deeply about these problems.