The outage was not the endpoint's fault. It was my retry logic.
I pointed a code-review bot at MonkeyCode's free server and watched it stall the review pipeline in eleven minutes. The server did not go down. It returned 429s, which is what a shared endpoint is supposed to do when a client sends a burst. My client then did the one thing that makes rate limits catastrophic: it retried immediately, in parallel, forever.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Rate limits are a control signal. They tell the client to slow down. My code treated them as a suggestion to try harder.
Here is the timeline. The bot reviewed every open pull request at 9:00 AM sharp. Twenty parallel requests hit the endpoint at once. The server answered the first few, then started returning 429s. The retry loop — three lines, no backoff — re-sent every failed request instantly. Those re-sent requests collided with the next wave of new requests. More 429s. More instant retries. Eleven minutes later, the bot was generating more requests than the CI runner could dispatch, and every response was a 429. The pipeline was not slow. It was dead.
The metrics told the story. Normal traffic was four requests per minute. During the cascade, the bot hit 212 requests in one minute, and 211 of them were 429s. The server was not the bottleneck. The client was.
The naive version looked exactly like what most tutorials show: a for loop, a try, a retry. It worked in demos because demos do not run twenty copies in parallel. In production, twenty copies retrying at the same instant is not a retry. It is an attack.
The failure was a feedback loop, not a server outage. The retry logic amplified a normal control signal into a thundering herd. And the fix is not "get a better endpoint". It is "make the client respect the signal".
The fix has three parts: exponential backoff, jitter, and a circuit breaker. Backoff spreads retries over time. Jitter breaks the synchronization that makes bursts worse. The circuit breaker stops the client from calling an endpoint that is clearly saturated.
# resilient_client.py
import random
import time
from dataclasses import dataclass
class RateLimitError(Exception):
pass
@dataclass
class RetryConfig:
base_delay: float = 0.5
max_delay: float = 30.0
max_retries: int = 5
jitter_factor: float = 0.4
class CircuitBreaker:
def __init__(self, failure_threshold: int = 10, cooldown: float = 60.0):
self.failure_threshold = failure_threshold
self.cooldown = cooldown
self.failures = 0
self.open_until = 0.0
def allow(self) -> bool:
if self.failures >= self.failure_threshold:
if time.monotonic() >= self.open_until:
self.failures = 0
return True
return False
return True
def record_failure(self) -> None:
self.failures += 1
if self.failures == self.failure_threshold:
self.open_until = time.monotonic() + self.cooldown
def record_success(self) -> None:
self.failures = 0
def call_with_recovery(func, config: RetryConfig, breaker: CircuitBreaker):
delay = config.base_delay
for attempt in range(config.max_retries):
if not breaker.allow():
raise RuntimeError("circuit open; endpoint is saturated")
try:
result = func()
breaker.record_success()
return result
except RateLimitError:
breaker.record_failure()
if attempt == config.max_retries - 1:
raise
jitter = random.uniform(0, config.jitter_factor * delay)
time.sleep(delay + jitter)
delay = min(delay * 2, config.max_delay)
raise RuntimeError("unreachable")
The math matters. With a base delay of 0.5 seconds and a doubling factor, the fifth retry waits eight seconds before trying again. That is not slow; that is polite. Add jitter — a random offset of up to 40% of the current delay — and two clients that failed at the same instant will not retry at the same instant. The circuit breaker closes the loop. After ten consecutive failures, it refuses to call the endpoint for a full minute, giving the server room to drain its queue.
I verified the fix with a mock server that returns 429 for two out of every three requests. The naive client — retry immediately, five times — failed every attempt and raised an exception. The resilient client succeeded on the third attempt, because the backoff had stretched the retry window long enough for the mock's rate window to reset.
# test_rate_limit_storm.py
import threading
import time
from http.server import BaseHTTPRequestHandler, HTTPServer
import requests
from resilient_client import CircuitBreaker, RateLimitError, RetryConfig, call_with_recovery
REQUEST_COUNT = 0
class FlakyHandler(BaseHTTPRequestHandler):
def do_GET(self):
global REQUEST_COUNT
REQUEST_COUNT += 1
if REQUEST_COUNT % 3 == 0:
self.send_response(200)
self.end_headers()
self.wfile.write(b"ok")
else:
self.send_response(429)
self.end_headers()
self.wfile.write(b"rate limited")
def log_message(self, *args):
pass
def run_server():
server = HTTPServer(("127.0.0.1", 8765), FlakyHandler)
server.serve_forever()
threading.Thread(target=run_server, daemon=True).start()
time.sleep(0.2)
breaker = CircuitBreaker(failure_threshold=10, cooldown=30.0)
config = RetryConfig(base_delay=0.1, max_delay=2.0, max_retries=5)
def fetch():
response = requests.get("http://127.0.0.1:8765/", timeout=5)
if response.status_code == 429:
raise RateLimitError()
return response.text
result = call_with_recovery(fetch, config, breaker)
print(f"result: {result}")
print(f"total requests sent: {REQUEST_COUNT}")
Run it and you will see a success plus a request count far below the naive version's total. The test is small, but it reproduces the exact failure mode from the incident: a shared endpoint that rejects most of a burst, and a client that either amplifies the burst or absorbs it.
The lesson is general. Any free or shared endpoint — MonkeyCode's free server included — enforces rate limits because it has to. The limits are not a defect. They are the mechanism that keeps the server usable for everyone. The only question is whether your client can survive them. Mine could not.
After the fix, I added one line of monitoring: count 429s per minute and alert when the circuit breaker opens. The alert has fired exactly once since — during a deploy when the bot's config pointed at the wrong endpoint. The pattern worked as designed: the breaker opened, the pipeline queued the work, and the next run succeeded.
Who should not use this pattern? Teams with strict latency SLOs. Backoff and circuit breaking are the opposite of fast. If a user is waiting for a chat response, sleeping eight seconds is not acceptable. This pattern belongs in background jobs, CI pipelines, and batch processes — places where a delayed answer is better than a failed one. For interactive workloads, you need a paid endpoint with a contractual rate limit, or a self-hosted model with no shared queue at all.
The pattern also has a ceiling. It handles transient saturation, not sustained load. If your steady-state request rate exceeds the endpoint's limit, no amount of backoff will help. The circuit breaker will keep the requests from making things worse, but it will not make the endpoint faster. At that point the answer is not better retries. It is a different endpoint.
The real artifact from this incident is not the code. It is the distinction between a retry and a recovery. A retry assumes the failure was temporary. A recovery assumes the system is under stress and acts accordingly. My original code retried. The new code recovers.
If you want to test this pattern against a real shared endpoint, MonkeyCode's free server is a reasonable place to start. Send it a burst, watch the 429s arrive, and see whether your client survives. Mine did not — until it did.
MonkeyCode provides free models that can run this workflow.
Top comments (0)