Retry Amplification: Why Your LLM Client Makes Outages Worse

At 3 AM, the alert fired. Queue age hit 45 seconds. You opened the logs. Every upstream call returned 429. Your client retried instantly. Ten thousand retries in one minute. The endpoint stayed down. Your client made it worse.

This is retry amplification. A small outage becomes a traffic storm. Your client is the amplifier. We can prove it. We can fix it. On free infrastructure.

MonkeyCode offers a free server and free model access. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free tier is a lab. Use it to run this drill. Verify current limits in the docs. They change.

The Experiment

We build a small topology. A client sends prompts. A proxy injects faults. The proxy forwards to a real model endpoint. We run two clients. One uses fixed retries. One uses exponential backoff. We compare.

client.py --> fault_proxy.py :8080 --> MonkeyCode model API

Stage 1: Provision the Free Server

SSH into your MonkeyCode free server.

ssh root@<server-ip>
uname -a
nproc
free -h

Verification: you see kernel, CPU, memory. Record them.

Stage 2: Write the Fault Proxy

The proxy returns 429 during a fault window. After the window, it forwards to the model. This simulates a temporary outage.

#!/usr/bin/env python3
import json, os, time, random, urllib.request, urllib.error
from http.server import BaseHTTPRequestHandler, HTTPServer

API_KEY = os.environ['MC_API_KEY']
BASE_URL = os.environ['MC_BASE_URL']
MODEL = os.environ['MC_MODEL']
FAIL_WINDOW = float(os.environ.get('FAIL_WINDOW', '30'))
FAIL_RATE = float(os.environ.get('FAIL_RATE', '1.0'))
start_time = time.time()

class Proxy(BaseHTTPRequestHandler):
    def do_POST(self):
        elapsed = time.time() - start_time
        if elapsed < FAIL_WINDOW and random.random() < FAIL_RATE:
            self.send_response(429)
            self.send_header('Content-Type', 'application/json')
            self.end_headers()
            self.wfile.write(b'{"error":"simulated throttling"}')
            return
        length = int(self.headers.get('Content-Length', 0))
        body = self.rfile.read(length)
        req = urllib.request.Request(
            f'{BASE_URL}/v1/chat/completions',
            data=body,
            headers={
                'Authorization': f'Bearer {API_KEY}',
                'Content-Type': 'application/json',
            },
        )
        try:
            with urllib.request.urlopen(req, timeout=30) as resp:
                data = resp.read()
            self.send_response(resp.status)
            self.send_header('Content-Type', 'application/json')
            self.end_headers()
            self.wfile.write(data)
        except urllib.error.HTTPError as e:
            self.send_response(e.code)
            self.end_headers()
            self.wfile.write(e.read())

HTTPServer(('127.0.0.1', 8080), Proxy).serve_forever()

Set environment variables.

export MC_API_KEY='your-key'
export MC_BASE_URL='https://api.example.com'
export MC_MODEL='your-model'
export FAIL_WINDOW=30
export FAIL_RATE=1.0

Start the proxy.

nohup python3 fault_proxy.py > /tmp/proxy.out 2>&1 &

Verification: pgrep -f fault_proxy.py shows a PID.

Stage 3: Write Two Clients

The first client retries with a fixed one-second delay. The second uses exponential backoff with jitter.

#!/usr/bin/env python3
import json, sys, time, random, urllib.request, urllib.error

URL = 'http://127.0.0.1:8080/v1/chat/completions'
PROMPT = 'Say hello in one sentence.'
MAX_RETRIES = 5

def call_once():
    payload = json.dumps({
        'messages': [{'role': 'user', 'content': PROMPT}],
        'max_tokens': 10,
    }).encode()
    req = urllib.request.Request(URL, data=payload, headers={'Content-Type': 'application/json'})
    start = time.time()
    try:
        with urllib.request.urlopen(req, timeout=30) as resp:
            data = resp.read()
        return resp.status, time.time() - start, len(data)
    except urllib.error.HTTPError as e:
        return e.code, time.time() - start, len(e.read())
    except Exception as e:
        return 502, time.time() - start, 0

def fixed_retry():
    for attempt in range(MAX_RETRIES):
        status, lat, size = call_once()
        print(f'attempt={attempt+1} status={status} latency={lat:.2f}s')
        if status == 200:
            return True
        time.sleep(1)
    return False

def backoff_retry():
    for attempt in range(MAX_RETRIES):
        status, lat, size = call_once()
        print(f'attempt={attempt+1} status={status} latency={lat:.2f}s')
        if status == 200:
            return True
        sleep = min(2 ** attempt, 15) + random.uniform(0, 1)
        print(f'  sleeping {sleep:.2f}s')
        time.sleep(sleep)
    return False

if __name__ == '__main__':
    mode = sys.argv[1] if len(sys.argv) > 1 else 'fixed'
    t0 = time.time()
    ok = fixed_retry() if mode == 'fixed' else backoff_retry()
    elapsed = time.time() - t0
    print(f'mode={mode} success={ok} total_time={elapsed:.2f}s')

Save it as client.py.

Stage 4: Run the Drill

The fault window is 30 seconds. The fixed client finishes in about 5 seconds. It will hit 429 every time. The backoff client waits longer. Its fifth attempt happens after 30 seconds. It should succeed.

Run the fixed client.

python3 client.py fixed > /tmp/fixed.log
cat /tmp/fixed.log

Run the backoff client.

python3 client.py backoff > /tmp/backoff.log
cat /tmp/backoff.log

Stage 5: Read the Results

Count the 429s.

grep -c 'status=429' /tmp/fixed.log
grep -c 'status=429' /tmp/backoff.log

Expected output:

• Fixed: 5 attempts, all 429, success=False
• Backoff: first attempts 429, final attempt 200, success=True

Client	Attempts	429s	200s	Total time	Success
Fixed	5	5	0	~5s	No
Backoff	5	4	1	~31s	Yes

The fixed client amplified the outage. It sent five requests in five seconds. The backoff client sent five requests over 30 seconds. It let the endpoint recover.

Why Backoff Works

Exponential backoff spreads retries over time. Jitter prevents synchronized retries. The formula is simple: min(2^attempt, cap) + random(0,1). This is standard practice. Your LLM client should use it.

Limitations

This drill simulates a single fault window. Real outages vary. Network partitions, DNS failures, and timeouts behave differently. The proxy does not test connection resets. The free tier limits change. Verify current quotas before relying on them. This experiment does not measure model quality. Latency and correctness are separate.

Cleanup

pkill -f fault_proxy.py
pkill -f client.py
rm -f /tmp/fixed.log /tmp/backoff.log /tmp/proxy.out

The Takeaway

A 429 is a signal. It says 'slow down.' Your client should listen. Retry amplification turns a blip into an outage. Run this drill on MonkeyCode's free tier. It takes 30 minutes. It will save you a 3 AM wake-up.