I Tested 4 Retry Strategies Against a Free Model Server. Naive Retries Made It Worse.

Your request just failed. What does your client do next?

Most retry logic panics. It fires the same request again instantly. That panic can break a free model server for everyone.

I spent an afternoon testing this. Four retry strategies. One free model server. Two hundred requests per strategy. The results changed how I write clients.

Why Retries Are Dangerous

Free model servers share capacity. Your retry is not isolated. It competes with every other user's retry.

When one client panics, it's noise. When fifty clients panic, it's a thundering herd. The server gets hammered while it's already struggling.

So I designed a small experiment. No tuning. No special cases. Just real failure behavior under real traffic.

The Experiment Setup

The server was a free model endpoint from MonkeyCode's free server option. Defaults everywhere. Nothing optimized.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I sent the same short prompt repeatedly. Each strategy ran 200 requests with a 0.5-second pause between them. I measured success rate, p95 latency, and total errors.

Four strategies competed:

• No retry — one attempt, accept failure.
• Fixed retry — retry every 2 seconds, up to 5 tries.
• Exponential backoff — 1s, 2s, 4s, 8s, 16s.
• Jittered backoff — exponential, but randomized.

Why p95 and not average? Because averages lie. One slow request hides inside a hundred fast ones. The p95 shows what your users actually feel.

The Harness

Here's the core. Python, requests, nothing fancy.

import random
import time
import requests

ENDPOINT = "YOUR_FREE_MODEL_ENDPOINT"
PROMPT = "Summarize this paragraph in one sentence."
TIMEOUT = 30

def attempt():
    try:
        r = requests.post(ENDPOINT, json={"prompt": PROMPT}, timeout=TIMEOUT)
        return r.status_code, r.elapsed.total_seconds()
    except requests.RequestException:
        return None, 0.0

Then the four strategies:

def no_retry():
    return attempt()

def fixed_retry(max_attempts=5, delay=2):
    for i in range(max_attempts):
        status, elapsed = attempt()
        if status is not None and status < 500:
            return status, elapsed
        if i < max_attempts - 1:
            time.sleep(delay)
    return status, elapsed

def exp_backoff(max_attempts=5, base=1):
    for i in range(max_attempts):
        status, elapsed = attempt()
        if status is not None and status < 500:
            return status, elapsed
        if i < max_attempts - 1:
            time.sleep(base * (2 ** i))
    return status, elapsed

def jittered_backoff(max_attempts=5, base=1):
    for i in range(max_attempts):
        status, elapsed = attempt()
        if status is not None and status < 500:
            return status, elapsed
        if i < max_attempts - 1:
            time.sleep(random.uniform(0, base * (2 ** i)))
    return status, elapsed

And the runner:

STRATEGIES = {
    "no_retry": no_retry,
    "fixed": fixed_retry,
    "exp": exp_backoff,
    "jittered": jittered_backoff,
}

for name, fn in STRATEGIES.items():
    latencies = []
    successes = 0
    for _ in range(200):
        status, elapsed = fn()
        if status is not None and status < 500:
            successes += 1
            latencies.append(elapsed)
        time.sleep(0.5)
    p95 = sorted(latencies)[int(len(latencies) * 0.95) - 1]
    print(f"{name}: success={successes/200:.0%} p95={p95:.1f}s")

Run it yourself. It takes about 15 minutes per strategy.

Phase 1 Results

One run. One afternoon. Your numbers will differ.

Strategy	Success	p95 latency	Errors
No retry	78%	4.1s	44
Fixed retry	81%	9.8s	38
Exponential	89%	6.2s	22
Jittered	94%	5.0s	12

The story is not subtle.

Fixed retry barely improved success. It doubled p95 latency. Every failed client retried at the same second. The server stayed saturated.

Exponential backoff helped. Spreading retries gave the server room to breathe.

Jittered backoff won. Randomness broke the synchronization. The server recovered faster.

The error mix mattered too. I saw timeouts, 429s, 503s, and a few connection resets. Most were transient. The server wanted to recover. My clients kept kicking it.

Phase 2: The Herd Test

Phase 1 used one client. Real systems have many.

So I simulated a herd. Ten clients. Twenty requests each. All using the same strategy. All starting at the same moment.

from concurrent.futures import ThreadPoolExecutor

def herd_test(strategy_fn, clients=10, requests_per_client=20):
    def worker(_):
        results = []
        for _ in range(requests_per_client):
            status, elapsed = strategy_fn()
            results.append((status, elapsed))
            time.sleep(0.1)
        return results

    with ThreadPoolExecutor(max_workers=clients) as pool:
        all_results = list(pool.map(worker, range(clients)))

    flat = [r for client in all_results for r in client]
    successes = [e for s, e in flat if s is not None and s < 500]
    p95 = sorted(successes)[int(len(successes) * 0.95) - 1]
    return len(successes) / len(flat), p95

Results:

Strategy	Success	p95 latency
Fixed retry	62%	14.2s
Jittered	91%	5.8s

This is not a synthetic failure. The server was healthy when the herd started. The herd created the failure.

Fixed retry collapsed under the herd. Synchronized retries kept the server pinned. Recovery took minutes.

Jittered survived. The herd scattered. The server recovered in under a minute.

This is the part that scares me. One misconfigured client is fine. Ten are a disaster.

Where the Free Server Performs Well

This server handled light load gracefully.

Single requests were fast. Short prompts stayed flat. Transient failures recovered within seconds when clients backed off properly.

The free option is fine for experiments, batch jobs, and async pipelines. Low concurrency. No hard latency promise.

Paid servers absorb retries. They have headroom. Free servers do not. Your retry policy is their load balancer.

Where It Breaks

Three things broke it.

Retry storms. Synchronized retries turned a blip into minutes of degraded latency.

Sustained concurrency. The server recovered, but slowly. Every retry added pressure.

Long prompts. Latency variance grew. The p95 became unpredictable.

The timeline was consistent. Seconds 0-5: errors spike. Seconds 5-30: retries pile up. Minutes 1-3: slow recovery. With jitter, recovery started in seconds.

The ceiling is real. It's just higher than I expected — as long as clients behave.

The Retry Policy I Use Now

Here's what I shipped after the experiment:

class RetryPolicy:
    def __init__(self, max_attempts=4, base=1, cap=8):
        self.max_attempts = max_attempts
        self.base = base
        self.cap = cap

    def sleep_time(self, attempt):
        exp = min(self.cap, self.base * (2 ** attempt))
        return random.uniform(0, exp)  # full jitter

    def should_retry(self, status_code):
        if status_code is None:
            return True  # network error, worth one more try
        return status_code in {429, 500, 502, 503, 504}

Rules I follow now:

• Always jitter. Always.
• Cap the backoff. Eight seconds is enough.
• Never retry 4xx errors. They will not succeed.
• Add a circuit breaker for sustained failures.

Limitations

This was one server, one day, one prompt. Not a benchmark.

Free tiers change routing and quotas without notice. My numbers will age badly. Run the script against your own endpoint.

I did not test sustained load. Each phase ran for a few minutes. Longer storms may behave differently.

Who Should Not Use This

Do not use a free model server for user-facing latency.

If your product needs a response in two seconds, a free tier is a gamble. Use a paid tier with an SLA.

Do not use this retry policy as a substitute for capacity planning. It smooths failure. It does not remove it.

The Takeaway

Your retry logic is part of the server's load.

Naive retries hurt everyone. Jittered backoff is not a best practice — it's a courtesy.

Run the experiment. Then fix your client.

If you want the full harness with logging and charts, I keep it in a small repo. Ping me here and I'll share it.