Free Model Endpoints Rate-Limit You. Build a Retry Layer You Can Verify.

Free model endpoints are shared infrastructure, so throttling is the default.
A naive retry loop turns a short rate limit into a long outage.
This tutorial builds a small resilience layer in Python, stage by stage.

The layer has four stages: classification, backoff, breaker, and idempotency.
Every stage has a verification step you can run on free compute.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The workflow uses MonkeyCode's free model access as the upstream.
No quotas, model names, or benchmarks are claimed here.

Why Naive Retries Fail

A simple while loop with a fixed sleep looks fine.
It is not fine when every client retries at the same moment.
That synchronized burst is a retry storm, and it keeps the endpoint throttled.

The fix is a layered policy with one failure mode per layer.
Each layer handles one failure mode and stays testable in isolation.

Stage 1: Classify Errors

Not every error deserves a retry, and classifying first saves requests.
A 400 means your request is wrong, so retrying wastes a request.
A 429 or 503 means the server is busy, so retrying is correct with delay.

# errors.py
RETRYABLE = {429, 500, 502, 503, 504}

def is_retryable(status: int) -> bool:
    return status in RETRYABLE

Verification: assert that 400 returns False and 429 returns True.
Put this in a test file so the rule stays true.

def test_classify():
    assert is_retryable(429)
    assert is_retryable(503)
    assert not is_retryable(400)
    assert not is_retryable(401)

Stage 2: Exponential Backoff with Jitter

Fixed delays synchronize clients, which is exactly what you do not want.
Exponential backoff spreads retries across a wider window.
Jitter removes the last bit of client synchronization.

import random

def backoff_delay(attempt: int, base: float = 0.5, cap: float = 8.0) -> float:
    exp = min(cap, base * (2 ** attempt))
    return random.uniform(0, exp)

Verification: run the function a thousand times and assert the delay never exceeds the cap.
Also assert early attempts are smaller than later ones on average.

Stage 3: Circuit Breaker

Backoff handles slow failures, but it does not handle a dead endpoint.
If the endpoint is down for ten minutes, your client retries for ten minutes.
A circuit breaker stops that waste by opening after repeated failures.

class Breaker:
    def __init__(self, threshold: int = 3):
        self.threshold = threshold
        self.failures = 0
        self.open = False

    def record_failure(self):
        self.failures += 1
        if self.failures >= self.threshold:
            self.open = True

    def record_success(self):
        self.failures = 0
        self.open = False

    def allow(self) -> bool:
        return not self.open

This minimal version tracks failures and exposes an allow check.
A production version adds half-open probing after a cooldown period.
One probe request decides whether the circuit closes or stays open.

Verification: three failures must open the circuit, and one success must close it.
Write that test before you wire the breaker into the client.

Stage 4: Idempotent Retries

Retrying a read is safe, but retrying a write is not.
If the server processed the request but the response was lost, the retry repeats it.
Free model endpoints often expose tool calls or mutations, so add a request ID header.

import uuid

def build_headers():
    return {"X-Request-ID": str(uuid.uuid4())}

The server can dedupe on that ID, and the client reuses it for every attempt.
Verification: send the same request ID twice to a local mock and assert one execution.
If the endpoint ignores request IDs, treat mutation retries as unsafe and log instead.

Putting It Together

Here is the full client loop with all four stages combined.

import time
import requests

def call_with_resilience(url, headers, payload, max_attempts=5):
    breaker = Breaker()
    for attempt in range(max_attempts):
        if not breaker.allow():
            # minimal: wait for cooldown, then retry
            time.sleep(backoff_delay(attempt))
            continue
        try:
            resp = requests.post(url, headers=headers, json=payload, timeout=10)
        except requests.RequestException:
            breaker.record_failure()
            time.sleep(backoff_delay(attempt))
            continue
        if resp.status_code == 200:
            breaker.record_success()
            return resp.json()
        if not is_retryable(resp.status_code):
            resp.raise_for_status()
        breaker.record_failure()
        time.sleep(backoff_delay(attempt))
    raise RuntimeError("max attempts exceeded")

Run this against a local mock that returns 429 twice and then 200.
Assert the client returns the payload and makes exactly three calls.
Then run it against a mock that always returns 503 and assert it raises.

from flask import Flask, jsonify

app = Flask(__name__)
calls = {"count": 0}

@app.post("/generate")
def generate():
    calls["count"] += 1
    if calls["count"] <= 2:
        return jsonify(error="rate limited"), 429
    return jsonify(ok=True, text="done")

Where the Free Server Fits

The retry layer belongs in a gateway, not in every application.
Your app sends one request, and the gateway handles backoff, breaker, and dedupe.
MonkeyCode's free server option can host this gateway for you.

That keeps the resilience logic outside your main service.
Every client that talks to the gateway gets the same retry policy.
The code above does not depend on MonkeyCode, so you can swap the upstream freely.

Limitations

This layer does not fix prompt quality or model latency.
Backoff adds delay by design, so this is not a low-latency path.
It also cannot fix server-side dedupe gaps.
If the endpoint ignores request IDs, mutation retries stay unsafe.

Who should not use this: teams with hard latency SLOs.
If a request must return in 200 milliseconds, a breaker cooldown will break you.
One-off scripts do not need this either, because a fixed-delay retry is enough there.

The Takeaway

Rate limits are not a bug; they are the contract.
Classify errors, back off with jitter, break the circuit, and dedupe requests.
Verify each stage with a small test before you point it at a real endpoint.
Your client will survive a throttled afternoon without making it worse.