Your Free API Key Will 429: A Resilience Playbook for LLM Endpoints

A free LLM endpoint will eventually return 429 Too Many Requests. The question is not whether it happens — it is whether your client survives it. Most agent code treats the model API as a reliable dependency: one call, one response, no surprises. On a free tier, that assumption breaks in three predictable ways, and each one needs a different countermeasure.

This post is a reliability playbook for anyone building on a free model tier. The artifact is a small Python client that combines exponential backoff, a circuit breaker, and quota-aware degradation — the three patterns that turn an unreliable free endpoint into a tolerable one.

The three failure modes of a free tier

Free endpoints fail differently from paid ones. Paid APIs fail rarely and briefly. Free tiers fail constantly and predictably. You need to distinguish between them because the fix is different.

1. Rate limiting (429) — the endpoint is up, but you exceeded the per-minute or per-second allowance. The fix is backoff and retry.
2. Quota exhaustion (429 with a different body, or 403) — the monthly token allowance is spent. The fix is degradation, not retry; retrying a spent quota is wasted work.
3. Infrastructure jitter (5xx, timeouts) — the free server is overloaded or restarting. The fix is a circuit breaker so you stop hammering a dying service.

A single retry loop handles none of these well. Retrying a rate limit works. Retrying a spent quota makes it worse. Retrying a 5xx storm amplifies the load that caused the storm.

The decision table

Failure	Status code	Retry?	Backoff?	Circuit breaker?	Degrade?
Rate limit	429	Yes	Exponential	No	No
Quota exhausted	429/403	No	No	No	Yes
Server error	5xx	Yes (limited)	Exponential	Yes	Yes
Timeout	—	Yes (limited)	Exponential	Yes	Yes

The pattern that matters most is the circuit breaker. It converts a failing dependency from a source of repeated exceptions into a fast-fail path that triggers degradation.

Implementation: a resilient client

Here is a minimal implementation of the three patterns. It wraps any OpenAI-compatible endpoint, which keeps the provider swappable.

# resilient_client.py — backoff + circuit breaker + fallback for LLM endpoints
import random
import threading
import time
from datetime import datetime, timedelta
from openai import OpenAI


class CircuitBreaker:
    """Opens after N consecutive failures, then allows a probe after cooldown."""
    def __init__(self, failure_threshold=5, cooldown_seconds=60):
        self.failure_threshold = failure_threshold
        self.cooldown_seconds = cooldown_seconds
        self.failures = 0
        self.open_until = None
        self.lock = threading.Lock()

    def allow(self):
        with self.lock:
            if self.open_until is None:
                return True
            if datetime.now() >= self.open_until:
                self.open_until = None
                self.failures = 0
                return True
            return False

    def record_failure(self):
        with self.lock:
            self.failures += 1
            if self.failures >= self.failure_threshold:
                self.open_until = datetime.now() + timedelta(seconds=self.cooldown_seconds)

    def record_success(self):
        with self.lock:
            self.failures = 0
            self.open_until = None


class QuotaBudget:
    """Tracks token usage and flips to degraded mode before exhaustion."""
    def __init__(self, limit, warn_at=0.8):
        self.limit = limit
        self.warn_at = warn_at
        self.used = 0
        self.lock = threading.Lock()

    def record(self, prompt_tokens, completion_tokens):
        with self.lock:
            self.used += prompt_tokens + completion_tokens

    @property
    def degraded(self):
        with self.lock:
            return self.used >= self.limit * self.warn_at


class ResilientLLMClient:
    def __init__(self, base_url, api_key, model,
                 quota_limit=10_000_000,
                 max_retries=5, base_delay=1.0, max_delay=30.0,
                 fallback=None):
        self.client = OpenAI(base_url=base_url, api_key=api_key)
        self.model = model
        self.breaker = CircuitBreaker()
        self.quota = QuotaBudget(quota_limit)
        self.max_retries = max_retries
        self.base_delay = base_delay
        self.max_delay = max_delay
        self.fallback = fallback  # (client, model) tuple or None

    def complete(self, messages, tools=None, temperature=0):
        # Quota degradation: stop calling the free tier before it is spent.
        if self.quota.degraded:
            return self._degrade(messages, tools, temperature)

        # Circuit breaker: fast-fail when the endpoint is unhealthy.
        if not self.breaker.allow():
            return self._degrade(messages, tools, temperature)

        for attempt in range(self.max_retries):
            try:
                resp = self.client.chat.completions.create(
                    model=self.model,
                    messages=messages,
                    tools=tools,
                    temperature=temperature,
                )
                self.breaker.record_success()
                self.quota.record(
                    resp.usage.prompt_tokens,
                    resp.usage.completion_tokens,
                )
                return resp
            except Exception as e:
                self.breaker.record_failure()
                status = getattr(e, "status_code", None)
                # Quota exhaustion is not retryable.
                if status in (403,) or (status == 429 and "quota" in str(e).lower()):
                    return self._degrade(messages, tools, temperature)
                if attempt == self.max_retries - 1:
                    return self._degrade(messages, tools, temperature)
                delay = min(self.base_delay * (2 ** attempt) + random.uniform(0, 1), self.max_delay)
                time.sleep(delay)

    def _degrade(self, messages, tools, temperature):
        if self.fallback is not None:
            fallback_client, fallback_model = self.fallback
            return fallback_client.chat.completions.create(
                model=fallback_model,
                messages=messages,
                tools=tools,
                temperature=temperature,
            )
        raise RuntimeError("free tier unavailable and no fallback configured")

Three details in this code are deliberate:

• The quota check happens before the API call, not after. By the time you get a quota-exhausted response, you have already spent the request. Degrading at 80% usage leaves headroom for in-flight requests.
• The circuit breaker opens after 5 consecutive failures. This prevents the retry loop from amplifying a 5xx storm. Once open, it rejects all traffic for 60 seconds, then lets one probe through.
• Quota exhaustion is detected by inspecting the error body, not just the status code. Some providers return 429 for both rate limits and quota exhaustion; retrying the former is correct, retrying the latter is wasted time.

Testing: inject failures, don't hope for them

The only way to verify a resilience design is to force the failures. A simple fault-injection wrapper can simulate each mode without touching the real endpoint.

# fault_injector.py — wrap any OpenAI client and inject failures
import random
from openai import OpenAI


class FaultInjector:
    def __init__(self, client, fail_rate=0.3, quota_exhausted=False):
        self.client = client
        self.fail_rate = fail_rate
        self.quota_exhausted = quota_exhausted
        self.calls = 0

    def chat(self):
        return _FaultyCompletions(self)


class _FaultyCompletions:
    def __init__(self, injector):
        self.injector = injector
        self.model = "faulty"

    def create(self, **kwargs):
        self.injector.calls += 1
        if self.injector.quota_exhausted:
            raise Exception("429 quota exceeded")
        if random.random() < self.injector.fail_rate:
            raise Exception("500 Internal Server Error")
        return _FakeResponse()


class _FakeResponse:
    class Usage:
        prompt_tokens = 100
        completion_tokens = 50
    usage = Usage()

    class Choice:
        class Message:
            content = "ok"
            tool_calls = None
        message = Message()
    choices = [Choice()]

Then assert the client's behavior under each failure mode:

# test_resilience.py
from resilient_client import ResilientLLMClient
from fault_injector import FaultInjector

injector = FaultInjector(client=None, fail_rate=0.5)
client = ResilientLLMClient(
    base_url="http://localhost:9999", api_key="test", model="test",
    max_retries=3, base_delay=0.1, max_delay=0.5,
)
client.client = injector  # swap in the faulty client

# Expect: retries absorb the 50% failure rate, no exception raised.
resp = client.complete([{"role": "user", "content": "hi"}])
assert resp.choices[0].message.content == "ok"

# Expect: quota exhaustion degrades immediately, no retry loop.
injector.quota_exhausted = True
client.quota.used = client.quota.limit  # force degraded state
resp = client.complete([{"role": "user", "content": "hi"}])
# Falls through to _degrade; with no fallback configured, raises RuntimeError.

This test suite is the real deliverable. It encodes the decision table into executable assertions, so a future change to the retry policy cannot silently break the degradation path.

Where the free tier fits

To exercise this playbook against a real free endpoint, I used MonkeyCode's free tier, which at the time of writing offers a 10M token allowance and a free server option. The resilient client above works against any OpenAI-compatible base URL, so the provider choice is an environment variable, not an architectural commitment.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Exact quotas, rate limits, and model availability change over time — verify the current terms in the official documentation before relying on them in a design. The code in this post does not depend on any specific provider behavior beyond the OpenAI-compatible interface.

Limitations

• This is not a benchmark. The fault-injection tests verify behavior under simulated failures; they say nothing about real-world latency or throughput on any specific free tier.
• A circuit breaker is not a fix. It is a containment strategy. If the free endpoint is chronically unhealthy, the correct move is to change providers, not to retune the breaker.
• Quota tracking is approximate. The QuotaBudget class counts tokens reported by the API response, which may not match the provider's internal accounting. Treat 80% as a safety margin, not an exact threshold.
• Do not use this pattern for real-time user-facing features, workloads with contractual SLAs, or high-concurrency production agents. Free tiers are for batch jobs, personal automation, and evaluation loops.

The takeaway

Free LLM access is a budget constraint with a failure profile. Treat it that way and it becomes a useful engineering input: it forces you to design degradation paths, quota awareness, and fast-fail behavior that your paid-API code probably lacks. Build the resilience layer once, and the free tier stops being a risk and becomes just another endpoint behind the same client.

If you want to try the pattern against a real free endpoint, MonkeyCode's free tier is a reasonable candidate — the client code needs only a base URL and an API key. But the playbook itself is provider-agnostic: the next time any dependency starts returning 429s, you will already have the right response.