Build for the 429: An Architecture for Unreliable Free LLM Tiers

The demo is halfway through. I click "Run." The screen shows 429 Too Many Requests. I refresh. Another 429. The audience is waiting. I switch to a local model. It's too slow. The demo fails.

This isn't the free tier's fault. It's mine. I built the app assuming the free tier was reliable. It never was. Free tiers are shared, rate-limited, and prone to downtime. They're designed for prototyping, not production.

But you can build an architecture that fails gracefully. This article shows how. It's not a measurement tool. It's a design pattern. You'll learn how to cache, retry, fall back, and degrade so your demo never dies from a 429.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server for development. The free tier includes a 10M-token allowance. Generous. But it's also shared. Your requests compete with everyone else's. Sometimes you get rate-limited. Sometimes the server is overloaded. That's the price of free.

Don't try to make the free tier reliable. You can't. Instead, build a system that works when the free tier doesn't. Here are five patterns.

Pattern 1: Cache

Caching is the simplest defense. If two requests are identical, send one. Cache the response. Subsequent requests read from the cache.

from functools import lru_cache

@lru_cache(maxsize=128)
def get_completion(prompt: str) -> str:
    # call the free tier here
    return response

lru_cache is built-in. It works for exact prompt matches. For more complex caching, use cachetools with a TTL.

from cachetools import TTLCache, cached

cache = TTLCache(maxsize=128, ttl=3600)

@cached(cache)
def get_completion(prompt: str) -> str:
    # call the free tier here
    return response

Caching reduces token consumption. It also reduces the chance of hitting rate limits. A double win.

Pattern 2: Retry with Backoff

When a request fails, don't retry immediately. Wait. Exponential backoff. First wait 1 second, then 2, then 4. Cap at 30 seconds.

import time

def call_with_retry(prompt: str, max_retries: int = 5) -> str:
    for attempt in range(max_retries):
        try:
            return call(prompt)
        except RateLimitError:
            wait = min(2 ** attempt, 30)
            time.sleep(wait)
    raise RuntimeError("Free tier is down")

The tenacity library makes this cleaner.

from tenacity import retry, stop_after_attempt, wait_exponential

@retry(stop=stop_after_attempt(5), wait=wait_exponential(multiplier=1, max=30))
def call_with_retry(prompt: str) -> str:
    return call(prompt)

Retries handle transient failures. They don't handle sustained outages. Set a max retry count. Then fail.

Pattern 3: Fallback

When the free tier fails, fall back to a backup. The backup can be a local model, a simpler heuristic, or a hardcoded response.

def call_with_fallback(prompt: str) -> str:
    try:
        return call_free_tier(prompt)
    except Exception:
        return call_local_model(prompt)

Fallback chains can be longer. Free tier -> local model -> cache -> default response. Each level adds resilience.

Pattern 4: Degrade

Degradation means returning a less ideal but still usable response when the free tier is unavailable. For example, if summarization fails, return the first 100 characters of the input.

def summarize(text: str) -> str:
    try:
        return call_free_tier(f"Summarize: {text}")
    except Exception:
        return text[:100] + "..."

Degradation keeps the user experience intact. It may not be perfect, but it's usable.

Pattern 5: Queue

For async workloads, use a queue to absorb rate limits. Requests go into a queue. A background worker processes them at a controlled rate.

import queue
import threading

request_queue = queue.Queue()

def worker():
    while True:
        prompt, result_queue = request_queue.get()
        try:
            result_queue.put(call_free_tier(prompt))
        except Exception:
            result_queue.put(None)
        finally:
            request_queue.task_done()

threading.Thread(target=worker, daemon=True).start()

def submit(prompt: str) -> queue.Queue:
    result_queue = queue.Queue()
    request_queue.put((prompt, result_queue))
    return result_queue

A queue turns bursts into a steady stream. This reduces the chance of hitting rate limits.

Putting It Together

Here's a ResilientLLMClient class that wraps all the patterns.

from cachetools import TTLCache
from tenacity import retry, stop_after_attempt, wait_exponential

class ResilientLLMClient:
    def __init__(self, primary, fallback=None, cache_size=128, ttl=3600):
        self.primary = primary
        self.fallback = fallback
        self.cache = TTLCache(maxsize=cache_size, ttl=ttl)

    @retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, max=10))
    def _call_primary(self, prompt: str) -> str:
        return self.primary(prompt)

    def call(self, prompt: str) -> str:
        if prompt in self.cache:
            return self.cache[prompt]
        try:
            result = self._call_primary(prompt)
            self.cache[prompt] = result
            return result
        except Exception:
            if self.fallback:
                result = self.fallback(prompt)
                self.cache[prompt] = result
                return result
            raise

This class caches responses, retries transient failures, and falls back when needed. It's free.

What About the Free Server?

MonkeyCode also provides a free server. The same principles apply. Don't assume the server is reliable. Build an app that handles restarts, cold starts, and shared resources.

• Use stateless design. Store state in an external database or object storage.
• Use health checks. If the server is unhealthy, route to a backup.
• Use environment variables for configuration. Don't hardcode keys.
• Use persistent volumes if available. But don't rely on them.

A free server is free. It's also shared. Treat it like a shared server.

When to Use This Architecture

This architecture is for prototyping, demos, and internal tools. It's great when you can tolerate occasional failures. It's not for when failures cost money or safety.

Don't use free tiers in production. Don't build customer-facing products on a free tier. A free tier is a playground. Treat it like one.

Conclusion

Free tiers aren't unreliable. They're different. They have different constraints. By building for failure, you can take full advantage of them. Cache, retry, fall back, degrade, queue. These patterns will save your demo from a 429.

Next time you build on a free tier, start with the 429 in mind. Your demo will thank you.