Your LLM Quota Will Run Out: A Fallback Ladder for Free-Tier Apps

Your free LLM quota will run out. Not maybe. Not eventually. It will run out on the day your demo has the most eyes on it. That is not bad luck. It is a design flaw.

Free tiers are not a scaling strategy. They are a constraint. And constraints need explicit handling. You already write retry logic for databases and circuit breakers for payment APIs. Your model endpoint deserves the same respect.

I have been building small LLM features on free tiers for a while. The pattern that keeps failing is the same one: a single model call, no fallback, no cache, no plan. When the quota dies, the feature dies with it.

This article is a concrete fix. You will build a fallback ladder with four rungs. Each rung trades quality for availability. The goal is not perfect answers. The goal is that your app never returns a 500 because a model quota ran dry.

The four rungs

A fallback ladder is a chain of attempts. You try the best option first. When it fails, you drop to the next. The ladder I use has four rungs:

1. Cache — exact-match responses from the last hour.
2. Free models — the default path for new requests.
3. Free server — a fallback endpoint when the main model is rate-limited.
4. Template — a deterministic, honest "I am at capacity" response.

Each rung has a different latency and quality profile. The cache is instant but only helps on repeats. The template always works but adds no intelligence. The middle two rungs are where real answers come from.

The implementation

Here is a compact Python class that implements the ladder. It uses the OpenAI client, so it works with any OpenAI-compatible endpoint.

import hashlib
import json
import time
from dataclasses import dataclass
from typing import Optional

from openai import OpenAI


@dataclass
class LadderResult:
    source: str  # "cache" | "free_models" | "free_server" | "template"
    content: str
    latency_ms: float
    cached: bool = False


class FallbackLadder:
    def __init__(
        self,
        free_model_client: OpenAI,
        free_server_client: OpenAI,
        cache_ttl_seconds: int = 3600,
    ):
        self.free_model_client = free_model_client
        self.free_server_client = free_server_client
        self.cache: dict[str, tuple[float, str]] = {}
        self.cache_ttl_seconds = cache_ttl_seconds

    def _cache_key(self, model: str, messages: list[dict]) -> str:
        payload = json.dumps({"model": model, "messages": messages}, sort_keys=True)
        return hashlib.sha256(payload.encode()).hexdigest()

    def _read_cache(self, key: str) -> Optional[str]:
        hit = self.cache.get(key)
        if not hit:
            return None
        expires_at, content = hit
        if time.time() > expires_at:
            del self.cache[key]
            return None
        return content

    def _write_cache(self, key: str, content: str) -> None:
        self.cache[key] = (time.time() + self.cache_ttl_seconds, content)

    def _call(self, client: OpenAI, model: str, messages: list[dict], max_tokens: int) -> str:
        response = client.chat.completions.create(
            model=model,
            messages=messages,
            max_tokens=max_tokens,
            temperature=0,
        )
        return response.choices[0].message.content

    def _template_response(self, messages: list[dict]) -> str:
        last_user = next(
            (m["content"] for m in reversed(messages) if m["role"] == "user"),
            "",
        )
        return (
            "I'm at capacity right now. "
            f"Your question was: {last_user[:80]}... "
            "Please retry in a few minutes."
        )

    def run(
        self,
        messages: list[dict],
        model: str,
        fallback_model: str,
        max_tokens: int = 300,
    ) -> LadderResult:
        key = self._cache_key(model, messages)

        cached = self._read_cache(key)
        if cached is not None:
            return LadderResult("cache", cached, 0.0, cached=True)

        start = time.perf_counter()

        try:
            content = self._call(self.free_model_client, model, messages, max_tokens)
            self._write_cache(key, content)
            return LadderResult(
                "free_models", content, (time.perf_counter() - start) * 1000
            )
        except Exception as exc:
            print(f"[ladder] free models failed: {exc}")

        try:
            content = self._call(
                self.free_server_client, fallback_model, messages, max_tokens
            )
            self._write_cache(key, content)
            return LadderResult(
                "free_server", content, (time.perf_counter() - start) * 1000
            )
        except Exception as exc:
            print(f"[ladder] free server failed: {exc}")

        template = self._template_response(messages)
        return LadderResult(
            "template", template, (time.perf_counter() - start) * 1000
        )

Let me walk through the important parts.

The _cache_key method builds a SHA-256 hash of the model name and messages. That gives you exact-match caching without a database. The cache lives in memory, so it is not for multi-process deployments. It is for a single worker, a demo box, or a script.

The run method is the ladder itself. It tries the cache first. Then the free models. Then the free server. Then the template. Each attempt is wrapped in its own try/except, so one failure does not abort the chain.

The _template_response method is the last rung. It echoes the user's question back with a clear capacity message. It is not smart. It is honest. Users would rather see "try again later" than "internal server error."

Wiring it up

To use the ladder, you need two clients. One for the free model endpoint, one for the free server endpoint.

from openai import OpenAI

free_models = OpenAI(
    base_url="https://<free-models-endpoint>",
    api_key="<your-key>",
)
free_server = OpenAI(
    base_url="https://<free-server-endpoint>",
    api_key="<your-key>",
)

ladder = FallbackLadder(
    free_model_client=free_models,
    free_server_client=free_server,
    cache_ttl_seconds=3600,
)

result = ladder.run(
    messages=[{"role": "user", "content": "Summarize this stack trace: ..."}],
    model="<main-model>",
    fallback_model="<fallback-model>",
)
print(result.source, result.content)

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project that fits this exact pattern. It offers free models and a free server option for developers who want to experiment without a GPU budget. The current free allowance is 10 million tokens at the time of writing. Quotas change, model names change, and endpoints change. Verify the current terms on the project page before you wire anything to it. Do not treat this article as a contract.

The point of the ladder is that it does not care which provider you use. Point the first client at MonkeyCode's free models. Point the second client at its free server. Or point both at any other OpenAI-compatible endpoint. The architecture is provider-agnostic. The free tier just happens to be the cheapest way to test it.

When the ladder makes sense

Use this pattern when:

• You are building a demo, a side project, or an internal tool.
• Your traffic is low and bursty, not steady.
• You can tolerate a template response for a small percentage of requests.
• You want zero-cost operation for as long as possible.

The decision table below summarizes the trade-offs.

Rung	Latency	Cost	Quality	Best for
Cache	~0 ms	0	Exact match	Repeated questions, retries
Free models	300–800 ms	0	High	Default path
Free server	500–1200 ms	0	Medium	Rate-limit recovery
Template	<10 ms	0	Low	Last resort

Read the table as a budget, not a hierarchy. The cache is not "better" than the free models. It is cheaper and faster, but it only answers questions you have already answered. The template is not a failure. It is a designed state.

Limitations

Who should not use this ladder?

Teams with a signed SLA. Production workloads with real users and real money. Anything that processes PII or regulated data. The ladder is a resilience pattern, not a compliance framework. Free tiers have rate limits, no uptime guarantee, and data policies you must read before sending anything sensitive.

The in-memory cache is also a limitation. It does not survive a restart. It does not share across workers. If you need a shared cache, swap the dict for Redis. The interface stays the same.

And the template rung is a product decision, not a technical one. Some products would rather fail loudly than give a canned response. That is a legitimate choice. The ladder makes the trade-off explicit instead of accidental.

The takeaway

Quota exhaustion is not an edge case. It is the natural state of a free tier. The question is whether your app degrades with intention or crashes with surprise.

A fallback ladder turns a hard failure into a soft one. Cache the repeats. Use free models for the default path. Keep a free server as the second chance. End with a template that tells the truth.

That is a small amount of code for a large amount of uptime.

If you are building on a free tier and want to test this pattern today, grab the class, point it at MonkeyCode's free models and free server, and run it against your own prompts. The numbers will tell you where the ladder needs more rungs.