A Free-Tier LLM Gateway: Routing, Caching, and Failover in About 200 Lines

A five-person team spent three weeks wiring three different free LLM endpoints directly into their internal code-review assistant. Each endpoint had its own error format, its own rate limit, and its own idea of what a JSON response should look like. The result was a tangle of conditionals that broke whenever a provider changed a header or a model name. The fix was not a better client library but a small gateway that sat between the application and every endpoint, and the entire thing fit in about two hundred lines of Python.

This article walks through that gateway design: a routing layer that picks an endpoint by model name, a cache that avoids repeated calls for identical inputs, and a retry loop that fails over to a second provider when the first one is slow or down. The reference implementation runs on a free server, and the same gateway can route to any OpenAI-compatible endpoint, including the free model access provided by the MonkeyCode project.

Why a Gateway Instead of a Client

A direct client works when the application talks to one endpoint. The moment a team adds a second or third provider, every call site needs to know which provider is which, how to format the request, and how to interpret the error. A gateway centralizes that knowledge in one place, and the application sends a single request format to the gateway.

The gateway also solves a problem that free endpoints make worse: instability. Free tiers change their model names, their rate limits, and their response schemas without much notice. A gateway gives the team one place to update those details instead of hunting through the codebase.

The Three Pieces of the Gateway

The gateway has three responsibilities. Routing maps a logical model name like "fast" or "cheap" to a concrete provider URL and API key. Caching stores successful responses keyed by the request payload, so repeated calls for the same input do not consume quota twice. Retry wraps each call with a timeout and a fallback list, so a slow endpoint does not stall the whole application.

# llm_gateway.py
import hashlib
import json
import os
import time
from typing import Any, Callable

import requests

ENDPOINTS = {
    "primary": {
        "base_url": os.environ.get("PRIMARY_BASE_URL"),
        "api_key": os.environ.get("PRIMARY_API_KEY"),
        "model": os.environ.get("PRIMARY_MODEL"),
    },
    "fallback": {
        "base_url": os.environ.get("FALLBACK_BASE_URL"),
        "api_key": os.environ.get("FALLBACK_API_KEY"),
        "model": os.environ.get("FALLBACK_MODEL"),
    },
}

_cache: dict[str, Any] = {}

def _cache_key(messages: list[dict], model: str) -> str:
    payload = json.dumps({"messages": messages, "model": model}, sort_keys=True)
    return hashlib.sha256(payload.encode()).hexdigest()

def _call_endpoint(endpoint: dict, messages: list[dict]) -> dict:
    response = requests.post(
        f"{endpoint['base_url']}/chat/completions",
        headers={"Authorization": f"Bearer {endpoint['api_key']}"},
        json={"model": endpoint["model"], "messages": messages, "temperature": 0},
        timeout=15,
    )
    response.raise_for_status()
    return response.json()

def chat(messages: list[dict], model: str = "primary", use_cache: bool = True) -> str:
    key = _cache_key(messages, model)
    if use_cache and key in _cache:
        return _cache[key]["content"]

    endpoint = ENDPOINTS[model]
    try:
        result = _call_endpoint(endpoint, messages)
    except requests.RequestException:
        fallback = ENDPOINTS["fallback"]
        result = _call_endpoint(fallback, messages)

    content = result["choices"][0]["message"]["content"]
    if use_cache:
        _cache[key] = {"content": content, "ts": time.time()}
    return content

The code above is intentionally minimal. It handles two endpoints, a simple cache, and a single retry. A production gateway would add a proper cache eviction policy, a circuit breaker, and structured logging, but the core pattern is visible in these lines.

Deploying on a Free Server

The gateway is a plain Python service, so it can run anywhere. The team in this story deployed it to a free server option provided by MonkeyCode, which gave them a public URL and a predictable environment for development. The deployment steps were ordinary: copy the code, install the dependencies, set the environment variables, and start the service.

git clone https://example.com/llm-gateway.git
cd llm-gateway
pip install -r requirements.txt
export PRIMARY_BASE_URL="https://api.monkeycode.example/v1"
export PRIMARY_API_KEY="..."
export PRIMARY_MODEL="free-model"
export FALLBACK_BASE_URL="https://api.other-provider.example/v1"
export FALLBACK_API_KEY="..."
export FALLBACK_MODEL="free-model-2"
uvicorn llm_gateway:app --host 0.0.0.0 --port 8000

The free server made the gateway reachable from the team's CI pipeline and local machines, and the free model access from MonkeyCode served as the primary endpoint during the first week of testing.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

At the time of writing, MonkeyCode advertises a free allowance of ten million tokens and a free server option for development workflows. Those numbers should be checked against the current project documentation before being relied on, because free-tier terms change.

Testing the Gateway with a Failure Drill

A gateway is only useful if it survives a provider outage. The team wrote a small drill that simulated a slow primary endpoint and verified that the fallback took over within the timeout window.

# test_failover.py
import time
from unittest.mock import patch

from llm_gateway import chat, _call_endpoint

def test_failover_on_timeout():
    messages = [{"role": "user", "content": "Summarize this issue"}]

    def slow_call(endpoint, messages):
        time.sleep(20)
        raise RuntimeError("timeout")

    with patch("llm_gateway._call_endpoint", side_effect=slow_call):
        start = time.time()
        # The gateway should hit the fallback quickly
        result = chat(messages, model="primary", use_cache=False)
        elapsed = time.time() - start
        assert elapsed < 5
        assert result is not None

The drill passed, but it exposed a subtlety: the fallback endpoint returned a different response format, and the gateway had to normalize it. The team added a small adapter layer that converted each provider's response into a common structure, and that adapter became the most valuable part of the gateway.

Limitations and Who Should Skip This Approach

A gateway adds a network hop, which increases latency by a few milliseconds on every call. Teams that need the absolute lowest latency should keep the client direct. A gateway also introduces a single point of failure, so the server running it needs to be at least as reliable as the endpoints it proxies. Free servers are fine for development and internal tools, but they are not a substitute for a managed service with an SLA.

Teams with a single endpoint and no plan to add another do not need a gateway. Teams with strict data-residency requirements should think carefully before sending prompts to a third-party free endpoint, even through a gateway. And teams that already use a managed LLM gateway from a cloud provider will find little value in a custom implementation.

The Pattern That Matters

The gateway pattern is not about the specific code. It is about separating the application from the volatility of free endpoints. A routing table, a cache, and a retry loop turn a collection of unreliable free APIs into something that behaves like a single, predictable service. The team in this story stopped editing call sites and started editing one configuration file, and that change made the free tier a practical choice for their internal tools.