Free Models and a Free Server: A 100-Line LLM Failover Proxy

Free LLM access is unreliable only when you have no fallback. A tiny proxy that routes between a managed free API and a self-hosted model on a free server turns intermittent rate limits into a non-event. This article builds that proxy from scratch, with code you can copy and run today.

MonkeyCode is an open-source project that gives you two free resources: free models via a managed API and a free server you can use to host your own open-source models. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Combining these two lets you experiment with both sides of the LLM deployment spectrum without spending a dollar. But raw access is not enough you need a way to switch between them automatically when one side hiccups.

Here is a concrete architecture: a Python service exposes an OpenAI-compatible API to your application. Internally, it has two backends. The first calls MonkeyCode's free model API. The second talks to a local model running on the free server via an OpenAI-compatible REST endpoint. The proxy tries the managed API first, watches for failures, and fails over to the local model when needed. This gives you the latency of a managed API during normal hours and the resilience of a self-hosted fallback during spikes.

Step 1: Provision the free server

Log into the MonkeyCode console and create a free server instance. The exact image is not important; choose any Linux image with Python 3.10 or newer. SSH into the box and install Docker if it is not present. This server will host your local model.

ssh root@your-free-server-ip
apt update && apt install -y docker.io
systemctl start docker

Keep the server address handy. You will point the proxy to it later.

Step 2: Deploy a local model on the free server

The free server needs a model that speaks the OpenAI protocol. Ollama is the simplest way to get one running in minutes. Install Ollama and pull a small instruct model that fits the server's memory.

curl -fsSL https://ollama.com/install.sh | sh
ollama pull qwen2.5:1.5b
ollama serve &

Your model now listens on http://localhost:11434/v1. Note the endpoint; it accepts standard OpenAI chat completions. You can verify it with a quick request.

curl http://localhost:11434/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen2.5:1.5b","messages":[{"role":"user","content":"ping"}]}'

If you get a JSON response, the model is ready.

Step 3: Write the failover proxy

The proxy itself is a single Python file. It wraps both backends behind one interface and decides which one to call. The logic is simple: try the managed free model first; if the call raises a rate-limit or a server error, retry once against the local model.

Save this as proxy.py.

import os
import time
import httpx
from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

FREE_API_KEY = os.getenv("FREE_API_KEY")
FREE_API_URL = os.getenv("FREE_API_URL")
LOCAL_API_URL = os.getenv("LOCAL_API_URL", "http://127.0.0.1:11434/v1")
LOCAL_MODEL = os.getenv("LOCAL_MODEL", "qwen2.5:1.5b")

class ChatRequest(BaseModel):
    model: str = "gpt-3.5-turbo"
    messages: list
    temperature: float = 0.7

async def call_free_api(req: ChatRequest):
    async with httpx.AsyncClient() as client:
        payload = req.model_dump(exclude={"model"})
        payload["model"] = "monkeycode-free-model"  # placeholder
        headers = {"Authorization": f"Bearer {FREE_API_KEY}"}
        r = await client.post(FREE_API_URL, json=payload, headers=headers)
        r.raise_for_status()
        return r.json()

async def call_local_model(req: ChatRequest):
    async with httpx.AsyncClient() as client:
        payload = req.model_dump(exclude={"model"})
        payload["model"] = LOCAL_MODEL
        r = await client.post(f"{LOCAL_API_URL}/chat/completions", json=payload)
        r.raise_for_status()
        return r.json()

@app.post("/v1/chat/completions")
async def chat(req: ChatRequest):
    try:
        return await call_free_api(req)
    except (httpx.HTTPStatusError, httpx.ConnectError) as e:
        if e.response and e.response.status_code == 429:
            time.sleep(0.5)
        # fallback to local model
        return await call_local_model(req)

Set the environment variables when you launch the proxy. The placeholder model name must match what MonkeyCode's free API expects; replace it with the actual model identifier from your MonkeyCode dashboard.

# Terminal 1: start the local model (already running)
# Terminal 2: start the proxy
FREE_API_KEY=your_key FREE_API_URL=https://api.monkeycode.example/v1 \
LOCAL_API_URL=http://your-free-server-ip:11434/v1 LOCAL_MODEL=qwen2.5:1.5b \
uvicorn proxy:app --host 0.0.0.0 --port 8000

Step 4: Test the failover behavior

Now send a request to your proxy and observe which backend answers.

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"test","messages":[{"role":"user","content":"Hello"}]}'

You can simulate a free-API outage by stopping the network or using an invalid API key. The proxy should automatically return a response from the local model. To see this in a log, add a print statement before each call. This manual probe takes five minutes and gives you a concrete feel for the trade-offs.

Step 5: Compare the two backends

Once the proxy works, write a small benchmark script. Send 100 requests and measure latency and error rate for each backend separately. You will likely discover that the managed free model is faster but occasionally returns 429, while the local model is slower but consistent.

import time
import httpx

urls = ["http://localhost:8000/v1/chat/completions", ...]
for target in urls:
    times = []
    errors = 0
    for _ in range(100):
        t0 = time.monotonic()
        try:
            httpx.post(target, json={"model":"x","messages":[{"role":"user","content":"hi"}]})
            times.append(time.monotonic() - t0)
        except Exception:
            errors += 1
    print(target, "avg", sum(times)/len(times), "errors", errors)

These numbers tell you which backend should be primary for your workload. The proxy code can be extended to use that info later, but for now the manual failover is enough.

Limitations

The free server has limited CPU and memory. A 1.5B model runs fine, but a 7B model will be painfully slow. The managed free model also has rate limits; they exist to protect the service, not to support production scale. This proxy handles occasional spikes, not sustained high traffic. Do not use it for patient records or classified data unless you verify the data flow and add encryption. The local model on a free server may not pass your compliance bar.

The code above is a starting point, not a finished product. It has no retry logic with backoff, no circuit breaker, and no observability. For a serious deployment, add structured logs, a health-check endpoint, and a fallback chain with more than two backends.

Who should use this

This design fits developers who want to learn the failure modes of free LLM tiers without risking real money. It also fits side projects that can tolerate occasional seconds of latency from the local model. If you are building a commercial product with a strict SLO, pay for a dedicated API or a GPU server. The free route is for experimentation, education, and prototypes.

The takeaway

Free models and a free server are not just two separate handouts. They are two halves of a resilient LLM setup when you pair them with a small failover proxy. The code in this article gives you a working pattern in about 100 lines. Use it to test your own workloads, measure the real differences, and then decide where to spend money once you know the facts.

Try the same two-backend pattern with MonkeyCode's free models and free server. The first run will take you an evening, and the lessons will stay with you for much longer.