Routing Around Rate Limits: A Free-Tier Model Proxy in 100 Lines

Free models are not useless; they are just rate-limited. The moment you try to run something real against them, you hit a wall: a 429 that arrives exactly when you need an answer. The standard advice is to pay your way out. This article takes the opposite path: build a small proxy that spreads your workload across several free endpoints, including the one from MonkeyCode, and treat rate limits as a scheduling problem instead of a budget problem.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project that currently offers free model access with a token allowance (10 million tokens at the time of writing) and a free server option. I use it here as one of the upstreams in the proxy. The pattern works with any free tier, so you are not locked in.

The Proxy Pattern

Think of a proxy as a traffic controller. Your app sends one request to the proxy, and the proxy decides which upstream model actually handles it. The upstreams are free tiers you have collected from various providers. Each one has its own quota, latency, and failure modes. The proxy's job is to hide that complexity behind a single endpoint.

The core logic is a simple loop: check which upstreams are healthy and under quota, pick one, call it, and return the result. If it fails, mark it unhealthy and try the next one. That is the whole pattern.

Here is a minimal implementation in Python. It uses httpx for async HTTP calls and an in-memory counter for quota tracking.

import httpx
import asyncio
import time
from dataclasses import dataclass

@dataclass
class Upstream:
    name: str
    url: str
    api_key: str
    quota_per_min: int
    used: int = 0
    window_start: float = 0.0
    healthy: bool = True

    def can_use(self) -> bool:
        now = time.monotonic()
        if now - self.window_start >= 60:
            self.window_start = now
            self.used = 0
        return self.used < self.quota_per_min

    def reserve(self):
        self.used += 1

UPSTREAMS = [
    Upstream("monkeycode", "https://api.monkeycode.example/v1/chat", "key1", 60),
    Upstream("other-free", "https://api.other-free.example/v1/chat", "key2", 30),
]

async def call_upstream(upstream: Upstream, prompt: str) -> str:
    async with httpx.AsyncClient() as client:
        r = await client.post(upstream.url, json={"prompt": prompt}, headers={"Authorization": f"Bearer {upstream.api_key}"}, timeout=30)
        r.raise_for_status()
        return r.json()["text"]

async def route(prompt: str) -> str:
    for upstream in UPSTREAMS:
        if upstream.healthy and upstream.can_use():
            try:
                upstream.reserve()
                return await call_upstream(upstream, prompt)
            except Exception:
                upstream.healthy = False
                continue
    raise RuntimeError("All upstreams exhausted or unhealthy")

The Upstream class tracks a sliding window of 60 seconds. can_use() resets the counter when the window expires, and reserve() increments it before each call. The route() function iterates over the list in order, skipping unhealthy or exhausted upstreams. On exception, it flips the healthy flag to False and moves on.

This is deliberately simple. In production, you would want a circuit breaker, exponential backoff, and a persistent quota store. But for a personal tool or a batch job, this is enough.

Making It Resilient

The first failure mode is a single upstream going down. The proxy handles that by marking it unhealthy, but it never retries it. You need a way to reset the flag after a cooldown period. Add a cooldown_until field and check it in can_use().

@dataclass
class Upstream:
    ...
    cooldown_until: float = 0.0

    def can_use(self) -> bool:
        if time.monotonic() < self.cooldown_until:
            return False
        # ... existing quota check

When an exception occurs, set cooldown_until = time.monotonic() + 60. This gives the upstream a minute to recover before you try again.

The second failure mode is a burst of requests that exhausts all quotas at once. You can handle that with a simple retry loop that waits a few seconds and tries again. But be careful: if you have multiple proxy instances, they will all retry simultaneously and make things worse. A lock or a distributed counter would be better, but that is a topic for another article.

Deploying to a Free Server

The proxy is a single Python file. You can run it on any free server, including the one MonkeyCode provides. The deployment is boring: install httpx, copy the file, run it with uvicorn or gunicorn. Here is a minimal Dockerfile if you prefer containers.

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY proxy.py .
CMD ["uvicorn", "proxy:app", "--host", "0.0.0.0", "--port", "8000"]

You will need to wrap the route() function in a web framework. The simplest is FastAPI:

from fastapi import FastAPI
app = FastAPI()

@app.post("/chat")
async def chat(prompt: str):
    try:
        text = await route(prompt)
        return {"text": text}
    except RuntimeError as e:
        return {"error": str(e)}, 503

Now you have a single endpoint that your application can call, regardless of which upstream is actually serving the request.

When This Makes Sense

This pattern is for people who are already living on free tiers and want to stop being blocked by individual quotas. It is not a production architecture. The free server has no SLA, the upstreams can change their terms, and your proxy is a single point of failure. Do not send regulated data through it. Do not rely on it for a customer-facing product.

But for a prototype, a personal assistant, or a batch evaluation harness, it is a legitimate way to multiply your free capacity. You are trading a little latency for a lot of cost savings. The code is small enough to audit, and the failure modes are visible.

The numbers in this article are current as of August 2026. Free allowances and server options change, so verify the project's documentation before you commit.

If you are already experimenting with free AI infrastructure, this proxy pattern is a practical next step. It turns rate limits from a hard wall into a soft constraint. Start with two upstreams, measure your real usage, and add more as needed. The code is yours, and it will run anywhere.