Free model tiers fail in predictable ways, but most applications treat them as infinite until the first 429. A small proxy that tracks quotas and fails over between endpoints turned my demo from a 2 AM outage into a self-healing system. The pattern is simple enough to implement in an afternoon and valuable enough to keep.
The Problem: Retries Are Not a Strategy
When a free endpoint rate-limits you, the standard reflex is to retry with exponential backoff. That works when the limit is temporary, but free tiers often have daily or hourly caps that reset on a schedule. Retrying into a hard cap just burns time and makes the failure feel worse. What you need is a way to know how much budget remains before you send a request, and a fallback when that budget is gone.
The Proxy: A Small Class That Thinks in Budgets
I built a tiny proxy that wraps several endpoints, each with its own quota window. The proxy checks the quota before every call, and if the first endpoint is exhausted, it moves to the next. This is not a load balancer; it is a failover router with a memory.
import time
import threading
import requests
class Quota:
def __init__(self, limit, window):
self.limit = limit
self.window = window
self.usage = []
self.lock = threading.Lock()
def allow(self):
with self.lock:
now = time.time()
self.usage = [t for t in self.usage if t > now - self.window]
if len(self.usage) < self.limit:
self.usage.append(now)
return True
return False
class FailoverProxy:
def __init__(self, endpoints):
self.endpoints = endpoints
def call(self, payload):
for name, url, quota in self.endpoints:
if quota.allow():
try:
resp = requests.post(url, json=payload, timeout=30)
if resp.status_code == 200:
return resp.json()
print(f"{name} returned {resp.status_code}")
except Exception as exc:
print(f"{name} failed: {exc}")
else:
print(f"{name} quota exhausted")
raise RuntimeError("All endpoints exhausted")
The quota class is a fixed-window counter. It is not perfectly accurate under concurrency, but it is good enough for a demo or a low-traffic tool. For production, you would want a token bucket backed by Redis, but the principle is identical.
Testing the Failover Without Waiting for a Real Limit
You do not need to exhaust a real quota to test the proxy. I wrote a small simulator that replaces the real endpoints with stubs that return 429 after a fixed number of calls. This lets me verify the failover order and the error path deterministically.
class FakeEndpoint:
def __init__(self, name, fail_after):
self.name = name
self.calls = 0
self.fail_after = fail_after
def __call__(self, payload):
self.calls += 1
if self.calls > self.fail_after:
return {"status": 429}
return {"status": 200, "name": self.name}
# Usage:
# proxy = FailoverProxy([("a", FakeEndpoint("a", 5), quota_a), ...])
The test plan is simple: configure two endpoints, one with a small quota and one with a large quota, then send more requests than the first quota allows. The proxy should route the overflow to the second endpoint. I also tested the case where both are exhausted, expecting a clear exception rather than a silent hang.
Where MonkeyCode Fits
I used this proxy to manage access to MonkeyCode's free model access and its free server option. The free model access gave me a generous token allowance for the primary endpoint, and the free server option acted as a secondary endpoint for failover. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The two endpoints behaved differently under load, which is exactly what you want from a failover pair. When the primary returned a rate-limit response, the proxy switched to the secondary without a single retry. The switch was fast enough that my demo script never noticed the difference.
Limitations and Who Should Not Use This
This proxy is not a substitute for a proper API gateway. It does not handle authentication, request signing, or dynamic discovery of new endpoints. It also assumes that all endpoints speak the same API shape, which is true for OpenAI-compatible APIs but not for every provider.
Do not use this approach if you need a guaranteed service level for a customer-facing product. Free tiers can change their terms at any time, and a failover proxy does not protect you from a provider that disappears entirely. You also need to monitor the proxy's logs, because a silent failover can hide a degraded experience.
The Lesson
The most important shift was treating quotas as a finite budget instead of a wall to hit. Once I knew the budget for each endpoint, I could plan for exhaustion instead of reacting to it. The proxy is twenty lines of code, but it changed how I think about free tiers: they are not a gamble, they are a resource to manage.
Top comments (0)