When a free endpoint returns 429, most agents do the most expensive thing possible: retry. Retrying looks harmless. A 200-millisecond request becomes a 2-second wait, then another attempt. But under peak load, that loop becomes a 30-second stall while your agent clicks refresh on an empty response. If the quota window resets during the stall, every retry burns tokens you could have spent on actual work.
The retry loop assumes the failure is temporary. For rate limits, that assumption is usually wrong. Quota counters reset on a fixed schedule, not on your convenience. You are not just waiting; you are burning wall-clock time that could have gone elsewhere.
The Cascade Pattern
A cascade router is the alternative. It sends requests to the free endpoint, backs off on rate-limit signals, then degrades gracefully to a backup endpoint. The free tier carries the load; the backup exists only when needed. You get the cost advantage of the free tier and the reliability of the paid tier.
The design has three parts: an endpoint abstraction layer, a rate-limit detector, and a circuit breaker that trips when the free endpoint fails repeatedly. Here is the core code:
# cascade_router.py — free tier first, paid/self-hosted as fallback.
import json
import os
import time
import urllib.error
import urllib.request
from dataclasses import dataclass
@dataclass
class Endpoint:
name: str
url: str
api_key: str
model: str
cooldown_until: float = 0.0
consecutive_failures: int = 0
def available(self) -> bool:
return time.time() >= self.cooldown_until
class CascadeRouter:
def __init__(self, endpoints: list[Endpoint]):
self.endpoints = endpoints
def _call_one(self, ep: Endpoint, messages: list[dict]) -> tuple[int, dict]:
body = json.dumps({"model": ep.model, "messages": messages, "max_tokens": 256}).encode()
req = urllib.request.Request(ep.url, data=body, headers={
"Content-Type": "application/json",
"Authorization": f"Bearer {ep.api_key}",
})
start = time.perf_counter()
try:
with urllib.request.urlopen(req, timeout=30) as resp:
data = json.loads(resp.read())
ep.consecutive_failures = 0
return 200, {"latency_ms": round((time.perf_counter() - start) * 1000), "endpoint": ep.name, **data}
except urllib.error.HTTPError as e:
ep.consecutive_failures += 1
if e.code == 429:
backoff = min(60, 5 * (2 ** (ep.consecutive_failures - 1)))
ep.cooldown_until = time.time() + backoff
return e.code, {"error": str(e), "endpoint": ep.name}
def complete(self, messages: list[dict]) -> dict:
for ep in self.endpoints:
if not ep.available():
continue
status, result = self._call_one(ep, messages)
if status == 200:
return result
return {"error": "all endpoints unavailable", "endpoint": "none"}
router = CascadeRouter([
Endpoint("free-tier", os.environ["FREE_URL"], os.environ["FREE_KEY"], os.environ["FREE_MODEL"]),
Endpoint("paid-backup", os.environ["PAID_URL"], os.environ["PAID_KEY"], os.environ["PAID_MODEL"]),
])
result = router.complete([{"role": "user", "content": "Summarize this PR diff in 3 bullets."}])
print(json.dumps(result, indent=2))
The backoff calculation is the key detail. It starts at 5 seconds, doubles with each consecutive failure, and caps at 60. That prevents a retry storm while still letting the free endpoint recover. The circuit breaker is implicit: once the cooldown exceeds 60 seconds, the free endpoint is effectively tripped until the window resets.
The 60-second cap is deliberate. Beyond that, the cost of waiting for the free endpoint exceeds the cost of just using the paid one. The breaker says: I gave you a minute, now I am moving on.
Errors and Quota Windows
Not all errors are equal. A 429 means "you are rate-limited, try later" — cooldown makes sense. A 5xx means "something broke server-side" — one quick retry then moving on is the better strategy, because the server may be restarting and will recover in seconds. Your router should treat them differently.
The shape of your quota window determines your backoff cap. If the quota resets every minute, a 5-second cooldown is plenty. If it resets hourly, the 60-second cap means you will be tripped for the rest of the hour. Know your quota window before configuring the router. A quick curl test will tell you: send requests until you hit a 429, then measure how long recovery takes. That number is your backoff cap.
A naive time.sleep(retry_count * 2) is catastrophic with a 60-second quota window. Your agent retries at 1, 3, and 5 seconds — all inside the window — then gives up, while the window resets 40 seconds later. If it had retried at 45 seconds, it would have succeeded. The naive loop never gets there. The cascade router does.
Quality and Observability
Now the quality trap. Free models and paid models produce different outputs. If the free model is a 7B parameter model and the paid one is 70B, fallback responses will differ in quality. For summarization or classification, that is fine. For precise code generation, it can be a problem. Mitigations: restrict fallback to stateless requests, or accept the context switch.
Logging is the underrated part of the pattern. The router already returns the endpoint field in the response, so you can record it. This is not just cost tracking; it is debugging. When an output looks strange, you need to know which model produced it. A simple structured log line — timestamp, endpoint, model, latency, status — is invaluable in a postmortem.
Where to Run It
Where does this pattern run? On an always-on server. Your laptop sleeps, VPNs jitter, shared Wi-Fi adds latency — all of that pollutes the measurements. MonkeyCode is an open-source agent toolchain whose free tier includes a 10-million-token model allowance and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free server is a natural place to run this router because it stays online. The free allowance acts as tier one; your existing paid endpoint or self-hosted model acts as tier two.
Who Should Skip This
Three groups should not use this. First, if the quality gap between your free and paid endpoints breaks downstream tasks — say, the free model produces malformed JSON regularly and your parser is strict — cascading just delays the failure. Second, if a data boundary forbids sending prompts to an external free endpoint, cascading is architecturally impossible. Third, if the quota window is very short (one request per minute, for example), the cooldown will never be short enough and the router will just fall back constantly — in that case, use the paid endpoint directly and skip the free tier.
The value of this pattern is that it treats the free tier as a real component, not a gift. Free tiers have constraints; the cascade router manages those constraints. When the free endpoint is available, you save money. When it is not, your agent still works. That is how a free tier should be used: not as the only endpoint, but as the first rung of a ladder. If you want to see it in a real environment, MonkeyCode's free tier includes the 10-million-token allowance and a free server, with quota details verifiable in the repository. Run the router, watch the fallbacks happen, and you will know whether the free tier earns its place in your stack.
Top comments (0)