Our desktop assistant sends a request to an LLM after almost every sentence in a live call. Users can connect a dozen providers: Groq, OpenRouter, Google Gemini, NVIDIA, Cloudflare Workers AI, a local Ollama server, CLI tools such as Codex and Claude Code. Most of them have free tiers with rate limits.
One afternoon the app felt "unstable": Google and OpenRouter both returned HTTP 429, and the app kept calling them anyway, once per utterance. Each new request landed inside the provider's rate-limit window and stretched it. The cooldown logic existed. It just wasn't running on the path that mattered.
What a cooldown is here
A cooldown is a timestamp in a dict. Until it passes, the provider is skipped without a network call:
def _cooldown_provider(provider, settings, seconds):
if seconds <= 0:
return
key = _provider_cooldown_key(provider, settings)
_PROVIDER_COOLDOWNS[key] = time.time() + seconds
_PROVIDER_SUCCESSES.pop(key, None)
Any successful answer clears the provider's cooldown.
The key is the part worth thinking about. A cooldown should cover exactly what the limit covers:
-
OpenRouter limits free models individually, so the key is
provider:model. A 429 on one free model doesn't block a different one. - Groq users can add several API keys. Each key has its own cooldown, and the Groq branch walks through the keys, skipping only the ones that are cooling down.
- Local servers (Ollama, LM Studio) and CLI tools get their own keys, because their failures have nothing to do with anyone's quota.
How long, per error
One function maps an HTTP status to a duration in seconds:
def _cooldown_for_http_error(provider, code):
if code in {401, 403}:
return 600
if provider in {GROQ, GROQ_BACKUP} and code == 429:
return COOLDOWN[GROQ] # 60
if provider in (OPENROUTER, OPENROUTER_BACKUP) and code in {400, 402, 429}:
return COOLDOWN[provider] # 300
if provider == GOOGLE and code in {400, 404, 429, 503}:
return COOLDOWN[GOOGLE] # 120
# ... one rule per provider
return 0
The numbers follow how each provider's limits behave:
- 401/403 means a bad or revoked key. Retrying in a minute won't fix it, so it gets 10 minutes.
- Groq's limits are per minute and recover fast, so 60 seconds.
- OpenRouter's free models have daily caps. A 429 there usually means the cap is gone for a while, so 5 minutes. The cooldown then expires and gets one probe, rather than a probe per sentence.
- For some providers, a 400 or 404 also starts a cooldown. Repeating the same request a second later fails the same way.
Errors that aren't HTTP get their own rules:
def _cooldown_for_exception(provider, error):
if isinstance(error, urllib.error.HTTPError):
return _cooldown_for_http_error(provider, error.code)
message = str(error).lower()
if provider == CODEX and "usage limit" in message:
return 600
if "connection refused" in _short_provider_error(error).lower():
return 30 # local server not running
if isinstance(error, TimeoutError) or "timed out" in message:
return min(15, COOLDOWN.get(provider, 60))
return 0
A timeout is the gentlest case: 15 seconds at most, because a single slow answer usually doesn't mean the next one will be slow too.
The bug: two functions, one of them not called
Before the fix, _cooldown_for_exception didn't have that first isinstance branch. It knew only about Codex usage limits and timeouts, and returned 0 for everything else. _cooldown_for_http_error, the one that knows about 429s, existed nearby but was called only from the non-streaming path.
Live answers are streamed, so on the path users actually hit, a 429 from Google or OpenRouter produced a cooldown of zero. Every utterance went straight back to a provider that had just said "slow down".
The fix was to make one function the source of truth and have the other delegate to it. Since then, every caller gets the same answer for the same error.
A second bug sat in the provider loop. It skipped the generic cooldown check for two providers:
if sp not in (AI_PROVIDER_GROQ, AI_PROVIDER_OPENROUTER):
# check cooldown
The Groq exemption was legitimate, since Groq checks cooldowns per key inside its own branch. The OpenRouter branch has no such check, so the exemption just let a cooling-down provider through. OpenRouter came off that list.
A regression test sends two requests in a row against a stubbed provider that returns 429. The second must not reach the provider and must report openrouter skipped: cooldown.
What cooldowns don't fix
A cooldown stops you from making things worse. It doesn't produce an answer. In Auto mode the app races or falls back across several providers, so a provider cooling down is mostly invisible. If a user picks one provider explicitly, a cooldown means no answers for one to five minutes. That was a deliberate product decision: an explicit choice means "use this one". The app reports how long until the next attempt, computed as the smallest remaining cooldown across the providers involved.
Takeaways
- Key cooldowns by whatever the provider rate-limits: per model, per key, per account.
- Choose the duration by cause. An auth failure, a per-minute limit, a daily cap and a timeout need very different backoffs.
- Have one function decide the backoff. Two half-overlapping ones will drift, and the one on the hot path will be the one that's missing a case.
- Test the second request, not the first. The first one fails either way; the second shows whether you learned anything.
I'm building Talkaside, a Windows app for job interviews and calls in another language.
Drafted with AI assistance from this project's code and logs.
Top comments (1)
Keying the cooldown on provider:model for OpenRouter but per API key for Groq is a smart detail, since that matches what each limit actually covers. Have you looked at honoring the Retry-After header when a provider sends one instead of a fixed 60 or 300 seconds? It could shorten the wait for providers that recover quickly. A little random jitter on the cooldown might also help if several users hit the same free tier at the same moment.