SentioGate for SentioGate

Posted on Oct 20 AI-assisted

Building a Rate-Limit-Aware LLM Router in Python: Retries, Backoff & Capacity Scoring

#python #llm #ai #architecture

If your app has ever fallen over because a provider returned 429 Too Many Requests, and your fallback was "sleep a second and try again", you know the pain. That pattern hurts twice: you hit the provider again while it's already telling you to slow down, and you still don't know how long to wait.

I built SentioGate partly because I kept writing that loop. Here's how it handles rate limits, and how to use it from Python or JavaScript.

The problem: a 429 is information

A 429 isn't a random failure. It's the provider telling you something specific: this model, on this key, is unavailable for a while. Often it even tells you how long, in a Retry-After header.

Most retry code throws that information away. It re-sends the same request to the same model and hopes. A rate-limit-aware router does the opposite:

Remember that this model on this key is throttled, and for how long.
Route around it to the next healthy candidate.
Come back once the window has passed, carefully.

What SentioGate does about it

SentioGate is an OpenAI-compatible router. You send a chat request with model: "auto" and your own provider keys (BYOK), or use the free community pool, and it picks a model for you. Rate limits are part of that decision, not an afterthought.

Automatic cooldowns

When a provider throttles a request, the hosted service puts that model on that key on a cooldown. It follows the provider's own Retry-After when there is one, so the cooldown lasts as long as the provider asked, within sensible bounds. During the cooldown, that model simply isn't a candidate, and your request moves on without waiting.

Other failures are handled on their own terms. A key that's out of credits, or a model your key can't access, is set aside instead of being retried over and over.

Circuit breakers

Separately from cooldowns, a model that keeps failing gets its breaker tripped, and it's skipped for a while. Breaker state is shared across the whole edge, so once one request finds a sick model, everyone else's requests avoid it too.

Fallback to the next healthy model

With model: "auto", SentioGate ranks candidates on capability, price, and measured latency and success rate. Models that are cooling down or tripped drop out of the ranking. If the chosen model fails, the request falls through to the next healthy one. For streams, that happens before the first token reaches you.

If every candidate is unavailable, you don't get a hang. You get a clear 429 with a Retry-After telling you when to try again.

The community pool: careful recovery

The free community pool is shared by everyone, so it's handled with extra care. When the upstream throttles it, the pool backs off for as long as the upstream asked. When that window ends, it checks the upstream cautiously before letting traffic flow again. That way, a crowd of waiting clients doesn't stampede the provider the instant the window opens.

Queue instead of 429

Some workloads would rather wait than fail. When the shared community pool is full or rate limited, a non-streaming request can opt in to queueing with one header:

curl https://api.sentiogate.dev/v1/chat/completions \
  -H "Authorization: Bearer $SENTIOGATE_API_KEY" \
  -H "Content-Type: application/json" \
  -H "X-SentioGate-Allow-Queue: true" \
  -d '{"model": "auto", "messages": [{"role": "user", "content": "Summarise this ticket..."}]}'

Instead of a 429, you get a 202 with a job_id and a poll_url to collect the result later.

How to use it from Python

Install the SDK (the SDKs are published as ai-auto-router):

pip install ai-auto-router-client

A basic call, with your own key:

from ai_auto_router_client import AutoRouterClient

router = AutoRouterClient(api_key="sg_live_...")  # defaults to https://api.sentiogate.dev

resp = router.chat(
    messages=[{"role": "user", "content": "Explain idempotency keys in two sentences."}],
    provider_keys={"openai": "sk-..."},
)
print(resp["choices"][0]["message"]["content"])

The server already does the routing and fallback, so the SDK doesn't retry by default. If you want a client-side safety net, opt in:

from ai_auto_router_client import AutoRouterClient, RouterClientError

router = AutoRouterClient(api_key="sg_live_...", max_retries=2, max_retry_wait=30)

try:
    resp = router.chat(messages=[{"role": "user", "content": "Hello"}])
except RouterClientError as err:
    # Still limited after the retries you allowed: you get the code and how long to wait.
    print(err.status_code, err.code, err.retry_after)

Opt-in retries only kick in for transient 429s, only after waiting the server's Retry-After, and never when that wait is longer than your max_retry_wait. There's no hot loop and no blind sleep.

To queue on the community pool from Python, pass allow_queue and poll the job:

import time

resp = router.chat(
    messages=[{"role": "user", "content": "Classify this support ticket..."}],
    extra_body={"allow_queue": True},
)
if resp.get("status") == "queued":
    job_id = resp["job_id"]
    while True:
        job = router.poll_batch(job_id)
        if job["status"] in ("completed", "failed"):
            break
        time.sleep(15)
    print(job["response"] or job["error"])

And from JavaScript

npm install @hushamdev/ai-auto-router-sdk

import { AutoRouter } from '@hushamdev/ai-auto-router-sdk';

const router = new AutoRouter({ apiKey: process.env.SENTIOGATE_API_KEY, maxRetries: 2, maxRetryWaitMs: 30000 });

const resp = await router.chat({
  messages: [{ role: 'user', content: 'Hello' }],
  providerKeys: { openai: process.env.OPENAI_API_KEY },
});

for await (const chunk of router.streamChat({ messages: [{ role: 'user', content: 'Stream me a haiku' }] })) {
  process.stdout.write(chunk.choices?.[0]?.delta?.content ?? '');
}

For streams, opt-in retries only apply before the first chunk arrives, so you never get a half-repeated answer.

Self-hosting

There's also a self-hosted Python router that ranks providers using their rate-limit headers, retries transient failures with backoff, and falls back down a chain. Point the SDK's base_url / baseUrl at it. The source isn't public yet; if you need it on your own infra, get in touch.

Practical advice, whatever you use

Read Retry-After. It's the single most useful header in a 429. Handle both forms: seconds and an HTTP date.
Bound every wait. Don't trust a 0 from a flaky upstream, and don't let one header stall a request for an hour. Clamp what you get.
Fall back before you retry. If another model can do the job, switching is almost always faster than waiting.
Retry in one layer only. If your router retries and your HTTP client retries and your app retries, one 429 turns into a burst. That's why SentioGate's SDKs leave retries off unless you ask.
Recover gently. When a window opens, send one request first, not everything you queued.
Connection-level retry middleware isn't enough. It re-sends the same body to the same URL; it can't decide to pick another model.

DEV Community