DEV Community: SentioGate

Building a Rate-Limit-Aware LLM Router in Python: Retries, Backoff & Capacity Scoring

SentioGate — Tue, 20 Oct 2026 16:15:51 +0000

If your app has ever fallen over because a provider returned 429 Too Many Requests, and your fallback was "sleep a second and try again", you know the pain. That pattern hurts twice: you hit the provider again while it's already telling you to slow down, and you still don't know how long to wait.

I built SentioGate partly because I kept writing that loop. Here's how it handles rate limits, and how to use it from Python or JavaScript.

The problem: a 429 is information

A 429 isn't a random failure. It's the provider telling you something specific: this model, on this key, is unavailable for a while. Often it even tells you how long, in a Retry-After header.

Most retry code throws that information away. It re-sends the same request to the same model and hopes. A rate-limit-aware router does the opposite:

Remember that this model on this key is throttled, and for how long.
Route around it to the next healthy candidate.
Come back once the window has passed, carefully.

What SentioGate does about it

SentioGate is an OpenAI-compatible router. You send a chat request with model: "auto" and your own provider keys (BYOK), or use the free community pool, and it picks a model for you. Rate limits are part of that decision, not an afterthought.

Automatic cooldowns

When a provider throttles a request, the hosted service puts that model on that key on a cooldown. It follows the provider's own Retry-After when there is one, so the cooldown lasts as long as the provider asked, within sensible bounds. During the cooldown, that model simply isn't a candidate, and your request moves on without waiting.

Other failures are handled on their own terms. A key that's out of credits, or a model your key can't access, is set aside instead of being retried over and over.

Circuit breakers

Separately from cooldowns, a model that keeps failing gets its breaker tripped, and it's skipped for a while. Breaker state is shared across the whole edge, so once one request finds a sick model, everyone else's requests avoid it too.

Fallback to the next healthy model

With model: "auto", SentioGate ranks candidates on capability, price, and measured latency and success rate. Models that are cooling down or tripped drop out of the ranking. If the chosen model fails, the request falls through to the next healthy one. For streams, that happens before the first token reaches you.

If every candidate is unavailable, you don't get a hang. You get a clear 429 with a Retry-After telling you when to try again.

The community pool: careful recovery

The free community pool is shared by everyone, so it's handled with extra care. When the upstream throttles it, the pool backs off for as long as the upstream asked. When that window ends, it checks the upstream cautiously before letting traffic flow again. That way, a crowd of waiting clients doesn't stampede the provider the instant the window opens.

Queue instead of 429

Some workloads would rather wait than fail. When the shared community pool is full or rate limited, a non-streaming request can opt in to queueing with one header:

curl https://api.sentiogate.dev/v1/chat/completions \
  -H "Authorization: Bearer $SENTIOGATE_API_KEY" \
  -H "Content-Type: application/json" \
  -H "X-SentioGate-Allow-Queue: true" \
  -d '{"model": "auto", "messages": [{"role": "user", "content": "Summarise this ticket..."}]}'

Instead of a 429, you get a 202 with a job_id and a poll_url to collect the result later.

How to use it from Python

Install the SDK (the SDKs are published as ai-auto-router):

pip install ai-auto-router-client

A basic call, with your own key:

from ai_auto_router_client import AutoRouterClient

router = AutoRouterClient(api_key="sg_live_...")  # defaults to https://api.sentiogate.dev

resp = router.chat(
    messages=[{"role": "user", "content": "Explain idempotency keys in two sentences."}],
    provider_keys={"openai": "sk-..."},
)
print(resp["choices"][0]["message"]["content"])

The server already does the routing and fallback, so the SDK doesn't retry by default. If you want a client-side safety net, opt in:

from ai_auto_router_client import AutoRouterClient, RouterClientError

router = AutoRouterClient(api_key="sg_live_...", max_retries=2, max_retry_wait=30)

try:
    resp = router.chat(messages=[{"role": "user", "content": "Hello"}])
except RouterClientError as err:
    # Still limited after the retries you allowed: you get the code and how long to wait.
    print(err.status_code, err.code, err.retry_after)

Opt-in retries only kick in for transient 429s, only after waiting the server's Retry-After, and never when that wait is longer than your max_retry_wait. There's no hot loop and no blind sleep.

To queue on the community pool from Python, pass allow_queue and poll the job:

import time

resp = router.chat(
    messages=[{"role": "user", "content": "Classify this support ticket..."}],
    extra_body={"allow_queue": True},
)
if resp.get("status") == "queued":
    job_id = resp["job_id"]
    while True:
        job = router.poll_batch(job_id)
        if job["status"] in ("completed", "failed"):
            break
        time.sleep(15)
    print(job["response"] or job["error"])

And from JavaScript

npm install @hushamdev/ai-auto-router-sdk

import { AutoRouter } from '@hushamdev/ai-auto-router-sdk';

const router = new AutoRouter({ apiKey: process.env.SENTIOGATE_API_KEY, maxRetries: 2, maxRetryWaitMs: 30000 });

const resp = await router.chat({
  messages: [{ role: 'user', content: 'Hello' }],
  providerKeys: { openai: process.env.OPENAI_API_KEY },
});

for await (const chunk of router.streamChat({ messages: [{ role: 'user', content: 'Stream me a haiku' }] })) {
  process.stdout.write(chunk.choices?.[0]?.delta?.content ?? '');
}

For streams, opt-in retries only apply before the first chunk arrives, so you never get a half-repeated answer.

Self-hosting

There's also a self-hosted Python router that ranks providers using their rate-limit headers, retries transient failures with backoff, and falls back down a chain. Point the SDK's base_url / baseUrl at it. The source isn't public yet; if you need it on your own infra, get in touch.

Practical advice, whatever you use

Read Retry-After. It's the single most useful header in a 429. Handle both forms: seconds and an HTTP date.
Bound every wait. Don't trust a 0 from a flaky upstream, and don't let one header stall a request for an hour. Clamp what you get.
Fall back before you retry. If another model can do the job, switching is almost always faster than waiting.
Retry in one layer only. If your router retries and your HTTP client retries and your app retries, one 429 turns into a burst. That's why SentioGate's SDKs leave retries off unless you ask.
Recover gently. When a window opens, send one request first, not everything you queued.
Connection-level retry middleware isn't enough. It re-sends the same body to the same URL; it can't decide to pick another model.

Ollama, vLLM & Private LLMs: Routing Your Own Models Next to Your Cloud Keys

SentioGate — Tue, 20 Oct 2026 16:15:12 +0000

You have cloud keys in your app, and a GPU box in the corner running vLLM or Ollama. Here's how SentioGate puts both behind one endpoint, with one request format, one stream format and one dashboard.

A question I get a lot: "I use the cloud for the hard tasks, but I have my own box for the easy stuff. Can one router handle both, without a separate integration for each?"

Yes, as long as your box speaks the OpenAI chat-completions API. vLLM, Ollama, LM Studio and most private inference appliances do.

The problem with "just call the API"

Most teams end up with something like this:

if task_difficulty > 0.7:
    response = openai_client.chat(...)      # cloud
else:
    response = ollama_client.generate(...)  # local

That gives you two SDKs, two error shapes, two streaming formats, no failover and no shared telemetry. Every new model means another branch, and every outage means a code change.

A router fixes the plumbing: one endpoint, one request format, one stream format, and one place to see what ran and what it cost.

Registering your endpoint

In SentioGate, your own endpoint is a custom provider. It's a provider_id, a base_url, an api_key, and the list of model IDs it serves.

curl -X POST https://api.sentiogate.dev/v1/custom-providers \
  -H "Authorization: Bearer $SENTIOGATE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "provider_id": "my-vllm",
    "display_name": "My vLLM",
    "base_url": "https://llm.example.com/v1",
    "api_key": "YOUR_ENDPOINT_KEY",
    "models": ["llama3.1:8b", "qwen2.5:14b"]
  }'

A few things to get right:

Make it reachable. The hosted service calls your model from the edge, so base_url must be reachable from the internet. A URL like http://localhost:11434 or a LAN hostname won't work. Put the endpoint behind HTTPS.
Protect it. An inference server on the open internet should require a key. vLLM supports --api-key, and for Ollama a small reverse proxy that checks a bearer token does the job.
Put the credential in api_key. It's encrypted at rest with AES-256-GCM, like every other key in your Vault, and decrypted in memory only when SentioGate calls your endpoint.

Calling your models

Once registered, you can call a model explicitly as provider_id/model:

from ai_auto_router_client import AutoRouterClient

router = AutoRouterClient(api_key="sg_live_...")

resp = router.chat(
    model="my-vllm/qwen2.5:14b",
    messages=[{"role": "user", "content": "Summarise this ticket in one line: ..."}],
)
print(resp["choices"][0]["message"]["content"])

Or from JavaScript:

import { AutoRouter } from '@hushamdev/ai-auto-router-sdk';

const router = new AutoRouter({ apiKey: process.env.SENTIOGATE_API_KEY });
const res = await router.chat({
  model: 'my-vllm/qwen2.5:14b',
  messages: [{ role: 'user', content: 'Summarise this ticket in one line: ...' }],
});
console.log(res.choices[0].message.content);

Your models also become candidates for "model": "auto", right next to the cloud models your keys can reach.

How `auto` treats your models

With auto, SentioGate picks a model per request from everything you've enabled. It looks at whether a model is capable enough for the task, whether the conversation fits its context window, what it costs, and how fast and reliable it has been recently. That last part is measured, not configured: if your box slows down or starts erroring, it drops down the list, and requests fall back to the next candidate before any bytes reach your client.

You stay in control of the pool:

Want your own model as the primary? Call it explicitly (my-vllm/...), or enable only your own models in the Vault so auto chooses among them.
Want cloud and local side by side? Enable both, and auto picks per request across the whole pool, with fallback between providers.
Want guardrails? Your policy (enabled models and spend caps) applies to every request, whichever provider serves it.

And you never have to guess what happened. Every response says which model was picked and why, in the router block, or in the X-SentioGate-Router header when streaming.

Everything else in the pipeline still applies

Your models go through the same request path as the cloud ones:

Streaming: standard OpenAI-style SSE, so your front end doesn't care where the tokens came from.
Fallback: on auto requests, a failed candidate moves on to the next one.
Context compression: long chats are kept inside a token budget, exactly as for cloud models, and router.compression in the response shows what was saved.
Telemetry: provider, model, latency and tokens show up in the same dashboard as everything else.

Streaming from your own model looks exactly like streaming from OpenAI:

for await (const chunk of router.streamChat({
  model: 'my-vllm/qwen2.5:14b',
  messages: [{ role: 'user', content: 'Write a haiku about GPUs' }],
})) {
  process.stdout.write(chunk.choices?.[0]?.delta?.content || '');
}

(Python: router.stream_chat(...).)

The hybrid pattern

The setup I'd recommend:

Cloud models for the tasks that need them: hard reasoning and long context.
Your own models, called explicitly for the workloads you know they handle well, such as classification, extraction, short summaries and internal tools.
auto wherever you want SentioGate to choose per request and fail over automatically.

You can mix all three in one app with one client, and pass cloud keys per request (provider_keys / providerKeys) if you'd rather not store them. Keys passed that way are used for that request only.

It's a simple mental model: explicit when you know what you want, auto when you want the router to decide, and one response format either way.

Want everything, router included, on your own infrastructure? The source isn't public yet, so get in touch.

How I Cut Long-Chat Token Bills by 85-95% With Semantic Context Compression

SentioGate — Tue, 20 Oct 2026 16:07:51 +0000

How I Cut Long-Chat Token Bills by 85–95% With Semantic Context Compression

A practical engineering post on semantic context compression for LLM agents — why "sliding window" truncation breaks agents, how a router detects the threshold, and what a good restore point looks like.

If you've built an agent that runs for hours or spans hundreds of turns, you know the bill: every turn you re-send the whole conversation to the model. Token cost grows linearly with session length, and eventually you hit the context window and the whole thing breaks.

The usual fix — sliding-window truncation ("keep the last N messages") — is terrible for agents. You throw away the plan you made in turn 3, the user's stated constraints from turn 10, and the decision you made at turn 47. The model starts repeating itself, contradicting earlier work, and the user notices immediately.

Semantic context compression is the fix. And it's one of the features I care about most in SentioGate, the BYOK LLM router I'm building.

What "semantic compression" means (vs. naive truncation)

Approach	What's kept	What's lost	Cost at turn 200
Sliding window (last 20 msgs)	Recent chatter	Plan, constraints, decisions	High — model re-derives context every turn
Naive "summarize older msgs"	A text blob	Structured state (files, tools, user intent)	Medium — summary drifts, loses key facts
Semantic compression + restore point	Structured state: goals, files, tool calls, user intent, open threads	Verbose intermediate chatter	Low — the compact state carries 90%+ of the useful signal

The key insight: a long conversation isn't a bag of text. It's state — goals, constraints, decisions, open tasks, files being edited, tools being called. If you compress the state rather than the prose, you get 85–95% token reduction with near-zero loss of continuity.

How the router detects the moment to compact

The router watches each session's message tree. Two signals trigger compaction:

Capacity threshold — when the session's context reaches a configured fraction of the model's context window (e.g., 75%), compact before the next turn overflows.
Token threshold — when the cumulative input tokens for the session cross a budget, compact proactively even if you're far from the window.

The compaction preserves the most recent N messages verbatim (default: 4) and collapses everything older into a structured restore point — a compact, machine-readable summary of the session's state.

What the restore point looks like

The restore point isn't a free-text summary. It's a structured document with slots for:

Goals — what the agent is trying to accomplish
User intent & constraints — anything the user explicitly asked for or ruled out
Files / resources — files being edited, configs, data sources
Tool state — which tools have been called, what's pending
Decisions made — choices that affect downstream turns
Open threads — things not yet resolved

When a new model (or a fresh session) picks up the conversation, it reads the restore point instead of the full history. Input tokens drop from, say, 60,000 to 4,000–8,000, and the model has everything it needs to continue without re-deriving context.

Why this matters for model-hopping

A big unlock: because the restore point is model-agnostic, you can switch models mid-conversation — finish a planning turn with a frontier model, then hand the execution to a cheaper / faster model, then back to the frontier model for a final review. Each handoff reads the same compact state. No re-reading of full history.

That's the "seamless model-hopping" I mentioned in my intro post to SentioGate. The compaction engine is what makes it possible.

What to watch for (and what I'm still iterating on)

Loss of rare-but-critical facts. The rare user message that's important later ("no, don't touch the prod DB") can get lost in compression if it's buried. The preserve-recent-N window helps, but I'm working on fact-extraction heuristics that flag "constraint" messages regardless of recency.
Cross-session continuity. Today, compaction is per-session. Multi-session agents (a "planner" session feeding an "executor" session) need a shared compact state. That's on the roadmap.
Quality scoring of the compact output. The restore point is only as good as the model that wrote it. I'm adding a lightweight QA pass that checks the compact against the original for the most critical facts before it's used.

The number I'm most proud of

85–95% input-token reduction on long sessions, measured across real agent workloads, with no measurable drop in task-completion quality. That's the entire value proposition of the compression engine in one sentence.

Links

If you're building long-running agents and your token bill is a problem, drop a comment — I'll follow up with a post on the exact scoring algorithm and how to tune the compaction thresholds for your workloads.

Stop Juggling 7 LLM API Keys: One BYOK Router That Ranks, Streams & Compacts for You

SentioGate — Tue, 20 Oct 2026 16:07:24 +0000

Stop Juggling 7 LLM API Keys: One BYOK Router That Ranks, Streams & Compacts for You

SentioGate is a production-ready BYOK (Bring Your Own Key) routing layer for LLM APIs. Supply your provider keys with each request, and the router picks the best provider for that task, streams OpenAI-compatible responses back, retries and fails over automatically, and — when sessions get long — compresses context by 85–95% without losing continuity.

One OpenAI-compatible endpoint. No per-token markup. Drop-in SDKs for Python and JavaScript/TypeScript in three lines.

The 3-line drop-in

Python

pip install ai-auto-router-client

from ai_auto_router_client import AutoRouterClient

router = AutoRouterClient(api_key="sg_live_...")
resp = router.chat(
    messages=[{"role": "user", "content": "Explain raft consensus in 2 sentences"}],
    provider_keys={"openai": "sk-...", "anthropic": "sk-ant-..."}
)
print(resp["choices"][0]["message"]["content"])

JavaScript / TypeScript

npm install @hushamdev/ai-auto-router-sdk

import { AutoRouter } from '@hushamdev/ai-auto-router-sdk';

const router = new AutoRouter({ apiKey: process.env.SENTIOGATE_API_KEY });
const res = await router.chat({
  messages: [{ role: 'user', content: 'Explain raft consensus in 2 sentences' }],
  providerKeys: {
    openai: process.env.OPENAI_API_KEY,
    anthropic: process.env.ANTHROPIC_API_KEY
  }
});
console.log(res.choices[0].message.content);

Both SDKs are built purely on the standard library — no httpx, no axios, no heavy runtime. Apache-2.0 licensed.

What it does

1. Multi-provider routing with automatic failover

Route across OpenAI, Anthropic, Google Gemini, Groq, Mistral, xAI, DeepSeek, OpenRouter — and your own self-hosted endpoints — through a single OpenAI-compatible API. The router continuously scores each provider on cost, latency, reliability, capacity and quality, and automatically fails over when a provider rate-limits or errors out. You don't write the retry loop.

2. Hardened transport

Wall-clock request deadlines so a hung provider can't stall your app
Transient-error retries with exponential backoff
Rate-limit-aware capacity scoring read from provider response headers

3. Streaming, OpenAI-style

Pass stream: true and receive standard OpenAI-style SSE chunks plus a final usage/cost event — so your front-end keeps working even when the router picks a different backend mid-stream.

4. Semantic context compression

For long agent / chat sessions, the router detects when your message tree approaches the model's context window and automatically compresses older history into a compact restore point, cutting input tokens by 85–95% while letting a different model pick up the conversation seamlessly.

5. Custom providers — route to any OpenAI-compatible endpoint

Add Ollama, vLLM, LM Studio, or a private LLM appliance at runtime. They participate in the same scoring / failover pipeline as the cloud providers. Your local models, routed intelligently next to your frontier keys.

6. Observability & governance

Prometheus metrics, structured logging, health checks, request tracing, and a built-in encrypted key vault so provider keys stay in scope, in memory, and never on disk.

When to reach for it

You're running multiple LLM providers and want one stable API for your app
Your agent / long-chat products are burning through token budgets
You want local models (Ollama, vLLM) in the same routing pool as the cloud
You need a BYOK layer so your customers' keys never touch your servers
You want to route on real cost/latency/health instead of hard-coding a provider

Building a Rate-Limit-Aware LLM Router in Python: Retries, Backoff & Capacity Scoring

SentioGate — Tue, 20 Oct 2026 15:40:34 +0000

I built SentioGate partly because I kept writing that loop. Here's how it handles rate limits, and how to use it from Python or JavaScript.

The problem: a 429 is information

A 429 isn't a random failure. It's the provider telling you something specific: this model, on this key, is unavailable for a while. Often it even tells you how long, in a Retry-After header.

Most retry code throws that information away. It re-sends the same request to the same model and hopes. A rate-limit-aware router does the opposite:

Remember that this model on this key is throttled, and for how long.
Route around it to the next healthy candidate.
Come back once the window has passed, carefully.

What SentioGate does about it

Automatic cooldowns

Other failures are handled on their own terms. A key that's out of credits, or a model your key can't access, is set aside instead of being retried over and over.

Circuit breakers

Fallback to the next healthy model

If every candidate is unavailable, you don't get a hang. You get a clear 429 with a Retry-After telling you when to try again.

The community pool: careful recovery

Queue instead of 429

Some workloads would rather wait than fail. When the shared community pool is full or rate limited, a non-streaming request can opt in to queueing with one header:

curl https://api.sentiogate.dev/v1/chat/completions \
  -H "Authorization: Bearer $SENTIOGATE_API_KEY" \
  -H "Content-Type: application/json" \
  -H "X-SentioGate-Allow-Queue: true" \
  -d '{"model": "auto", "messages": [{"role": "user", "content": "Summarise this ticket..."}]}'

Instead of a 429, you get a 202 with a job_id and a poll_url to collect the result later.

How to use it from Python

Install the SDK (the SDKs are published as ai-auto-router):

pip install ai-auto-router-client

A basic call, with your own key:

from ai_auto_router_client import AutoRouterClient

router = AutoRouterClient(api_key="sg_live_...")  # defaults to https://api.sentiogate.dev

resp = router.chat(
    messages=[{"role": "user", "content": "Explain idempotency keys in two sentences."}],
    provider_keys={"openai": "sk-..."},
)
print(resp["choices"][0]["message"]["content"])

The server already does the routing and fallback, so the SDK doesn't retry by default. If you want a client-side safety net, opt in:

from ai_auto_router_client import AutoRouterClient, RouterClientError

router = AutoRouterClient(api_key="sg_live_...", max_retries=2, max_retry_wait=30)

try:
    resp = router.chat(messages=[{"role": "user", "content": "Hello"}])
except RouterClientError as err:
    # Still limited after the retries you allowed: you get the code and how long to wait.
    print(err.status_code, err.code, err.retry_after)

Opt-in retries only kick in for transient 429s, only after waiting the server's Retry-After, and never when that wait is longer than your max_retry_wait. There's no hot loop and no blind sleep.

To queue on the community pool from Python, pass allow_queue and poll the job:

import time

resp = router.chat(
    messages=[{"role": "user", "content": "Classify this support ticket..."}],
    extra_body={"allow_queue": True},
)
if resp.get("status") == "queued":
    job_id = resp["job_id"]
    while True:
        job = router.poll_batch(job_id)
        if job["status"] in ("completed", "failed"):
            break
        time.sleep(15)
    print(job["response"] or job["error"])

And from JavaScript

npm install @hushamdev/ai-auto-router-sdk

import { AutoRouter } from '@hushamdev/ai-auto-router-sdk';

const router = new AutoRouter({ apiKey: process.env.SENTIOGATE_API_KEY, maxRetries: 2, maxRetryWaitMs: 30000 });

const resp = await router.chat({
  messages: [{ role: 'user', content: 'Hello' }],
  providerKeys: { openai: process.env.OPENAI_API_KEY },
});

for await (const chunk of router.streamChat({ messages: [{ role: 'user', content: 'Stream me a haiku' }] })) {
  process.stdout.write(chunk.choices?.[0]?.delta?.content ?? '');
}

For streams, opt-in retries only apply before the first chunk arrives, so you never get a half-repeated answer.

Self-hosting

Practical advice, whatever you use

Read Retry-After. It's the single most useful header in a 429. Handle both forms: seconds and an HTTP date.
Bound every wait. Don't trust a 0 from a flaky upstream, and don't let one header stall a request for an hour. Clamp what you get.
Fall back before you retry. If another model can do the job, switching is almost always faster than waiting.
Retry in one layer only. If your router retries and your HTTP client retries and your app retries, one 429 turns into a burst. That's why SentioGate's SDKs leave retries off unless you ask.
Recover gently. When a window opens, send one request first, not everything you queued.
Connection-level retry middleware isn't enough. It re-sends the same body to the same URL; it can't decide to pick another model.

Ollama, vLLM & Private LLMs: Routing Your Own Models Next to Your Cloud Keys

SentioGate — Tue, 20 Oct 2026 15:31:12 +0000

You have cloud keys in your app, and a GPU box in the corner running vLLM or Ollama. Here's how SentioGate puts both behind one endpoint, with one request format, one stream format and one dashboard.

A question I get a lot: "I use the cloud for the hard tasks, but I have my own box for the easy stuff. Can one router handle both, without a separate integration for each?"

Yes, as long as your box speaks the OpenAI chat-completions API. vLLM, Ollama, LM Studio and most private inference appliances do.

The problem with "just call the API"

Most teams end up with something like this:

if task_difficulty > 0.7:
    response = openai_client.chat(...)      # cloud
else:
    response = ollama_client.generate(...)  # local

That gives you two SDKs, two error shapes, two streaming formats, no failover and no shared telemetry. Every new model means another branch, and every outage means a code change.

A router fixes the plumbing: one endpoint, one request format, one stream format, and one place to see what ran and what it cost.

Registering your endpoint

In SentioGate, your own endpoint is a custom provider. It's a provider_id, a base_url, an api_key, and the list of model IDs it serves.

curl -X POST https://api.sentiogate.dev/v1/custom-providers \
  -H "Authorization: Bearer $SENTIOGATE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "provider_id": "my-vllm",
    "display_name": "My vLLM",
    "base_url": "https://llm.example.com/v1",
    "api_key": "YOUR_ENDPOINT_KEY",
    "models": ["llama3.1:8b", "qwen2.5:14b"]
  }'

A few things to get right:

Make it reachable. The hosted service calls your model from the edge, so base_url must be reachable from the internet. A URL like http://localhost:11434 or a LAN hostname won't work. Put the endpoint behind HTTPS.
Protect it. An inference server on the open internet should require a key. vLLM supports --api-key, and for Ollama a small reverse proxy that checks a bearer token does the job.
Put the credential in api_key. It's encrypted at rest with AES-256-GCM, like every other key in your Vault, and decrypted in memory only when SentioGate calls your endpoint.

Calling your models

Once registered, you can call a model explicitly as provider_id/model:

from ai_auto_router_client import AutoRouterClient

router = AutoRouterClient(api_key="sg_live_...")

resp = router.chat(
    model="my-vllm/qwen2.5:14b",
    messages=[{"role": "user", "content": "Summarise this ticket in one line: ..."}],
)
print(resp["choices"][0]["message"]["content"])

Or from JavaScript:

import { AutoRouter } from '@hushamdev/ai-auto-router-sdk';

const router = new AutoRouter({ apiKey: process.env.SENTIOGATE_API_KEY });
const res = await router.chat({
  model: 'my-vllm/qwen2.5:14b',
  messages: [{ role: 'user', content: 'Summarise this ticket in one line: ...' }],
});
console.log(res.choices[0].message.content);

Your models also become candidates for "model": "auto", right next to the cloud models your keys can reach.

How `auto` treats your models

You stay in control of the pool:

Want your own model as the primary? Call it explicitly (my-vllm/...), or enable only your own models in the Vault so auto chooses among them.
Want cloud and local side by side? Enable both, and auto picks per request across the whole pool, with fallback between providers.
Want guardrails? Your policy (enabled models and spend caps) applies to every request, whichever provider serves it.

And you never have to guess what happened. Every response says which model was picked and why, in the router block, or in the X-SentioGate-Router header when streaming.

Everything else in the pipeline still applies

Your models go through the same request path as the cloud ones:

Streaming: standard OpenAI-style SSE, so your front end doesn't care where the tokens came from.
Fallback: on auto requests, a failed candidate moves on to the next one.
Context compression: long chats are kept inside a token budget, exactly as for cloud models, and router.compression in the response shows what was saved.
Telemetry: provider, model, latency and tokens show up in the same dashboard as everything else.

Streaming from your own model looks exactly like streaming from OpenAI:

for await (const chunk of router.streamChat({
  model: 'my-vllm/qwen2.5:14b',
  messages: [{ role: 'user', content: 'Write a haiku about GPUs' }],
})) {
  process.stdout.write(chunk.choices?.[0]?.delta?.content || '');
}

(Python: router.stream_chat(...).)

The hybrid pattern

The setup I'd recommend:

Cloud models for the tasks that need them: hard reasoning and long context.
Your own models, called explicitly for the workloads you know they handle well, such as classification, extraction, short summaries and internal tools.
auto wherever you want SentioGate to choose per request and fail over automatically.

It's a simple mental model: explicit when you know what you want, auto when you want the router to decide, and one response format either way.

Want everything, router included, on your own infrastructure? The source isn't public yet, so get in touch.

How I Cut Long-Chat Token Bills by 85-95% With Semantic Context Compression

SentioGate — Tue, 20 Oct 2026 15:30:47 +0000

How I Cut Long-Chat Token Bills by 85–95% With Semantic Context Compression

A practical engineering post on semantic context compression for LLM agents — why "sliding window" truncation breaks agents, how a router detects the threshold, and what a good restore point looks like.

Semantic context compression is the fix. And it's one of the features I care about most in SentioGate, the BYOK LLM router I'm building.

What "semantic compression" means (vs. naive truncation)

Approach	What's kept	What's lost	Cost at turn 200
Sliding window (last 20 msgs)	Recent chatter	Plan, constraints, decisions	High — model re-derives context every turn
Naive "summarize older msgs"	A text blob	Structured state (files, tools, user intent)	Medium — summary drifts, loses key facts
Semantic compression + restore point	Structured state: goals, files, tool calls, user intent, open threads	Verbose intermediate chatter	Low — the compact state carries 90%+ of the useful signal

How the router detects the moment to compact

The router watches each session's message tree. Two signals trigger compaction:

Capacity threshold — when the session's context reaches a configured fraction of the model's context window (e.g., 75%), compact before the next turn overflows.
Token threshold — when the cumulative input tokens for the session cross a budget, compact proactively even if you're far from the window.

What the restore point looks like

The restore point isn't a free-text summary. It's a structured document with slots for:

Goals — what the agent is trying to accomplish
User intent & constraints — anything the user explicitly asked for or ruled out
Files / resources — files being edited, configs, data sources
Tool state — which tools have been called, what's pending
Decisions made — choices that affect downstream turns
Open threads — things not yet resolved

Why this matters for model-hopping

That's the "seamless model-hopping" I mentioned in my intro post to SentioGate. The compaction engine is what makes it possible.

What to watch for (and what I'm still iterating on)

Loss of rare-but-critical facts. The rare user message that's important later ("no, don't touch the prod DB") can get lost in compression if it's buried. The preserve-recent-N window helps, but I'm working on fact-extraction heuristics that flag "constraint" messages regardless of recency.
Cross-session continuity. Today, compaction is per-session. Multi-session agents (a "planner" session feeding an "executor" session) need a shared compact state. That's on the roadmap.
Quality scoring of the compact output. The restore point is only as good as the model that wrote it. I'm adding a lightweight QA pass that checks the compact against the original for the most critical facts before it's used.

The number I'm most proud of

Stop Juggling 7 LLM API Keys: One BYOK Router That Ranks, Streams & Compacts for You

SentioGate — Tue, 20 Oct 2026 15:22:06 +0000

Stop Juggling 7 LLM API Keys: One BYOK Router That Ranks, Streams & Compacts for You

One OpenAI-compatible endpoint. No per-token markup. Drop-in SDKs for Python and JavaScript/TypeScript in three lines.

The 3-line drop-in

Python

pip install ai-auto-router-client

from ai_auto_router_client import AutoRouterClient

router = AutoRouterClient(api_key="sg_live_...")
resp = router.chat(
    messages=[{"role": "user", "content": "Explain raft consensus in 2 sentences"}],
    provider_keys={"openai": "sk-...", "anthropic": "sk-ant-..."}
)
print(resp["choices"][0]["message"]["content"])

JavaScript / TypeScript

npm install @hushamdev/ai-auto-router-sdk

import { AutoRouter } from '@hushamdev/ai-auto-router-sdk';

const router = new AutoRouter({ apiKey: process.env.SENTIOGATE_API_KEY });
const res = await router.chat({
  messages: [{ role: 'user', content: 'Explain raft consensus in 2 sentences' }],
  providerKeys: {
    openai: process.env.OPENAI_API_KEY,
    anthropic: process.env.ANTHROPIC_API_KEY
  }
});
console.log(res.choices[0].message.content);

Both SDKs are built purely on the standard library — no httpx, no axios, no heavy runtime. Apache-2.0 licensed.

What it does

1. Multi-provider routing with automatic failover

2. Hardened transport

Wall-clock request deadlines so a hung provider can't stall your app
Transient-error retries with exponential backoff
Rate-limit-aware capacity scoring read from provider response headers

3. Streaming, OpenAI-style

Pass stream: true and receive standard OpenAI-style SSE chunks plus a final usage/cost event — so your front-end keeps working even when the router picks a different backend mid-stream.

4. Semantic context compression

5. Custom providers — route to any OpenAI-compatible endpoint

6. Observability & governance

Prometheus metrics, structured logging, health checks, request tracing, and a built-in encrypted key vault so provider keys stay in scope, in memory, and never on disk.

When to reach for it

You're running multiple LLM providers and want one stable API for your app
Your agent / long-chat products are burning through token budgets
You want local models (Ollama, vLLM) in the same routing pool as the cloud
You need a BYOK layer so your customers' keys never touch your servers
You want to route on real cost/latency/health instead of hard-coding a provider

DEV Community: SentioGate

Building a Rate-Limit-Aware LLM Router in Python: Retries, Backoff & Capacity Scoring

The problem: a 429 is information

What SentioGate does about it

Automatic cooldowns

Circuit breakers

Fallback to the next healthy model

The community pool: careful recovery

Queue instead of 429

How to use it from Python

And from JavaScript

Self-hosting

Practical advice, whatever you use

Links

Ollama, vLLM & Private LLMs: Routing Your Own Models Next to Your Cloud Keys

The problem with "just call the API"

Registering your endpoint

Calling your models

How auto treats your models

Everything else in the pipeline still applies

The hybrid pattern

Links

How I Cut Long-Chat Token Bills by 85-95% With Semantic Context Compression

How I Cut Long-Chat Token Bills by 85–95% With Semantic Context Compression

What "semantic compression" means (vs. naive truncation)

How the router detects the moment to compact

What the restore point looks like

Why this matters for model-hopping

What to watch for (and what I'm still iterating on)

The number I'm most proud of

Links

Stop Juggling 7 LLM API Keys: One BYOK Router That Ranks, Streams & Compacts for You

Stop Juggling 7 LLM API Keys: One BYOK Router That Ranks, Streams & Compacts for You

The 3-line drop-in

Python

JavaScript / TypeScript

What it does

1. Multi-provider routing with automatic failover

2. Hardened transport

3. Streaming, OpenAI-style

4. Semantic context compression

5. Custom providers — route to any OpenAI-compatible endpoint

6. Observability & governance

When to reach for it

Links

Building a Rate-Limit-Aware LLM Router in Python: Retries, Backoff & Capacity Scoring

The problem: a 429 is information

What SentioGate does about it

Automatic cooldowns

Circuit breakers

Fallback to the next healthy model

The community pool: careful recovery

Queue instead of 429

How to use it from Python

And from JavaScript

Self-hosting

Practical advice, whatever you use

Links

Ollama, vLLM & Private LLMs: Routing Your Own Models Next to Your Cloud Keys

The problem with "just call the API"

Registering your endpoint

Calling your models

How auto treats your models

Everything else in the pipeline still applies

The hybrid pattern

Links

How I Cut Long-Chat Token Bills by 85-95% With Semantic Context Compression

How I Cut Long-Chat Token Bills by 85–95% With Semantic Context Compression

What "semantic compression" means (vs. naive truncation)

How the router detects the moment to compact

What the restore point looks like

Why this matters for model-hopping

What to watch for (and what I'm still iterating on)

The number I'm most proud of

Links

Stop Juggling 7 LLM API Keys: One BYOK Router That Ranks, Streams & Compacts for You

Stop Juggling 7 LLM API Keys: One BYOK Router That Ranks, Streams & Compacts for You

The 3-line drop-in

Python

JavaScript / TypeScript

How `auto` treats your models

How `auto` treats your models