Inside the ModelPlane routing engine

Every LLM request your app sends is a bet: that the provider you hard-coded will be up, fast, and cheap enough. That bet is why routing layers exist. But a routing layer only helps if it's fast, reliable, and transparent enough to sit on every single call you make. This post walks through exactly what happens inside the ModelPlane routing engine from the moment your request hits the gateway to the moment tokens come back — and why the design keeps accounting off the hot path.

The request lifecycle: auth, resolve, gate, route, account

A single request through ModelPlane passes through five distinct stages. Understanding them matters because each one is a place where a naive gateway either slows you down, leaks credentials, or drops usage data.

1. Authentication — the gateway resolves your Bearer gw-* key into a tenant context.
2. Config resolution — it figures out which routing config applies to this request.
3. Billing gate — a pre-request check confirms you have credits.
4. Routing — it walks the target tree, picks a backend, and makes the upstream call.
5. Accounting — it records usage and decrements credits asynchronously.

Let's go through each stage with the concrete mechanisms, because the details are where the reliability lives.

Stage 1: Authentication — the gw-* key is auth only, never forwarded

The first thing the gateway does is authenticate the request. It resolves the Bearer gw-* token (or a Supabase JWT for management calls) into a TenantContext. That context carries your workspace identity and the credentials needed to call upstream providers.

The critical security property here: the Authorization header containing your gw-* key is stripped from the request before it's ever forwarded upstream. Your gateway key is tenant authentication only. It is never the credential an upstream provider sees. That separation is what makes BYOK safe — your provider keys are encrypted per-tenant and only hydrated at routing time, never exposed to your clients.

Stage 2: Config resolution — request.model is a name you control

Once authenticated, the gateway needs to decide how to route. This is where the core abstraction kicks in: request.model is not a provider model ID. It's a model group — a name you control that maps to a set of targets and a routing strategy.

The resolution order is:

1. Explicit per-request config (x-portkey-config header, legacy) — overrides everything.
2. request.model as a model group — looked up in KV cache first, then Supabase. Credentials for the group's backends are decrypted from their kv_refs at this point.
3. Direct provider call (x-portkey-provider header) — a single-provider bypass.

For most users, path #2 is the one that matters. You define a model group called prod-chat in the dashboard, point it at three backends with a fallback strategy, and then in your code you just send model="prod-chat". The gateway resolves that name to the full routing config.

from openai import OpenAI

client = OpenAI(
    base_url="https://modelplane.dev/v1",
    api_key="gw-...",  # your ModelPlane gateway key
)

# "prod-chat" is a model group, not a provider model ID.
# The gateway resolves it to targets + strategy server-side.
response = client.chat.completions.create(
    model="prod-chat",
    messages=[{"role": "user", "content": "Explain routing engines."}],
)

Stage 3: The billing gate — a snapshot, not a lock

Before routing, the preRequestValidator checks your available credits. If you're out, you get a 402. If you're on the unlimited plan, the gate is skipped entirely.

This is a deliberate design choice: a pre-request balance snapshot gate plus post-request async deduction. It's fast — no distributed transaction on the hot path — but it means there's a theoretical window where concurrent requests could overspend. The team has documented this as a known gap (PRD/002 covers reliability fixes like an outbox pattern and atomic balance updates). For the vast majority of usage, the snapshot gate is the right tradeoff: it catches the "I'm out of credits" case instantly without adding a database round-trip to every inference call.

Stage 4: The routing engine — walking the target tree

This is the heart of the system. The routing engine's tryTargetsRecursively walks the target tree according to the strategy mode you configured on the model group. There are four modes:

Mode	Behavior	Use case
single	One target, no alternatives	Dev/test, or when you want zero ambiguity
fallback	Try in order; advance on specified status codes	High availability — the minimum viable LLM stack
loadbalance	Pick by weight (selectProviderByWeight)	A/B tests, cost-weighted splits
conditional	Evaluate a query DSL against request metadata to pick a named target	Customer-tier routing, region-based routing, feature flags

For each target, tryPost builds the upstream call, runs any configured hooks, checks the cache, and calls the retry handler. If a target fails with a status code that the strategy says to retry on, the engine advances to the next target in the tree.

The fallback mode is where the reliability value lives. A production-grade setup looks like this:

{
  "strategy": { "mode": "fallback" },
  "targets": [
    { "provider": "openai", "model": "gpt-4o" },
    { "provider": "anthropic", "model": "claude-3-5-sonnet" },
    { "provider": "deepseek", "model": "deepseek-v3" }
  ]
}

When OpenAI returns a 5xx or times out, the engine automatically tries Anthropic, then DeepSeek. Your app sees one endpoint and one model name; the gateway handles the chaos.

Stage 5: Accounting — off the hot path, but not off the record

After the upstream response is returned, recordUsage runs asynchronously. It writes to usage_logs and decrements credits, using a reference ID to keep the operation idempotent. This is what keeps the inference hot path fast — the gateway doesn't block your response on a database write.

The honest caveat: because accounting is fire-and-forget, there's a small risk of dropped usage records under extreme load. This is a documented tradeoff, not a hidden bug. The team's roadmap includes moving to a more durable outbox pattern. For now, the async model buys you latency at the cost of a tiny accounting tail-risk.

Why the provider abstraction matters

The routing engine doesn't care which provider it's calling. The provider system abstracts every upstream behind a uniform, OpenAI-compatible interface. Requests are transformed from the OpenAI parameter shape into each provider's native format, and responses are normalized back.

This is what makes the provider catalog so powerful. You can route to OpenAI, Anthropic, Google Gemini, DeepSeek, MiniMax, Zhipu, OpenRouter, or Novita AI — all through the same base_url and the same client library. The gateway handles the protocol differences, the parameter mapping, and the response normalization.

The abstraction also extends to reasoning models. A single thinking parameter on the request is encoded per provider — effort scalars for some, thinking objects for others, passthrough for the rest. Your code doesn't branch on provider; the gateway's ReasoningMapping handles it.

Multi-tenancy without slowing down inference

The routing engine runs in a multi-tenant environment, which means isolation has to be baked in, not bolted on. The design keeps it off the hot path:

• Credentials are encrypted per-tenant (AES-GCM) and stored in Workers KV. They're decrypted only when a model group's config is hydrated.
• Routing config is cached in KV, so the gateway doesn't hit the database on every request.
• Tenancy is workspace-scoped — all resources (API keys, backends, model groups, usage) belong to a workspace, and the billing account owns the plan and credits.

The result: tenant isolation and encrypted credential handling don't add a round-trip to your inference call. The security model is a property of the infrastructure, not a per-request cost.

The target tree is the unit of resilience

Here's the mental model that makes all of this click: your model group is a target tree, and the routing strategy is how the tree is traversed. A single model ID in your code is a leaf. A fallback chain is a branch. A weighted load-balance is a branch with probabilities. A conditional router is a branch with a decision function.

Once you think in target trees, the engineering choices become obvious:

• Fallback isn't a feature — for anything production-bound, it's the minimum viable LLM stack. Provider outages are when-not-if.
• Load-balancing by weight lets you express cost policy in the routing config, not in application code.
• Conditional routing moves business logic (customer tier, region) into the gateway, where it's auditable and changeable without a deploy.

The hot path stays hot

The whole design philosophy can be summarized in one sentence: the inference hot path does only what it must, and everything else happens around it. Auth is a fast token lookup. Config resolution is a KV cache hit. The billing gate is a snapshot check. Routing is an in-memory tree walk. Accounting is async.

That's why ModelPlane can sit on every call your app makes without becoming the bottleneck. The gateway isn't doing anything clever per request — it's doing the right things per request and deferring everything else.

Build your first target tree

The fastest way to understand the routing engine is to build a model group with a fallback chain. Create a group called prod-chat, add two or three backends, set the strategy to fallback, and point your OpenAI client at https://modelplane.dev/v1. Then kill one provider and watch the gateway route around it.

Start free — $5 credits, no card. Your first model group is 60 seconds away.