How We Route 1M+ API Calls Daily Across 60 Models

A unified AI API has a simple face. One key, one base_url, one price list, one balance. Everything behind that face is hard.

The endpoint on the other side of tokenpapa.ai/v1 does not call one model. It fans out to 65 live model IDs across a dozen independent labs, each with its own auth, its own rate-limit model, its own idea of what a token is, its own error vocabulary and its own bad afternoons. The gateway's job is to make that set behave like one dependable service.

This is a design article, not a victory lap. It describes the layers a high-volume unified gateway needs, and the failure modes each layer exists to absorb.

Dimension	What it means in practice
Model IDs exposed	65 live IDs across DeepSeek, OpenAI, Anthropic, Google, Alibaba, Moonshot, Zhipu, MiniMax, Tencent, ByteDance and Xiaomi
Client surface	One OpenAI-compatible endpoint: https://tokenpapa.ai/v1
Traffic shape	Bursty, mixed streaming and non-streaming, wildly uneven per-model
Failure domain	Any single upstream can degrade, rate-limit, re-price or vanish
Billing unit	Per token, per model, with cache hits priced differently
Most dangerous bug class	Silent wrong routing, which shows up as a wrong bill, not a stack trace

Key insight: at scale, a gateway's job is not to forward requests. It is to make dozens of heterogeneous, independently failing upstreams look like one endpoint with one price list and one predictable error contract.

---

The problem, stated honestly

Serving one model is a proxy. Serving 60 is a scheduling and truth-maintenance problem.

Four properties make LLM routing different from routing ordinary HTTP traffic:

1. Cost per request is not constant. Two requests to the same model can differ by three orders of magnitude in cost, depending on input length and on how much output the model decides to produce. Routing decisions are pricing decisions.
2. Failures are partial. An upstream can accept a connection, return HTTP 200, and then stream nothing, or stream half a sentence and stall. Status codes are not enough.
3. Streaming is the default. Most production traffic uses server-sent events, so a failure can arrive 40 tokens into a 900-token answer.
4. The unit of consumption is a token, not a call. Rate limits, quotas and billing all operate on token counts that are only known after the response finishes.

Every layer below exists because of one of those four properties.

---

Layer 1 — The channel registry

A channel is one upstream credential pointing at one model. The same model ID usually has several channels, deliberately: two DeepSeek V4 Flash channels from different reseller routes fail at different times, and their latency and effective price differ.

The registry is the source of truth for routing. Each channel carries:

Field	Purpose
Model IDs served	Maps an incoming model= to candidate channels
Provider and region	Groups channels for policy and latency assumptions
Weight and priority	Controls traffic split inside a model
Concurrency and token caps	Local view of the upstream quota
Price override	Per-channel input, output and cache rates
Health state	Live status: healthy, degraded, cooling, disabled
Cost tier	Used when a caller asks to optimise for price over speed

The registry is read on every request and written rarely. That asymmetry is the whole design constraint: reads must be a memory lookup, not a database query, or the gateway adds latency to every call it serves.

Key takeaway: routing tables are configuration, but they are read at request rate. Keep them in memory, version them, and treat every write as a hot reload rather than a migration.

---

Layer 2 — Health, circuit breaking and failover

The naive approach — mark a channel down when it returns a 5xx — fails on all three counts that matter: the upstream that returns 200 and stalls, the upstream that is slow rather than broken, and the upstream whose failure is specific to one model.

A working health model watches four signals:

• Liveness. A cheap periodic probe per channel, scoped to the model it serves.
• Passive health from real traffic. Rolling success rate, time-to-first-token and stall rate computed from production requests rather than probes.
• Circuit breaking. After a threshold of consecutive failures, the channel is cooled — removed from rotation for a short window, then tested with a small fraction of traffic before full reinstatement.
• Error classification. Not every 4xx is the channel's fault. A malformed request should never trip a breaker; a 401, a 429 storm, or repeated 5xx should.

Failover then splits into two cases, and conflating them is the classic mistake:

Failure point	Correct behaviour	Why
Before the first token reaches the client	Retry on a second healthy channel for the same model	The caller cannot observe the difference except as extra latency
After streaming has started	Close the stream, report the failure, let the client decide	Silently swapping upstreams mid-answer produces a Frankenstein response
Upstream returned 200 with empty content	Treat as failure, retry once, then surface	A 200 with nothing in it is the most common silent failure
Upstream rate-limited (429)	Route to a different channel rather than backing off immediately	Retrying the same exhausted channel is the worst option available

The third row is the one that costs teams real money in debugging time. An upstream that returns a well-formed empty completion looks like a success in every metric that only counts status codes.

---

Layer 3 — Choosing among healthy channels

Once more than one channel can serve a model, the gateway needs a policy. Five policies cover almost all real traffic:

1. Weighted round-robin — the default. Spread load, keep every healthy channel warm, avoid over-concentrating on one route.
2. Least-in-flight — send to the channel with fewest active requests. Best when upstreams have different concurrency ceilings.
3. Cost-first — always pick the cheapest healthy channel for the requested model. Correct for batch and non-urgent traffic.
4. Latency-first — pick by recent time-to-first-token, with a decay so a single fast sample cannot dominate.
5. Stickiness — pin a caller or a session to a channel, used sparingly for prompt-cache affinity.

The important engineering detail is that policy selection must be cheap and stable. A router that recomputes a global cost model on every request will collapse under its own bookkeeping long before the upstreams do.

Key insight: cost-first routing is where a gateway earns its keep. According to published TokenPAPA platform rates (September 2026), DeepSeek V4 Flash input is $0.14 per 1M tokens against $13.50 for GPT-5.6 Sol — so a routing policy that keeps routine traffic on the budget tier is worth far more than any negotiated discount.

---

Layer 4 — Caching, in two flavours

Caching at the gateway happens at two levels, and they have different economics.

Exact-match response caching stores a full request-response pair under a hash of the model, the messages and the sampling parameters. It is the cheapest possible win, but only when the same prompt recurs verbatim — and it is only correct for deterministic settings. With a non-zero temperature, returning a cached completion silently changes the product's behaviour.

Prompt-prefix caching is upstream and matters much more. DeepSeek's automatic context caching cuts repeat-input cost by roughly 90% when the prompt prefix is byte-stable between calls: the same system prompt, the same retrieved context, the same tool definitions. The gateway's contribution is to avoid defeating it — not reordering system messages, not injecting a timestamp into the prompt, not jittering tool ordering on each request.

That is a design rule with revenue attached. A single non-deterministic token at the top of a system prompt, rebuilt per request, can invalidate the cache for every call in the fleet.

---

Layer 5 — Rate limits, quotas and fairness

Rate limiting has three distinct jobs, and a single limiter cannot do all of them:

Limit type	Scope	Purpose
Request rate	Per API key, per minute	Blunt abuse protection
Token throughput	Per API key, per minute, input plus output	The limit that actually corresponds to cost
Concurrency	Per key and per channel	Protects slow upstreams from being buried
Budget quota	Per key, per day or month	Stops a runaway loop before it becomes a bill

Token throughput is the one that matters and the one that is hardest, because output tokens are discovered only as a stream progresses. The practical approach is to reserve an allowance at request time based on max_tokens, then reconcile against actual usage when the stream completes. Without that reservation step, a large number of long-output requests can all pass the pre-flight check simultaneously.

Per-channel limits are equally important. A client's 429 should not be caused by another client saturating the same upstream route, so channel concurrency has to be tracked separately from key concurrency, and the two budgets reconciled at the point of routing.

---

Layer 6 — Observability and billing truth

The observability layer has one non-negotiable requirement: every request must be attributable after the fact. At minimum, a request record needs the API key, the requested model, the channel that actually served it, the routing decision that chose it, the upstream request ID, input and output token counts, cache-hit tokens, time-to-first-token, total duration, the retry chain, and the computed cost.

That record is what makes three otherwise impossible questions answerable:

• Why was last month's bill 30% higher with flat traffic volume?
• Which upstream caused the latency regression at 03:00?
• Did this model's quality change, or did traffic quietly move to a different channel?

Billing is the same data viewed from the other end. Every lab reports usage differently — some return prompt and completion tokens, some separate cached from uncached input, some omit usage entirely on streamed responses — so the gateway normalises each response into one internal usage record and applies one rate card. That normalisation is the entire reason a single balance is possible across 65 model IDs.

Key takeaway: if you cannot reconstruct the cost of a single request from your own logs, you do not have a billing system — you have a bill from someone else.

---

The routing path, end to end

Put the layers together and a single request looks like this:

# Pseudocode for the gateway's request path — the layers above, in order.
def handle_request(request):
    key = authenticate(request)                    # who is calling
    model = resolve_model_id(request.model)        # display name -> real ID
    check_key_limits(key, request)                 # rate, concurrency, budget

    candidates = registry.channels_for(model)      # Layer 1
    channel = select(candidates, policy=request.policy)  # Layer 3

    reserve_tokens(key, request.max_tokens)        # Layer 5
    with channel.slot():                           # per-channel concurrency
        try:
            resp = channel.forward(request)
            if resp.is_empty():                    # the silent-failure case
                raise EmptyCompletion()
        except RetryableError:
            channel.mark_degraded()                # Layer 2
            channel = select(registry.healthy_for(model), policy=request.policy)
            resp = channel.forward(request)        # last-resort failover

    usage = normalise_usage(resp)                  # Layer 6
    reconcile_tokens(key, usage)
    record(key, model, channel, resp, usage)       # observability
    return resp

Read the order carefully. The two steps that most often run in the wrong place are reserve_tokens — which must happen before the upstream call, not after — and channel.slot(), which must be held for the duration of the stream, not just the connection setup.

---

What this buys the caller

Every layer above exists to support one claim: that a caller can treat 60-plus models as one service.

Concern	Direct integration, per lab	Through a unified gateway
Credentials	One key per provider, each with its own rotation	One key, one rotation
SDKs and conventions	Auth headers, streaming formats and tool-call schemas differ	One OpenAI-compatible client
Failover	Built and maintained by you, per provider	Handled at routing time
Cost visibility	One dashboard and one invoice per provider	One usage record, one balance
Signup friction	Several Chinese labs require a Chinese phone number and local payment	Email, Google or GitHub, and international cards in USD
Switching model	A new integration project	A model= change

The trade-off is real and worth stating: a direct connection puts you closest to the provider and gets day-zero features first. A gateway buys coverage, failover and a single contract, and charges a routing markup on most models to do it. Judge it on total cost of ownership rather than on the rate card alone.

---

Quick start

from openai import OpenAI

client = OpenAI(
    api_key="your-tokenpapa-key",
    base_url="https://tokenpapa.ai/v1",
)

def ask(prompt: str, model: str = "deepseek-v4-flash") -> str:
    resp = client.chat.completions.create(
        model=model,                 # any live ID: gpt-5.6-luna, claude-sonnet-4-6, qwen3.7-plus, kimi-k3
        messages=[{"role": "user", "content": prompt}],
        max_tokens=600,              # cap output: output tokens cost 3x to 10x input
    )
    return resp.choices[0].message.content

print(ask("Summarise this changelog in five bullets."))

Two habits make a fleet reliable rather than merely connected:

1. Cap output tokens on every call. Output is priced at 3x to 10x input across the models above, and an uncapped generation is also an uncapped concurrency hold.
2. Keep the prompt prefix byte-stable. It is what unlocks the roughly 90% repeat-input saving from automatic context caching.

---

FAQ

Q: How do you route API calls across dozens of models without adding latency?

A: Routing has to happen in parallel with request setup, not before it. The gateway resolves a model ID to a channel from an in-memory registry, picks a healthy upstream and starts opening the socket while the request body is still being read. On the healthy path the cost is a lookup and a connection choice, not an extra round trip to a separate routing service. Extra hops appear only on the failover path, where a failed attempt is retried on a second channel.

Q: What happens when an upstream provider goes down mid-request?

A: The gateway treats a failure before the first token differently from a failure after streaming has started. If nothing has been returned to the client yet, the request can be retried on a second healthy channel for the same model, so the caller sees a slower success instead of an error. If tokens are already streaming, the stream is closed cleanly and the failure is reported, because silently switching upstreams mid-answer would produce a corrupted response.

Q: How do you keep 60 models from turning into 60 invoices?

A: Billing is normalized at the gateway. Every upstream reports usage differently — some in prompt and completion tokens, some separating cached from uncached input — so each response is mapped to one internal usage record with a model, an input token count, an output token count and a cache-hit count. A single rate card is then applied, which means the caller sees one balance and one invoice regardless of which upstream served the request.

Q: Why would a unified gateway cost more than calling a provider directly?

A: On most models a gateway does add a routing markup, because it pays upstream rates and carries infrastructure, failover capacity and support costs. Some models price below first-party list — Kimi K3 runs roughly 10% under official, for example. The honest comparison is total cost of ownership: one integration, one credential and one bill, versus maintaining a separate SDK, auth flow, retry policy and invoice for every lab you integrate. The published rate card is the authority, not an article.

---

Get Started

1. Sign up at tokenpapa.ai/register with email, Google or GitHub — no Chinese phone number required.
2. Create an API key in the console and top up from $10 with an international card. Billing is pay-as-you-go in USD.
3. Point any OpenAI-compatible client at the endpoint and start on the budget tier:

from openai import OpenAI

client = OpenAI(api_key="your-key", base_url="https://tokenpapa.ai/v1")

response = client.chat.completions.create(
    model="deepseek-v4-flash",   # or deepseek-v4-pro, gpt-5.6-luna, claude-sonnet-4-6
    messages=[{"role": "user", "content": "Hello!"}],
    max_tokens=400,              # always cap output tokens
)

print(response.choices[0].message.content)

Full rate card: tokenpapa.ai/pricing. Current model list: GET https://tokenpapa.ai/v1/models.

---

This article describes routing architecture and design principles, not internal capacity or customer metrics. Model prices are TokenPAPA platform rates as of September 2026 and are subject to change; verify current rates on the pricing page before committing to a budget.

---

Originally published at https://doc.tokenpapa.ai/en/docs/blog/routing-1m-api-calls-daily-60-models.