DEV Community: chenxiao5580-cmd

Stop hand-picking an LLM per request: a practical case for auto-routing

chenxiao5580-cmd — Tue, 16 Jun 2026 16:24:38 +0000

Most LLM features ship with the model name hardcoded. You picked it once — usually the strongest one you could justify — and now every request, trivial or gnarly, hits the same expensive model. The easy ones overpay; if you down-picked to save money, the hard ones quietly degrade. You're paying the frontier price for "reformat this list," or shipping a weak answer on "find the bug in this trace."

Routing per request fixes the mismatch: classify each request's difficulty, then send it to the cheapest model in your quality tier that can actually handle it.

What "difficulty" can mean in practice

You don't need a research model to route well. Cheap, legible signals get you most of the way:

Input shape and length. A 30-token reformat is not a 6,000-token "reason over this codebase" request.
Task type keywords / structure. Extraction and classification skew easy; multi-step reasoning, code debugging, and math skew hard.
A tiny classifier up front. A small, fast model can score difficulty in a few milliseconds and a fraction of a cent, then hand off to the right tier.

The router doesn't have to be perfect. It has to be better than a single hardcoded choice — which is a low bar, because a hardcoded choice is wrong for half your distribution by construction.

Where routing misfires (and why you must plan for it)

Anyone who's run this in production will tell you the failure modes are the interesting part:

Deceptively short hard prompts. "Prove this is NP-complete" is 5 tokens and very hard. Length-only routing sends it to a weak model and you get a confident-wrong answer.
Misclassification is silent. A bad route doesn't error — it just returns a worse answer. Without eval, you won't see it.
Tier boundaries are fuzzy. Requests near the easy/hard line will flip routes run-to-run, making behavior feel non-deterministic.

So routing is not "set and forget." It needs guardrails.

Keeping it safe

Route within a quality floor, never below it. Let the user pick a tier; the router chooses within it, so the worst case is still acceptable. Never silently drop to a model the user wouldn't accept.
Bias toward the stronger model on uncertainty. When the difficulty score is ambiguous, round up. A few cents of overspend beats a wrong answer.
Make routing decisions observable. Log which model served each request and why. You can't debug or tune a router you can't see.
Eval the routes, not just the models. Periodically check that easy-routed requests would've gotten the same answer from the strong model, and that hard-routed ones aren't degrading.

What it buys you

When it's tuned, you stop overpaying on the (usually majority) easy traffic without down-grading the hard tail — and you stop hand-maintaining a model choice that drifts out of date every time providers ship something new. The routing logic lives in one place instead of smeared across feature code.

Takeaway

Hardcoding one model per feature optimizes for nothing — it's a coin flip that's wrong for half your request distribution. Difficulty-based routing within a quality tier, with an "round up when unsure" bias and real observability, is a better default. I've built this into a small OpenAI-compatible gateway called Modelis (send model: "auto", it routes within your tier and bills a flat per-call price, free tier) at modelishub.com — but you can build the same idea yourself with a small front classifier. I'd love to hear the nastiest "short prompt, secretly hard" example you've hit — those are the routing killers.

One base_url for GPT, Claude, and Gemini: cutting three SDKs down to one

chenxiao5580-cmd — Tue, 16 Jun 2026 16:16:17 +0000

The first time you add a second LLM provider to a codebase, it feels manageable. By the third, you've got three SDKs, three auth schemes, three slightly different messages shapes, three retry policies, and three places a model deprecation can break you. The "use the best model for each job" advice is correct — but the integration tax is real, and it compounds.

Here's the pattern I keep coming back to: put everything behind one OpenAI-compatible endpoint and stop maintaining three clients.

The actual cost of multi-SDK glue

It's not the happy path that hurts — it's everything around it:

Auth drift. OpenAI wants a Bearer key, Anthropic wants x-api-key + a version header, Google wants its own scheme. Three secrets, three rotation schedules.
Response-shape divergence. choices[0].message.content vs. a content block array vs. candidates[].content.parts[]. Every call site that reads a response needs a per-provider branch.
Streaming differences. SSE framing and event names differ enough that your stream parser grows a switch statement.
Failure-mode sprawl. Rate-limit codes, timeout behavior, and error bodies all differ. Your retry/backoff logic forks per provider.

None of it is hard. All of it is maintenance — the kind that quietly slows a small team down.

The OpenAI-compatible shim

Most providers can be normalized to the OpenAI Chat Completions contract, because the ecosystem already standardized on it. So you point the official OpenAI SDK at a different base_url:

from openai import OpenAI

client = OpenAI(
    base_url="https://your-gateway/v1",
    api_key="YOUR_GATEWAY_KEY",
)

resp = client.chat.completions.create(
    model="gpt-4o-mini",          # or claude-*, gemini-*, etc.
    messages=[{"role": "user", "content": "Summarize this PR in two lines."}],
)
print(resp.choices[0].message.content)

Same call site, same response shape, one key. To switch the underlying model you change one string. Your retry logic, your streaming parser, your logging — all written once, against one contract.

What you actually gain

One auth + one secret to rotate, not three.
One response shape at every call site — no per-provider branching.
No lock-in. Because you code against the OpenAI contract (an open de-facto standard), moving off any single provider — or off the gateway itself — is a base_url change, not a rewrite. That cuts both ways, and that's the point.

What you give up (the honest part)

A normalization layer can't be a free lunch:

Provider-specific features get flattened. Anthropic's prompt caching, Google's grounding, OpenAI's structured-output modes — anything that doesn't map cleanly to the common contract is either unavailable or exposed through non-standard fields you'll have to special-case anyway.
You add a hop. One more network segment and one more thing that can be down. For latency-sensitive paths, measure it.
You inherit the shim's coverage gaps. If the gateway hasn't mapped a parameter you need, you're blocked until it does. Pick one whose mapping is transparent and documented.

The trade is: lose access to the long tail of provider-specific knobs, gain a dramatically smaller integration surface. For most app teams — who use maybe 5% of any provider's surface — that's a good deal. For teams leaning hard on one provider's exclusive features, it isn't.

Takeaway

If your codebase has grown a per-provider branch at every LLM call site, collapsing to one OpenAI-compatible base_url removes a whole category of maintenance — at the cost of the provider-specific long tail. I build this into a small gateway called Modelis (one key for GPT/Claude/Gemini, optional auto routing, free tier) at modelishub.com, but the shim pattern works with anything OpenAI-compatible. Curious which provider-specific feature you'd refuse to give up — that's usually the deciding factor.

Stop getting surprise per-token LLM bills: a flat-rate, auto-routing API approach

chenxiao5580-cmd — Tue, 16 Jun 2026 16:13:36 +0000

If you ship anything on top of an LLM API, you've probably had this moment: you check the dashboard at the end of the month and the bill is 3x what you modeled. Nothing broke. Usage just... drifted. A few prompts got chattier, one model started "thinking" more, and your per-token math quietly fell apart.

I've been living in that loop, so I want to lay out why per-token pricing is hard to forecast, and a different billing shape that trades some theoretical savings for a number you can actually predict.

Why per-token spend is so hard to model

Per-token billing looks simple — price × tokens — but three things make it slippery in practice:

1. Output length is not yours to control. max_tokens is a hard ceiling, but the model decides how much of that ceiling it actually uses. Two models given the identical prompt can produce wildly different output lengths, and the verbose one costs you more for the same task.

2. "Reasoning" tokens are invisible until billed. Some newer models (e.g. OpenAI's o-series) emit internal reasoning tokens that you're charged for but don't see in the default response. Your logs show a 200-token answer; the invoice counts the 1,500 tokens it took to get there.

3. Input grows silently. RAG context, longer chat histories, system prompts that accrete over time — input token counts creep up release by release, and without active monitoring nobody notices until the bill does.

Individually these are fine. Together they mean your cost-per-request has a fat tail, and a fat tail is exactly what you can't put in a pricing page or a unit-economics spreadsheet.

The alternative: bill a flat price per call

The idea is simple: instead of charging for whatever tokens happened, charge a flat rate per request within a quality tier you pick. The bill becomes calls × tier_price, full stop. Output length, reasoning tokens, which specific model answered — none of it changes what you pay.

You give up something real here (more on that below), but you gain the one thing per-token can't offer: a cost you can forecast before you ship. For a lot of products — fixed-price SaaS features, internal tools with a budget, anything where you quote a customer a number — that predictability is worth more than shaving a few percent off the theoretical optimum.

Concretely: say a request averages 1,000 tokens and a cheap model bills $0.0005/1K — that's ~$0.0005/call. A flat tier at, say, $0.002/call costs more on that uniform workload. But the moment your requests vary — some 200 tokens, some 8,000, some routed to a frontier model — the per-token average climbs and its variance explodes, while the flat price stays put. Flat pricing wins on the spread, not the average.

Pairing it with auto-routing

Flat-per-call pricing gets more interesting when you stop hand-picking the model. If every request costs the same regardless of which model serves it, you can let a router classify each request's difficulty and send it to the cheapest model that can actually handle it — a small model for "summarize this", a frontier model for "debug this race condition" — without you wiring up that logic or eating a surprise when a hard request lands on an expensive model.

The developer-facing contract stays boring on purpose:

curl https://your-gateway/v1/chat/completions \
  -H "content-type: application/json" \
  -H "authorization: Bearer $KEY" \
  -d '{
    "model": "auto",
    "messages": [{"role":"user","content":"Explain quantum entanglement in one sentence."}]
  }'

It's an OpenAI-compatible /chat/completions body, so existing SDKs work by changing base_url and the key. You send one model name (auto) and let routing + a flat tier price do the rest. Zero migration is part of the pitch — if it needs a rewrite, nobody switches.

When flat pricing does NOT pay off (the honest part)

This isn't a free lunch, and pretending otherwise would be dishonest:

High-volume, short, predictable calls. If your requests are tiny and uniform (think: classification of one-line inputs at scale), per-token on a cheap model will almost certainly beat a flat per-call rate. Flat pricing's value is in variance reduction, and you have no variance.
You've already done the FinOps work. If you have tight token budgets, prompt-length guards, and good observability, you've manufactured your own predictability and a flat tier buys you less.
You need features flat tiers cap. Vision inputs, tool/function-calling, huge outputs — flat per-call tiers often bound these (depending on the gateway) to keep the price honest. If you need them, a metered model fits better.
You have strict latency SLAs. Cost-only routing can pick a cheap-but-slow model. If you're under a hard latency budget, the router needs a latency filter too — extra complexity that eats some of the simplicity win.

The rule of thumb: flat pricing is insurance against variance. The more your per-request cost jumps around — mixed task difficulty, verbose or reasoning-heavy models, growing context — the more that insurance is worth. The flatter your workload already is, the less you need it.

Takeaway

Per-token billing isn't wrong, but it optimizes for a metric (tokens) that isn't the one you're trying to control (a predictable bill). If you're quoting fixed prices to customers, or you just want your LLM line item to stop surprising you, a flat per-call rate — ideally with auto-routing underneath — is worth a look.

I've been building this approach into a small OpenAI-compatible gateway called Modelis (one key for GPT/Claude/Gemini, auto routing, flat per-plan pricing, free tier to try). If you want to kick the tires, it's at modelishub.com. But the billing argument stands on its own — I'd genuinely like to hear where flat-vs-metered breaks for your workload in the comments.