DEV Community: BoostRail

BYOK: Using Your Own Provider Keys Behind One LLM API

Lee Gao — Sat, 17 Oct 2026 08:34:11 +0000

Originally published on boostrail.com on August 5, 2026.

Teams adopting a multi-model API usually hit the same procurement question: "We already have contracts and negotiated prices with our model providers — why would we re-buy inference through a middleman?"

Bring Your Own Key (BYOK) is the answer to exactly that. You keep your provider accounts; one API in front of them keeps the integration simple.

How it works

With BYOK on BoostRail, an organization stores its own provider API keys with the platform. When a request targets a model covered by one of those keys, BoostRail serves it with your key against the lab's official endpoint. That means:

Inference charges land on your own provider account, at your contract prices — not on platform credit.
Request data stays within your existing provider agreements, which matters where procurement or compliance requires a direct provider relationship.
Your application never touches the stored keys — it keeps using one BoostRail API key, and the provider keys stay on the platform side.

What BoostRail still handles: the single OpenAI-compatible endpoint, one key for your application, the unified catalog, and platform fallback. If your own key fails, runs out of provider credit or is rate-limited, the request can continue on platform routes of the same model instead of failing; that needs a balance. Where data boundaries are strict, platform routes can be switched off per lab, and requests then fail rather than fall back.

What it costs

The free plan includes 1,000,000 BYOK requests per month. Paid plans raise the monthly quota and the number of labs you can connect, for a flat monthly fee. Inference itself is billed by your provider, so BYOK is counted in requests rather than as a percentage of what the usage would cost. When the monthly quota runs out, requests switch to platform routes at the published platform prices until the quota resets on the 1st of the month (UTC); models that run only on your own key pause until then.

Zero lock-in, by construction

Lock-in is the standard objection to gateways, and BYOK inverts it: the provider keys are yours, and your application talks to an OpenAI-compatible endpoint. Leaving is pointing base_url back at a provider's official endpoint. There is no proprietary SDK and no custom protocol.

When BYOK covers a model — and when it doesn't

BYOK needs two things: the lab is on the supported list for your plan, and the model can be served from that lab's official endpoint. Models from labs without a public official API run on platform routes only. The current lab list and per-plan quotas are on the BYOK page.

Typical setups

Contract-price teams: inference on negotiated provider prices, BoostRail for the single API and usage records, platform fallback for spillover.
Compliance-bound teams: requests to sensitive models pinned to their own provider agreement with platform routes switched off for that lab, everything else on platform routes.
Cost-split teams: frontier models via their own keys (where contracts are), long-tail models on platform credit — one BoostRail bill alongside unchanged provider bills.

BYOK is set up in the console at app.boostrail.com; plans and the supported lab list are on the BYOK page.

The Hidden Complexity of Multi-API AI Integrations

Lee Gao — Sat, 17 Oct 2026 08:33:34 +0000

Originally published on boostrail.com on August 5, 2026.

The first LLM integration looks deceptively simple: one HTTP POST to a /v1/chat/completions endpoint and you have working intelligence in under an hour. The trouble starts when the application matures. To survive provider outages, control inference cost, or use each model where it is strongest, teams inevitably go multi-provider — and what was one API call becomes a maze of incompatible schemas, divergent auth models, backoff logic, fragile streaming parsers and fragmented billing.

These are the four failure modes that show up in practice, and the architectural pattern that removes them.

1. Schema drift and payload normalization

Providers do not agree on request or response formats. Many advertise "OpenAI compatibility", but the payload variances are real:

OpenAI-style APIs put system instructions inside the messages array as "role": "system"; temperature ranges 0.0–2.0.
Anthropic's native API pulls the system prompt out into a top-level system parameter, accepts only user/assistant roles, caps temperature at 1.0, and requires max_tokens.
Google's native Gemini API uses contents instead of messages, "role": "model" instead of "role": "assistant", and nests text in a parts array.

Swap a payload from one to another without a serialization layer and the destination rejects it with a 400.

// OpenAI-style request
{
  "model": "gpt-5.5",
  "messages": [
    {"role": "system", "content": "You are a precise JSON extractor."},
    {"role": "user", "content": "Extract data from this text..."}
  ]
}

// Anthropic native request — top-level system prompt, max_tokens required
{
  "model": "claude-sonnet-5",
  "system": "You are a precise JSON extractor.",
  "messages": [
    {"role": "user", "content": "Extract data from this text..."}
  ],
  "max_tokens": 1024
}

Tool calling cuts deeper. Providers enforce JSON Schema with different strictness: one rejects a definition over a missing optional field, another silently drops constraints it does not support — and malformed arguments flow into your backend. Without a normalization layer you end up maintaining separate serializers, parsers and schema validators for every vendor in the stack.

2. Rate limits, retries and transient outages

Classic microservices fail predictably: 500 means server error, 503 means try later. LLM APIs throttle on two independent dimensions — requests per minute and tokens per minute — so an endpoint can return 429 during a traffic spike even when your monthly spend has plenty of headroom.

A naive fixed-interval retry loop makes a 429 worse: every retry counts against the same window, and the violation compounds into an extended lockout. The standard remedy is exponential backoff with full jitter:

import time
import random

def calculate_backoff(attempt: int, base_delay: float = 1.0, max_delay: float = 32.0) -> float:
    """Exponential backoff with full randomized jitter."""
    calculated_delay = min(max_delay, base_delay * (2 ** attempt))
    # Full jitter prevents a thundering herd of synchronized retries
    return random.uniform(0, calculated_delay)

Backoff alone does not cover extended outages — that needs failover to another provider, and model failover is not transparent:

Context window mismatch. Fail over from a 200K-token model to a 32K-token model without validating prompt size first and you trade a 429 for a truncation error.
Prompt sensitivity. A prompt tuned for one model family can produce unstructured output on another without adaptation.
Mid-stream disconnects. SSE streams drop mid-response; unlike a plain HTTP error, the client is left holding a partial completion and must re-stitch or continue the prompt.

Production setups pair health checks with circuit breakers: after consecutive failures the breaker opens, traffic routes to a backup line, and the primary is re-tested after a cooldown.

3. Key sprawl and billing fragmentation

Beyond code mechanics, direct multi-provider integration is an operational tax. Separate SDKs mean separate secrets in every environment; rotating a key or scoping team access is a chore repeated across as many consoles as you have vendors. Each provider bills on its own cycle and prepayment model, so finance reconciles four or five dashboards a month while engineering has no single view of token spend per feature. And one unhandled retry loop against a frontier-priced endpoint can burn serious money in minutes if no centralized budget guardrail exists.

4. The architecture that removes the problem

Three patterns cover the design space:

Direct SDK integration — fastest to start, but provider conditionals leak into business logic, failover is hardcoded, and every model swap touches application code.
In-house abstraction layer — decouples the app, but you now own a normalization-and-failover codebase that must track every provider API change, forever.
Unified gateway — one OpenAI-compatible endpoint in front of all providers; normalization, failover and billing consolidation happen below your code.

For most teams the gateway is the pattern that scales. Pointing your HTTP client at BoostRail gives you one OpenAI-compatible API for {{MODELS}} text, image and video models from {{LABS}} labs. Request formats are normalized, and when a route for a model fails, the request moves to another route of the same model before any data reaches you — it never switches to a different model, so the context-window and prompt mismatches above don't arise. Once data has started streaming, the request stays on its route. With BYOK, inference can bill to your own provider contracts, with platform routes of the same model as fallback.

import openai

# Switching models is a one-string change.
client = openai.OpenAI(
    base_url="https://api.boostrail.com/v1",
    api_key="YOUR_BOOSTRAIL_API_KEY"
)

response = client.chat.completions.create(
    model="claude-sonnet-5",  # or gpt-5.5, gemini-3.1-pro, deepseek-v4-pro...
    messages=[
        {"role": "user", "content": "Explain circuit breaker patterns in microservices."}
    ]
)

print(response.choices[0].message.content)

A gateway adds a network hop. For most workloads that is small next to generation time, which runs to hundreds or thousands of milliseconds; measure it on your own traffic before you cut over.

A reliability checklist

Before scaling a production AI application across providers, verify:

Schema abstraction — provider payload formats isolated behind one OpenAI-compatible layer.
Adaptive backoff — exponential backoff with full jitter on 429s, never fixed-interval retries.
Automated failover — health checks and circuit breakers that account for context-window and prompt differences.
Stream recovery — SSE parsers that survive abrupt disconnects without unhandled exceptions.
Unified telemetry and billing — token spend, budgets and API health visible in one place.

The full model list is in the BoostRail catalog; an API key takes a minute at app.boostrail.com.

Migrating to an OpenAI-Compatible API: a Practical Guide

Lee Gao — Sat, 17 Oct 2026 08:33:28 +0000

Originally published on boostrail.com on August 5, 2026.

"OpenAI-compatible" is the most load-bearing phrase in AI infrastructure, and it means something concrete: a service that accepts the same requests and returns the same responses as OpenAI's API — same endpoints, same JSON shapes, same streaming format. Your existing OpenAI SDK code works against it after changing exactly one thing: the base URL.

That's why the OpenAI format has become the industry's common protocol, the way S3's API became the standard for object storage. This guide walks through what a migration actually involves — which is less than you'd think — and the few places where checking first saves a surprise later.

The one-line migration

Both official OpenAI SDKs accept a base_url (Python) / baseURL (JavaScript) parameter. Point it at a compatible endpoint, swap the key, and everything else stays:

# Python — before and after
from openai import OpenAI

client = OpenAI(
    api_key="YOUR_BOOSTRAIL_KEY",
    base_url="https://api.boostrail.com/v1"   # the only new line
)

response = client.chat.completions.create(
    model="claude-sonnet-5",   # any model in the catalog
    messages=[{"role": "user", "content": "Hello"}]
)

// JavaScript / TypeScript
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: "YOUR_BOOSTRAIL_KEY",
  baseURL: "https://api.boostrail.com/v1",
});

const response = await client.chat.completions.create({
  model: "claude-sonnet-5",
  messages: [{ role: "user", content: "Hello" }],
});

No new SDK, no new dependency, no request-shape changes. The same works for any framework that exposes a base-URL setting — LangChain, LlamaIndex, Vercel AI SDK and most others do.

What carries over unchanged

The point of compatibility is that the boring majority of your integration just works:

Chat completions — client.chat.completions.create with the full messages array semantics
Streaming — server-sent events arrive in the same chunk format, so existing stream parsers keep working
Tool calling — tools, tool_calls, content parts and streaming tool-call deltas work in the same format
Model listing — /v1/models returns the catalog, so model pickers keep working

One endpoint difference worth knowing about: an API can also expose other clients' native protocols alongside the OpenAI surface. On BoostRail, Anthropic-protocol clients like Claude Code talk to /v1/messages and Codex CLI talks to /v1/responses — each client uses the protocol it was built for, with no local adapter. Details are on the OpenAI-compatible API page.

What to check before you flip the switch

Compatibility covers the protocol; a few behaviors legitimately differ per model and are worth a look:

Model names. gpt-4o-era strings won't name the same model everywhere. Check the catalog and pick explicitly — switching later is a one-string change.
Parameter ranges. Temperature runs 0–2 in the OpenAI convention but 0–1 on Anthropic's API. BoostRail passes the value on as you send it rather than rescaling it, so if your product exposes a temperature slider, check its range against the models you call.
Token limits. Different models have different context windows and default output caps. If you hard-coded max_tokens for one model, confirm it fits the new one.
Prompt behavior. A prompt tuned hard against one model may need a round of adjustment on another. Migrate the plumbing first, then A/B the prompts — don't change both at once.

Migrating coding agents

Coding agents migrate the same way, usually via environment variables rather than code:

// Claude Code — ~/.claude/settings.json
{
  "env": {
    "ANTHROPIC_BASE_URL": "https://api.boostrail.com",
    "ANTHROPIC_API_KEY": "YOUR_API_KEY",
    "ANTHROPIC_MODEL": "claude-sonnet-5",
    "CLAUDE_CODE_ATTRIBUTION_HEADER": "0",
    "CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS": "1",
    "CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY": "1"
  }
}

Cline, OpenCode, Continue and Aider take a standard OpenAI-compatible provider block. Per-tool walkthroughs live on the coding tools page, and the rate-limit reasons agents benefit most from this migration are covered in Coding Agents and Rate Walls.

Rollback is also one line

The migration is symmetric: point base_url back at the provider's official endpoint and you've left. No proprietary SDK and no export process.

FAQ

Will responses be byte-identical to OpenAI's?
The JSON structure is identical; the content depends on which model you call. Field-level extras (like provider-specific metadata) may appear but won't break standard parsers.

Do I need to migrate everything at once?
No. Because both sides speak the same protocol, you can run a percentage of traffic through the new endpoint, compare, and cut over gradually.

What about my fine-tuned or provider-specific models?
Models that exist only in your own provider account, such as fine-tunes, are not in the catalog, so BoostRail can't serve them — keep calling those on the provider's own endpoint. If you have contract prices for catalog models, BYOK runs them on your own provider keys; see the BYOK guide.

What Is an LLM Gateway? A Plain-English Guide

Lee Gao — Sat, 17 Oct 2026 08:32:51 +0000

Originally published on boostrail.com on August 5, 2026.

An LLM gateway is a single entry point that sits between your application and every AI model provider you use. Your code talks to one API; the gateway talks to OpenAI, Anthropic, Google, DeepSeek and the rest, translating formats, routing traffic and consolidating billing along the way.

That's the whole idea. The rest of this guide explains why that one layer has become standard kit for production AI applications, what varieties exist, and how to decide whether you need one at all.

The problems it solves

Teams rarely adopt a gateway on day one — they adopt it after hitting some combination of four walls. We've written up the engineering detail in The Hidden Complexity of Multi-API AI Integrations; the short version:

Every provider speaks a different dialect. All the major APIs handle message formats, system prompts and parameters slightly differently. Code written for one breaks against another. A gateway normalizes everything behind one format — in practice, the OpenAI format, which has become the de-facto standard the way S3's API did for storage.
Rate limits and outages don't announce themselves. Providers throttle by requests and tokens per minute, and even top-tier models have bad days. A gateway tracks the health of each route and moves failed requests automatically, so fewer errors reach your users.
Keys and bills multiply. Four providers means four secrets to rotate, four dashboards to reconcile, four places spending can leak. A gateway collapses that to one key and one bill.
Switching models means touching code. Without a gateway, trying a new model is an integration project. With one, it's changing a model name in a string.

The three kinds of gateway

The market has sorted itself into three shapes:

Self-hosted open source. You run the router on your own infrastructure — maximum control and data residency, in exchange for owning deployment, upgrades and provider-API chasing forever. LiteLLM is the best-known example.
Hosted multi-model gateways. A managed service exposes one OpenAI-compatible endpoint backed by a large catalog; routing, failover and billing consolidation happen behind it. BoostRail is in this camp: {{MODELS}} text, image and video models from {{LABS}} labs behind one key, with every per-token price published.
Observability- and governance-first platforms. Products whose center of gravity is logging, cost analytics, access policies or guardrails, with routing attached. Helicone and Portkey lead here.

These categories overlap at the edges — most products do some of everything — but knowing a product's center of gravity is the fastest way to predict where it will serve you well.

How to choose: seven questions

Am I using, or likely to use, more than one model? If genuinely no, a gateway is optional.
What happens to my product when my provider has an outage? If the answer is "it goes down," failover is your first requirement.
Must request data stay inside my infrastructure? Yes → self-hosted. No → hosted saves you the operations.
How does the gateway charge? Compare per-token prices against the official list, top-up and payment fees, and any platform or subscription fee. At scale the differences add up.
Can I bring my own provider contracts? If you have negotiated prices, BYOK support — using your own provider keys through the gateway — protects them. Check the free allowance; on BoostRail's free plan it's 1M BYOK requests a month.
What modalities do I need? Text is table stakes; image input is common; image and video generation vary more between platforms — check the catalog covers what your roadmap needs.
How hard is leaving? With an OpenAI-compatible gateway, leaving is changing one URL back. A proprietary SDK requirement makes it harder.

FAQ

Does a gateway slow my requests down?
It adds a network hop. For most workloads that is small next to generation time, which runs to hundreds or thousands of milliseconds; measure it on your own traffic.

Is an LLM gateway the same as an API gateway?
Same concept, specialized differently. Classic API gateways (Kong, Cloudflare) manage generic HTTP traffic; LLM gateways add model-specific work — payload translation between provider dialects, token-aware rate limit handling, model routing and failover, per-token cost tracking.

Do I need one for a prototype?
No — one provider, one SDK is the right way to start. The gateway earns its place when you add a second model, hit your first rate limit, or need the bill explained. Since adopting one is a base_url change, deferring the decision costs nothing.

LLM API Fallback: Keep Your App Up When a Model Fails

Lee Gao — Sat, 17 Oct 2026 08:32:45 +0000

Originally published on boostrail.com on October 16, 2026.

An LLM API fallback plan decides what your application does when a model call fails: wait and retry, send the same request somewhere else, or ask a different model. Without one, a provider's bad hour becomes your outage. With a careless one, a retry loop can double your bill or send a user two half-answers. This guide covers the failures that happen in practice, the patterns that handle them, a short Python implementation, and which parts a multi-route API already takes off your hands.

What actually fails in production

LLM calls fail in a handful of distinct ways, and each needs a different response.

Rate limits (429). Too many requests or tokens per minute for an account, a key or a model. Usually temporary, and the response often says how long to wait.
Server errors (500, 502, 503). The model's service is overloaded or broken. Often temporary, sometimes for hours.
Timeouts (504 or a client timeout). No answer within the time allowed. A timeout is not always an outage: when we sent a long coding prompt to two reasoning models on 16 October 2026 without streaming, neither had finished within the 180 seconds limit for non-streaming requests, and both returned provider_timeout.
Broken streams. The connection drops after part of the answer has arrived.
Request errors (400). The request itself is wrong: an unknown model, a parameter the model does not accept, input longer than its context window. Retrying the same request gives the same error.
Account errors (401, 402, 403). A bad key, an empty balance, a disabled key. Retrying makes no difference until someone fixes the account.

Retry, failover and fallback are different things

The three words get used loosely. Keeping them apart makes the design easier.

Retry sends the same request to the same place again after a pause. It fixes brief overloads and rate limits.
Failover sends the same request for the same model to another place that serves it: another region, deployment or provider. The answer should be equivalent, because the model is the same.
Fallback sends the request to a different model. It keeps the application answering when a model is unavailable everywhere, at the price of a different answer: other strengths, other formatting, sometimes other tool-calling habits.

Retry and failover are safe to automate. Fallback to another model is a product decision, because users may notice the change.

LLM API fallback patterns that work

1. Classify the error before acting. A small table in code prevents the two classic mistakes: retrying requests that can never succeed, and giving up on errors that would clear in seconds.

Error	Retry	Fail over	Fall back to another model
429 rate limit	Yes, after the wait it asks for	Yes	If it persists
500, 502, 503	Yes, with backoff	Yes	If it persists
504 or client timeout	Stream instead, or raise the limit	Yes	If it persists
400 bad request	No; fix the request	No	Only if another model accepts the request
401, 402, 403	No; fix the account	No	No

2. Back off exponentially, with jitter, and respect retry-after. Doubling the wait between attempts and adding a random offset stops many clients from retrying in lockstep. When the response carries retry-after, wait at least that long.

3. Stream long generations and set timeouts in layers. A connect timeout catches a dead network. A first-token timeout catches a stuck request. A total timeout stops a runaway answer. For prompts that make a model think for minutes, streaming keeps the connection alive and lets the client see progress, where a non-streaming request would hit a time limit first.

4. Keep the fallback chain short and tested. Two or three models, chosen in advance, each tested with your real prompts. A chain of models nobody has tried with your prompts is a guess, not a plan.

5. Stop sending traffic to a target that keeps failing. After repeated failures, pause requests to that model or endpoint for a short time instead of letting every request discover the same outage. This is the circuit breaker pattern, and it saves both latency and money during an incident.

6. Do not silently retry a stream the user is already reading. If text has reached the user and the stream breaks, a fresh request produces a different answer, not a continuation. Show what arrived, mark it as incomplete, and offer a retry.

7. Log the request id. Every failed call should record the model, the error code and the request id. That turns "the AI was down this morning" into something support can trace.

A minimal implementation in Python

The OpenAI Python SDK already retries connection errors, 408, 409, 429 and 5xx responses with exponential backoff, twice by default, as its README describes. The code below adds the part it does not do: moving to the next model in a short chain when one stays unavailable, while letting request and account errors surface immediately.

import openai
from openai import OpenAI

client = OpenAI(
    api_key="YOUR_API_KEY",
    base_url="https://api.boostrail.com/v1",
    max_retries=2,
    timeout=120,
)

FALLBACK_CHAIN = ["claude-sonnet-5", "kimi-k3", "deepseek-v4.1-flash"]
RETRYABLE = (
    openai.RateLimitError,
    openai.APITimeoutError,
    openai.APIConnectionError,
    openai.InternalServerError,
)


def complete(messages):
    last_error = None
    for model in FALLBACK_CHAIN:
        try:
            return client.chat.completions.create(model=model, messages=messages)
        except RETRYABLE as error:
            last_error = error
    raise last_error


reply = complete([{"role": "user", "content": "Summarise this incident report in three bullet points."}])
print(reply.model, reply.choices[0].message.content)

Because every model sits behind the same endpoint and key, the chain is a list of strings. reply.model tells you which model answered, which is worth logging and, in some products, showing.

What a multi-route API handles for you

On BoostRail, failover for the same model happens before the response reaches you. The rules are published on Errors and failover:

When the route serving a request fails, BoostRail tries the same model on another route before anything is sent to you. It never switches to a different model.
Server errors, rate limiting and the route refusing BoostRail's own access are moved to another route. Request errors and non-streaming timeouts are not, because another route would fail the same way.
For streaming requests, if no first event arrives within 45 seconds, the request moves to the next route.
Once any data has been sent to you, the request is not moved.
When every route fails, you get the error from the last attempt, and the request is not charged.

That changes what your own code needs to do. Failover for the same model is covered, so your retries mostly handle what is left: 429 with retry-after, and 502 or 504 after BoostRail has already tried its routes. Fallback to a different model stays your decision, because BoostRail does not make it for you. The price is the model's published price on the Models page, whichever route serves the request.

For the broader integration problems that sit around fallback, such as schema differences and key sprawl, see The Hidden Complexity of Multi-API AI Integrations.

Testing the fallback path

A fallback path that has never run will fail the first time it matters. Three cheap ways to exercise it:

Force a timeout. Give the first model in the chain a client timeout far shorter than any real answer, and check that the second model answers and the log records why.
Break the network. Point the client at an address that refuses connections and confirm the error is classified as retryable, not as a request error.
Check the account errors. Use a disabled key in a test environment and confirm the application reports the problem instead of cycling through the chain.

Run these whenever the chain changes, and watch the cost of a test run: every attempt that succeeds is billed.

Frequently asked questions

Should my application switch to a different model automatically?

Only to models you have tested with your real prompts, and only when users can accept a different answer style. For internal tools and batch jobs, automatic fallback is usually fine. For customer-facing features, log or show which model answered.

How long should LLM API timeouts be?

Long enough for the slowest legitimate answer, and no longer. For anything that may reason or write at length, stream the response: on BoostRail a non-streaming request is limited to 180 seconds, while a stream only needs its first event within 45 seconds on each route.

Am I charged for failed attempts?

On BoostRail, a request that fails on every route is not charged. Each retry or fallback your code sends is a new request, billed normally if it succeeds.

What happens if a stream breaks halfway?

BoostRail does not move a request once data has been sent, because a new attempt would produce a different answer. The client should keep what arrived, mark it incomplete and offer a retry. Billing for interrupted streams follows the rule on the pricing page.

Is retrying a 400 error ever useful?

Only after changing the request: trimming the input to fit the context window, removing a parameter the model rejects, or choosing a model that accepts it. Sending the same request again returns the same error.

Chinese AI Models in 2026: Kimi, Qwen, GLM, DeepSeek

Lee Gao — Sat, 17 Oct 2026 08:32:08 +0000

Originally published on boostrail.com on October 16, 2026.

Chinese AI models had a busy 2026: the major labs released new flagships aimed at coding and agent work, and most of them shipped with open weights and one-million-token context windows. For a team outside China, the practical questions are narrower than the headlines: which model is meant for which job, what it costs per token, and how to call several of them without opening an account with each lab.

This guide covers seven labs whose current models are in the BoostRail catalog, as of 16 October 2026. Model descriptions come from each lab's own model card or announcement, linked in each section. Benchmark figures in those sources are reported by the labs themselves, so treat them as a starting point for your own tests. The capabilities described are the models' own; which input types a given API passes through depends on the provider serving it.

Why Chinese AI models are worth evaluating

Open weights. Most of the models below publish their weights, under licences that range from MIT and Apache 2.0 to each lab's own terms. That gives you a way out if you later want to host a model yourself, and it lets the wider community test the models independently.
Long context as standard. Kimi K3, Qwen3.8-Max, DeepSeek V4.1 Flash, MiniMax M3, MiMo V2.6 Pro and Hy4 preview all list a context window of about one million tokens.
Price. On the BoostRail catalog, DeepSeek V4.1 Flash's output price is about one sixteenth of Claude Sonnet 5's. Prices vary: Kimi K3 costs more per output token than Claude Sonnet 5, while the other flagships in this guide cost less per output token.
Coding and agents first. Nearly every flagship released this year leads its announcement with coding, tool use and long-running agent tasks.

The seven labs and their current models

Moonshot AI: Kimi

Kimi K3 is a mixture-of-experts model with 2.8 trillion total parameters, 104 billion of them active per token, and a context window of 1,048,576 tokens. Moonshot's model card describes it as built for long-horizon coding, agentic knowledge work and reasoning, accepts text and images, and notes that it works best with Moonshot's own Kimi Code CLI. The weights are released under the Kimi K3 License. The catalog also carries the earlier kimi-k2.6, kimi-k2.7-code and kimi-k2.5 at lower prices.

Alibaba: Qwen

Alibaba announced Qwen3.8-Max on 3 August 2026 as the most capable model in the Qwen series so far: about 2.4 trillion total parameters with roughly 95 billion active, a context window of up to one million tokens, and native image and video understanding. The announcement leads with autonomous coding and long-horizon execution, and said the weights would follow. Qwen's strength for builders is the ladder below the flagship: qwen3.7-plus, qwen3.7-flash and qwen3-coder-flash cover cheaper tiers from the same lab.

Z.ai: GLM

GLM-5.3 is a 753-billion-parameter open-weights model. Z.ai's model card calls it its most capable open-weights model for coding and says it shares GLM-5.2's base model, with the gains coming from post-training. The card highlights terminal and software-engineering benchmarks such as Terminal Bench and DeepSWE. For high-volume work, glm-5.3-flash is the low-cost option.

DeepSeek

DeepSeek V4.1 Flash arrived in September 2026 as the successor to V4 Flash. The model card describes 552 billion backbone parameters, with 8 billion active per token when reading input and 16 billion when writing output, a context of up to one million tokens, native image input, and an MIT licence. DeepSeek positions it for cost-efficient, input-heavy agent workloads. For how the V4 family split work between its two tiers, see DeepSeek V4 Pro vs Flash.

MiniMax

MiniMax M3 has about 428 billion total parameters with about 23 billion active, and a one-million-token context. The model card emphasises long-horizon agent work and coding, and long-context efficiency from MiniMax's sparse attention design. It accepts text, images and video. The weights are under the MiniMax Community License. minimax-m2.7 is the cheaper previous generation.

Xiaomi: MiMo

MiMo V2.6 Pro is Xiaomi's flagship: a sparse mixture-of-experts model with 1.02 trillion total and 42 billion active parameters, a one-million-token context, and input across text, images, video and audio. Xiaomi publishes it under the MIT licence; see the model card. mimo-v2.6-flash is the smaller companion model.

Tencent: Hy

Hy4 preview is Tencent's new flagship, with 770 billion total parameters, about 49 billion active, a one-million-token context and an Apache 2.0 licence. The model card is explicit that this is an early version, and lists known issues including overly long reasoning and over-verification. Worth testing, with that in mind.

Prices on BoostRail

Prices in USD per 1M tokens on 16 October 2026. Live prices for every model are on the Models page.

Lab	Model id	Input	Output	Cache read
Moonshot	`kimi-k3`	2.85	15	0.3165
Alibaba	`qwen3.8-max`	1.9	5.7	0.19
Alibaba	`qwen3.7-flash`	0.1	0.4	0.02
Z.ai	`glm-5.3`	1.33	4.18	0.247
Z.ai	`glm-5.3-flash`	0.1425	0.475	0.0285
DeepSeek	`deepseek-v4.1-flash`	0.15	0.6	0.018
MiniMax	`minimax-m3`	0.3	1.2	0.06
Xiaomi	`mimo-v2.6-pro`	0.41325	0.8265	0.00342
Tencent	`hy4-preview`	0.834	2.501	0.042

For reference, claude-sonnet-5 is 1.9 input and 9.5 output on the same date. For how these prices turn into a monthly bill, see LLM API Pricing Explained.

Which model for which job

This is a starting shortlist based on what each lab says the model is built for. Your own test is what decides.

Job	Models to try first
Main model for a coding agent	`kimi-k3`, `glm-5.3`, `qwen3.8-max`
Long documents and repositories	`kimi-k3`, `qwen3.8-max`, `deepseek-v4.1-flash`
High-volume, low-cost steps	`deepseek-v4.1-flash`, `qwen3.7-flash`, `glm-5.3-flash`
Long agent runs on a budget	`minimax-m3`, `deepseek-v4.1-flash`

Several of these models also work as the main model inside Claude Code through the Anthropic format; see Claude Code with Other Models for the setup and our test results.

Calling Chinese AI models from outside China

There are three routes, each with a real trade-off.

Each lab's international platform. Most of these labs run their own international API platform. You get the lab's own pricing and features, and you manage one account, key, balance and set of docs per lab.
Self-hosting the open weights. Full control, and full responsibility. A model with hundreds of billions or trillions of parameters needs a multi-GPU cluster to serve at useful speed, so this pays off mainly at large, steady volume.
One API in front of all of them. BoostRail serves every model above through one OpenAI-compatible endpoint and one Anthropic-compatible endpoint, with one key and one USD balance. Switching models is a change to the model string. The OpenAI-compatible API page has the details.

Testing them on your own workload

Because the endpoint is the same for every model, a side-by-side test is a loop. This streams one prompt to five models and prints each answer as it arrives, followed by its token usage:

from openai import OpenAI

client = OpenAI(api_key="YOUR_API_KEY", base_url="https://api.boostrail.com/v1")
prompt = "Write a Python function that parses ISO 8601 durations such as PT1H30M."

for model in ["kimi-k3", "glm-5.3", "qwen3.8-max", "deepseek-v4.1-flash", "minimax-m3"]:
    print(f"\n=== {model}")
    stream = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        stream=True,
    )
    for chunk in stream:
        if chunk.choices:
            print(chunk.choices[0].delta.content or "", end="")
        if chunk.usage:
            print("\n", chunk.usage)

Streaming matters here. Reasoning models differ a lot in how much they think before the first visible word, and on BoostRail a non-streaming request stops at 180 seconds. When we ran this exact prompt on 16 October 2026, GLM-5.3 reasoned for about ten minutes and used 29,570 output tokens before finishing its answer, and Qwen3.8-Max took about three minutes and 17,329 output tokens. With max_tokens set to 4000, both spent the whole budget on reasoning and returned no text.

Most reasoning models also accept a reasoning effort setting, which BoostRail's Chat Completions endpoint takes as reasoning_effort. With reasoning_effort set to low, the same prompt took GLM-5.3 3,348 output tokens and Qwen3.8-Max 1,918, about a ninth of what they used by default. The effect varies by model, so measure it on each one.

Use prompts from your real workload, not puzzles, and compare three things: whether the answer is right, how many output tokens it took, and how long it took. All three show up in the bill and in what users wait for.

Frequently asked questions

Are Chinese AI models open source?

Most of the current flagships publish their weights. Licences differ: DeepSeek V4.1 Flash and MiMo V2.6 Pro use MIT, Hy4 preview uses Apache 2.0, and Kimi K3, MiniMax M3 and GLM-5.3 use each lab's own licence. Read the licence file before you build on self-hosted weights.

Can I use Chinese AI models from outside China?

Yes. Most of these labs run an international API, and BoostRail serves all the models in this guide through one key with USD billing.

Do they support tool calling?

On 16 October 2026 we ran a two-turn tool call through BoostRail's Anthropic-format endpoint on kimi-k3, glm-5.3, qwen3.8-max, minimax-m3 and deepseek-v4.1-flash, and all five called the tool and used its result correctly. The Models page also marks capabilities that passed automated checks on every route serving a model.

Which one is the cheapest?

In the table above, qwen3.7-flash has the lowest input and output prices, with glm-5.3-flash and deepseek-v4.1-flash close behind. Price per token is only half the comparison; check how many tokens each one needs to finish your task.

How do I switch between them?

Change the model string. The endpoint, the key and the request format stay the same, so you can route different tasks to different models in the same application.

LLM API Pricing Explained: How to Estimate Your Bill

Lee Gao — Sat, 17 Oct 2026 08:32:02 +0000

Originally published on boostrail.com on October 16, 2026.

LLM API pricing looks simple on a price list, two numbers per model, and becomes hard to predict once real traffic arrives. The bill depends on how many tokens go in, how many come out, how much of the input repeats, and how much hidden reasoning the model does before it answers. This guide explains each part, shows where the numbers come from with measurements we took on 16 October 2026, and works through a monthly estimate you can repeat with your own figures.

How LLM API pricing works

Almost every text model is priced per million tokens, with separate prices for input and output. Input is everything you send: the system prompt, the conversation so far, tool definitions, retrieved documents and the new message. Output is everything the model generates.

Three things about tokens matter more than the exact definition:

Tokens are not words. Each lab uses its own tokenizer, so the same text can be a different number of tokens on different models. Code, non-English text and long numbers often take more tokens than plain English prose.
Your request is bigger than your message. On Kimi K3, the three-word message "Reply with OK" was counted as 88 input tokens. Templates and formatting count too. Read the usage field in each response instead of estimating from the text you typed.
Output usually costs several times more than input. On the BoostRail catalog, Claude Sonnet 5's output price is five times its input price, and DeepSeek V4.1 Flash's is four times. Workloads that generate a lot of text feel this most.

The four token types on a bill

A modern LLM bill has up to four lines, even when the price list shows two numbers.

Token type	What it is	How it is usually billed
Input	New, uncached tokens you send	Input price
Cached input	Repeated prefix the provider recognises from an earlier request	Cache-read price, a fraction of the input price
Output	Text and tool calls the model returns	Output price
Reasoning	Thinking the model does before answering	Counted as output

Reasoning tokens are easy to miss. When we asked Kimi K3 to "Reply with OK", the visible answer was two characters and the response reported 98 output tokens, because the model reasoned before answering. With a budget of 16 tokens, the same request returned no text at all. Reasoning models need room in max_tokens, and that room is billed as output.

Cache writes are priced differently by each provider. Anthropic's own API, for example, charges more than the input price to write to its cache and a tenth of it to read, according to its prompt caching documentation. On BoostRail, cache writes carry no surcharge: the first pass bills at the normal input price, and later cache reads bill at the cache-read price published for each model, where the model's route supports caching.

Why agents cost more than chat

A chat turn sends one message and gets one answer. An agent turn sends the whole working context again: instructions, tool definitions, files it has read, command output and every earlier step. In Claude Code 2.1.295, a one-word reply still sent 20,751 input tokens. A long session repeats a prefix like that dozens of times per task.

Two consequences follow:

Input dominates. For coding agents, input tokens are usually many times the output tokens, so the input price and the cache-read price decide most of the bill.
Cache share decides the rest. If most of each request is a repeated prefix that hits the cache, the effective input price drops close to the cache-read price. If the prefix changes every turn, nothing is cached and every token bills at the full input price.

The cost calculator has three usage profiles built on this, and explains why the cache share matters more than the headline price. For the coding-specific side, see Claude Code with Other Models.

Worked example: estimating a monthly bill

Take the calculator's middle profile, roughly one developer using a coding agent every day: 30M input tokens and 6M output tokens a month, with 70% of the input served from cache. That splits the input into 21M cached tokens and 9M uncached tokens.

Prices from the BoostRail catalog on 16 October 2026, in USD per 1M tokens:

Model	Input	Output	Cache read
`deepseek-v4.1-flash`	0.15	0.60	0.018
`claude-sonnet-5`	1.90	9.50	0.19

The formula is the same for every model: uncached input × input price, plus cached input × cache-read price, plus output × output price, each divided by one million.

Line, in USD	`deepseek-v4.1-flash`	`claude-sonnet-5`
9M uncached input	1.35	17.10
21M cached input	0.38	3.99
6M output	3.60	57.00
Month	5.33	78.09

So the same month of work costs about $5.33 on DeepSeek V4.1 Flash and about $78.09 on Claude Sonnet 5. Without caching, the Sonnet month would be $114.00, because all 30M input tokens would bill at the full input price. The gap between models is large, and so is the gap between a cache-friendly setup and one that is not.

Two cautions keep the estimate honest. The token counts are the hard part: measure a week of real usage from the usage fields before you trust a monthly figure. And a cheaper model that needs more turns to finish a task can cost more per task than its price suggests, so compare cost per finished task, not cost per token.

Other charges that can appear

Images and video. Image inputs are counted in tokens on most text models. Video generation is usually priced per second of output instead, by resolution; see the video models page.
Context caps. Some models on BoostRail accept inputs up to a stated cap at a single flat rate. Larger inputs return a context_length_exceeded error before any charge.
Interrupted streams. A streaming request closed before it completes is billed for the usage the provider reports, which can include reasoning tokens. The exact rule is in the pricing FAQ.
Failed requests. On BoostRail, when every route for a model fails, the request is not charged. Your own retries are new requests and are billed if they succeed.
Top-up fees. Card top-ups carry a 5.5% processing fee with a $0.80 minimum; USDT top-ups carry 1% with a $0.10 minimum. Pay-as-you-go use has no monthly subscription.

How to bring the bill down

Keep the prompt prefix stable. Put fixed instructions and tool definitions first and changing content last, so repeated requests share a cacheable prefix.
Route by task. Send planning and hard reasoning to a strong model, and bulk summarising, classification and simple edits to a cheap one. The DeepSeek V4 Pro vs Flash post walks through one such split.
Choose the reasoning effort. Many reasoning models accept a lower effort setting (reasoning_effort on Chat Completions). On a coding prompt we tested, low cut GLM-5.3's output from 29,570 tokens to 3,348.
Set max_tokens on purpose. Enough for reasoning models to finish, not so much that a runaway answer gets expensive.
Cap spending per key. BoostRail keys can carry a daily or monthly spend limit; requests past it return key_spend_limit_exceeded instead of growing the bill. See Errors and failover.

Frequently asked questions

Is a token the same across models?

No. Each lab has its own tokenizer, so the same text produces different token counts on different models. Compare models on the usage they report for the same request, not on the price list alone.

Why is output more expensive than input?

Generating a token takes more compute time than reading one, because output is produced one token at a time, and labs price output accordingly. Prices in the catalog show output at several times the input price for most models.

How are reasoning tokens billed?

As output. The model's thinking is counted in the output tokens of the response, even when only the final answer is shown. A short answer from a reasoning model can therefore bill far more output tokens than its visible length.

What is the cheapest way to run a coding agent?

Use a cache-friendly setup, a low-priced model for routine steps, and a strong model only where it changes the result. Then measure cost per finished task over a week; the catalog price alone does not tell you that.

Where can I see what each request cost?

In the BoostRail console. It records the usage and the price applied to every request, which is the number that reaches your balance.

Claude Code with Other Models: DeepSeek, Kimi, GLM, Qwen

Lee Gao — Sat, 17 Oct 2026 08:31:56 +0000

Originally published on boostrail.com on October 16, 2026.

Running Claude Code with other models comes down to one fact: Claude Code speaks the Anthropic Messages format, and any endpoint that serves that format can answer it. Point Claude Code at such an endpoint, name a model, and the same terminal agent, tools and workflow run on DeepSeek, Kimi, GLM, Qwen or MiniMax instead of Claude.

This guide covers the setup, which model suits which kind of work, what we saw when we tested it on 16 October 2026, and the limits worth knowing before a long session. One point first, so it is not buried: Anthropic's own documentation says it does not support routing Claude Code to non-Claude models through any gateway. It works because the protocol is shared, not because anyone certifies the combination. Test with your own repository before you depend on it.

Why run Claude Code with other models

Three reasons come up repeatedly.

Cost per task. A coding agent resends its system prompt, tool definitions and conversation on every turn. In our test, a one-word reply in Claude Code 2.1.295 still sent 20,751 input tokens. At those volumes, the per-token price of the model decides the bill more than any single prompt does.
No usage window. A subscription meters agent use in time windows, and a heavy session can run out mid-task. Pay-per-token access through an API key has no window to wait out.
Choice per task. Some teams keep Claude for planning and hard debugging, and hand routine edits, test runs and repository exploration to a cheaper model. Claude Code lets you set the main model, the background model and the subagent model separately.

For the wider argument about subscription windows and rate limits, see Coding Agents and Rate Walls.

How it works

Claude Code reads its endpoint from ANTHROPIC_BASE_URL and its credential from ANTHROPIC_AUTH_TOKEN (sent as a Bearer token) or ANTHROPIC_API_KEY (sent as x-api-key). Every request goes to /v1/messages on that base URL, in Anthropic's format, with the model name you configured.

BoostRail serves /v1/messages for every text model in its catalog, not only Claude. The request is translated for the route serving that model, and the response comes back as Anthropic streaming events, including tool calls. The fields that cannot be applied are listed on the Messages reference and reported in the x-boostrail-degraded response header, so nothing is dropped silently.

Setup in three steps

Step 1. Create a key in the BoostRail console. The same key works for every model in the catalog.

Step 2. Put the settings in ~/.claude/settings.json. A settings file reaches desktop launches and background agents, which shell exports do not reliably do.

{
  "env": {
    "ANTHROPIC_BASE_URL": "https://api.boostrail.com",
    "ANTHROPIC_AUTH_TOKEN": "YOUR_API_KEY",
    "ANTHROPIC_MODEL": "kimi-k3[1m]",
    "ANTHROPIC_DEFAULT_HAIKU_MODEL": "deepseek-v4.1-flash",
    "CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS": "1"
  }
}

What each line does:

ANTHROPIC_AUTH_TOKEN is the credential variable Anthropic's documentation recommends when a gateway accepts Bearer tokens. While it is set, requests carry this key instead of your claude.ai login, and the subscription's limits do not apply to them.
ANTHROPIC_MODEL sets the main model. The [1m] suffix tells Claude Code the model has a one-million-token window; Claude Code strips it before sending, so BoostRail receives kimi-k3.
ANTHROPIC_DEFAULT_HAIKU_MODEL moves background work, such as session titles and summaries, to a cheaper model.
CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS stops Claude Code from sending pre-release fields that only Anthropic's own API accepts.

Step 3. Check the connection before a long session. This is a direct request with the same key:

curl https://api.boostrail.com/v1/messages \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{"model": "kimi-k3", "max_tokens": 256, "messages": [{"role": "user", "content": "Reply with OK"}]}'

A JSON body whose content array holds the reply means the key and base URL work. Leave room in max_tokens: reasoning models spend part of it thinking, and with a budget of 16 tokens Kimi K3 returned an empty content array. Then start claude, run /status, and look for the Anthropic base URL line. If it is missing, the variables did not reach the session. The coding agents page has the same settings for Codex CLI, Cline and other tools.

Which model for which job

The model ids below are the ones BoostRail accepts. The positioning comes from each lab's own model card or announcement, linked in the last column, so read it as the lab's claim rather than ours.

Model id	What the lab says it is built for	Context	Source
`kimi-k3`	Long-horizon coding, agentic knowledge work and reasoning	1M	Moonshot model card
`glm-5.3`	Coding; Z.ai calls it its most capable open-weights coding model	see card	Z.ai model card
`qwen3.8-max`	Autonomous coding and long-horizon execution	1M	Alibaba announcement
`minimax-m3`	Long-horizon agent work, coding and long-context efficiency	1M	MiniMax model card
`deepseek-v4.1-flash`	Cost-efficient, input-heavy agent workloads	1M	DeepSeek model card

A practical split many teams start with: a strong model as the main model for planning and multi-file changes, and deepseek-v4.1-flash for background work and subagents that read and summarise code. Prices for each model, input and output, are on the Models page. For a broader look at these labs, see Chinese AI Models in 2026.

What we tested

On 16 October 2026 we ran two checks with BoostRail's internal test key.

A two-turn tool call over /v1/messages. Each model received a read_file tool and the question "What Python version does this project need?". All five models in the table above called the tool with the right file, accepted the tool result, and answered correctly with end_turn. This shows the request and response shapes Claude Code relies on round-trip correctly for these models. It is a single test per model, not a guarantee across every route.

Claude Code itself. We ran Claude Code 2.1.295 in print mode against BoostRail, with glm-5.3 as the main model, and asked it to create a Python file and run it. It wrote the file, ran it with python3, and reported the correct output in three turns. Shorter runs with kimi-k3 and deepseek-v4.1-flash as the main model also returned normally. Two details from those runs are useful:

Usage showed cache reads on Kimi K3 and DeepSeek V4.1 Flash (cache_read_input_tokens above zero), so repeated context was billed at the cache-read price on those runs. GLM-5.3 showed almost none.
Claude Code's own cost figure is computed from its built-in price table, not from your bill. For the one-word DeepSeek reply it showed $0.083; the same tokens at DeepSeek V4.1 Flash's catalog price come to about $0.003. Check spending in the BoostRail console, which records the price applied to each request.

Limits to know before a long session

The /model picker will not list them. With gateway model discovery on, Claude Code only keeps ids that contain "claude" or "anthropic". Type the id instead: /model glm-5.3 switches the session.
Context window. For an id it does not recognise, Claude Code assumes 200K tokens and compacts the conversation before that. Add [1m] for models with a one-million-token window. For a model capped below 200K, such as qwen3-coder-flash on BoostRail, set CLAUDE_CODE_AUTO_COMPACT_WINDOW to its limit.
Token counting. BoostRail does not serve /v1/messages/count_tokens. Claude Code falls back to its own estimate, so /context shows approximate numbers.
Thinking blocks. Earlier thinking blocks in the conversation history are accepted and dropped, and named in x-boostrail-degraded. Each model reasons in its own way.
Auto mode. Auto mode sends a separate safety-classifier request for each action, and in our run it went to a Claude model rather than the one we configured. With Claude Code 2.1.295 and a non-Claude main model, auto mode blocked the shell command it could not evaluate. Use the default permission mode with allow rules for the commands you trust.
A diagnostic line. Print mode starts with [claude-code:unrecognized_model] for ids Claude Code does not know. It is informational; the request still goes through.

Mixing Claude and other models

Nothing forces an all-or-nothing switch. Because every model sits behind the same endpoint and key, you can keep claude-sonnet-5 as the main model and move only background work to deepseek-v4.1-flash, or the other way round. CLAUDE_CODE_SUBAGENT_MODEL sets the default model for subagents, and a subagent definition can still name its own model. When a session needs Claude's judgement for one hard step, /model claude-sonnet-5 switches back without touching the settings file.

If your team already holds contracts with model providers, BYOK runs catalog models on your own provider keys behind the same endpoint; see the BYOK guide.

Frequently asked questions

Does Anthropic support Claude Code with other models?

No. Anthropic's documentation states that it does not support routing Claude Code to non-Claude models through any gateway. The setup works because the Messages format is shared. Treat it as a configuration you own and test it on your own code.

Will my Claude subscription still be charged?

No. While ANTHROPIC_AUTH_TOKEN is set, Claude Code sends that key instead of your claude.ai login, and the subscription's usage limits do not apply to those requests. Usage is billed per token to the BoostRail account that owns the key.

Why is DeepSeek, Kimi or GLM missing from /model?

Gateway model discovery only adds ids that contain "claude" or "anthropic". Set the model in ANTHROPIC_MODEL, or type /model followed by the id during a session.

Does prompt caching work with non-Claude models?

It depends on the model's route. Where caching is supported, cache reads appear in cache_read_input_tokens and are billed at the cache-read price shown on the Models page. In our runs Kimi K3 and DeepSeek V4.1 Flash showed cache reads and GLM-5.3 showed very few.

Which settings file should I use?

Use ~/.claude/settings.json for your own machine. Do not put the key in a project's .claude/settings.json, which is committed to the repository and shared with everyone who clones it.