DEV Community: smakosh

OpenAI Subscription Update: ChatGPT Pro vs DevPass

smakosh — Tue, 29 Sep 2026 16:41:03 +0000

Originally published on LLM Gateway.

The subscription price can stay the same while the usage behind it changes.
The OpenAI subscription update does exactly that for the $200 Pro tier:
OpenAI says its new usage calculation represents half the old allowance in
API-dollar terms.

DevPass, the coding subscription from LLM Gateway, is changing too. Its
planned October 15 update reduces monthly included usage from 3× to 2× the
plan price and adds daily caps. A useful comparison needs both changes on the
table, not yesterday's more generous numbers.

Here's what each subscription buys, where the limits differ, and which one
fits your coding workflow. Pricing and announcements checked September 29, 2026;
amounts below are in USD.

Understand the OpenAI subscription update

In OpenAI's announcement,
Tibo says new $200 Pro subscriptions are reopening with a revised usage
calculation. The stated change is half the API-dollar equivalent of the old
$200 Pro plan, not a lower subscription price.

For Codex/Work usage, the revised comparison is:

ChatGPT plan	Monthly price	Usage relative to Plus
Plus	$20	1×
Pro, $100 tier	$100	5×
Pro, $200 tier	$200	10× equivalent under the announced recalculation

The 10× figure reflects the halving of the previous 20× tier. At publication,
OpenAI's Pro help page
still describes the earlier 20× allowance and signup pause. Treat this table as
the announced Codex/Work change, not a claim that every account or every ChatGPT
feature has already switched. Check your account for the limits that apply to
your subscription.

The practical difference: the $200 tier now scales proportionally with the
$100 tier in this comparison. Twice the price buys twice the relative usage,
not four times.

There is an important benefit on OpenAI's side. The announcement commits to
not bringing back the five-hour limit, so subscribers can use their weekly
allowance when they need it. OpenAI also argues that cheaper, more capable
models can deliver more work per dollar. A 50% reduction in API-dollar
allowance is not proof of 50% fewer completed coding tasks.

Include DevPass's upcoming 2× allowance

PR #4039 contains the
upcoming DevPass allowance and pacing changes. It is still open as of this
post. The published October 15 notice
sets out when they apply.

The reduction is from 3× to 2×, not a halving. You lose one-third of the
standard monthly allowance; the subscription price stays the same.

DevPass plan	Monthly price, unchanged	Current 3× monthly allowance	Upcoming 2× monthly allowance
Lite	$29	$87	$58
Pro	$79	$237	$158
Max	$179	$537	$358

These allowance amounts measure eligible model usage at provider rates. They
are not cash, unrestricted API credits, or a promise of a fixed number of
prompts. The model, context length, output, and caching affect how fast you
use them. Check the live DevPass model directory
for supported models and their categories.

The rollout has two clocks:

Monthly allowance: new subscriptions from October 15, 2026 start at 2×. Existing subscriptions move to 2× at their first renewal on or after that date. Immediate upgrades starting a new billing cycle from that date also use 2×, subject to the upgrade rollover rule.
Daily caps, premium weekly caps, and Reset Pass changes: these apply on October 15, including during a billing cycle that started earlier. Cap percentages use the new standard monthly allowance, even while that cycle retains its earlier monthly pool.

Budget for daily and premium limits

The monthly number is not the amount you can spend in one sitting. PR #4039
adds a rolling 24-hour cap across all models and tightens the premium weekly
cap from 12% / 15% / 18% to 10% / 12% / 15% for Lite / Pro / Max.

Planned limit or benefit	Lite	Pro	Max
Daily cap, all models	8% ($4.64)	9% ($14.22)	10% ($35.80)
Premium weekly cap	10% ($5.80)	12% ($18.96)	15% ($53.70)
Price per Reset Pass	$5	$15	$45
Included Reset Passes per billing cycle	0	0	2

Daily and weekly amounts above are portions of the monthly pool, not extra
usage. Premium requests count against both caps. Pro loses its included
Reset Pass; Max keeps two. A Reset Pass restores only the premium weekly
allowance. It does not replenish the daily or monthly allowance.

At a cap, requests are rejected with 402, not paused or queued, unless
pay-as-you-go overflow is enabled and credits are available. Otherwise, callers
must retry after the applicable reset. Overflow costs extra; it is not included
in the subscription. The PR adds an overflow choice at signup, off by default.

Compare the workflow, not just the multiplier

OpenAI's 10× means usage relative to ChatGPT Plus. DevPass's 2× means
provider-rate usage relative to the subscription price. They have different
baselines. Dividing one by the other tells you nothing about how many coding
tasks either plan can complete.

What matters to you	ChatGPT subscription	DevPass after the planned update
Where you work	Native ChatGPT and Codex experience	Supported coding tools with one DevPass key
Model choice	OpenAI's subscription offering	Eligible models across providers in the DevPass directory
How usage is expressed	Relative plan allowances and product-specific limits	Provider-rate monthly pool, daily cap, and premium weekly cap
Bursty coding sessions	Announced $200 Pro plan: no five-hour limit; weekly allowance still applies	Rolling 24-hour cap, plus premium weekly cap
Beyond coding	ChatGPT's broader product features	Not a replacement for ChatGPT's general-purpose subscription

Choose ChatGPT Plus or Pro if you want OpenAI's native experience. If your
work stays in ChatGPT and Codex, switching providers may solve no problem for
you. For the announced $200 Pro plan, concentrating weekly usage into a busy
day also matters. A ChatGPT subscription does not include OpenAI API billing
credits.

Choose DevPass if you want your coding budget to follow your tools and
model choices. Use one key across supported tools such as Codex CLI, Claude
Code, Cursor, and OpenCode. You can change the model you use for a task without
buying a separate subscription for each provider. For example, DevPass Pro's
planned $79 monthly price covers $158 of eligible usage, provided your work
fits its daily and premium limits.

DevPass is not unlimited. It does not grant ChatGPT Pro membership or every
native Codex feature. Under the
upcoming scope terms, billing periods
starting on or after October 15 are for interactive software development
through approved coding tools only. General-purpose chat, content generation,
and automation are not included. For applications or direct API workloads,
use standard LLM Gateway credits instead.

Keep Codex CLI, change how you pay for inference

You do not have to abandon Codex CLI to try DevPass. Follow the
Codex CLI custom-provider guide,
configure LLM Gateway as the provider, and use your DevPass key. Requests on
that configuration draw from DevPass, not your ChatGPT subscription allowance.
Your native ChatGPT subscription remains separate.

Before choosing a tier, look at a normal coding day and your busiest one.
Check total usage, premium-model usage, and whether a daily cap would interrupt
you. A larger monthly pool is useful only if you can use it when you need it.

Choose your DevPass plan after reviewing the pricing and October changes.
Set up your coding tool.
Compare other AI coding plans.

An LLM Decision API That Returns Values, Not Text

smakosh — Sat, 26 Sep 2026 12:59:42 +0000

Originally published on LLM Gateway. Disclosure: I work on LLM Gateway; this republication was prepared with AI assistance.

A lot of production LLM traffic is not generation at all. It is a model being
asked a question with a small, known answer set: which team owns this ticket,
is this passage relevant to the query, does this citation support the claim, is
this record a duplicate. The answer is one of three options, or a number
between zero and one — and yet the call goes to a chat model, which writes a
sentence about it.

That costs more than it looks. You write prompt scaffolding to force JSON, you
parse and validate the result, you retry when the model prefixes it with
"Sure!", and you pay for every output token spent saying something your code
immediately throws away. Worst of all, you get a verdict with no uncertainty
attached: the model says billing with exactly as much apparent conviction
when it is sure as when it is guessing.

LLM Gateway now has a dedicated LLM decision API for this shape of
work. POST /v1/systemone serves System One models: models that read a
state, answer questions you name, and return typed values with calibrated
probabilities. There is no free-form text in the response.

Three question types

A request has a state — the thing being evaluated — and a map of questions
keyed by ids you choose. Answers come back under the same ids.

Type	Ask	Answer
`noul`	A yes/no question	`noul`: probability the answer is yes, 0 to 1
`choice`	One option from a set you define	`choice`, `probabilities` per option, `confidence`
`score`	A rating across ordered levels	`score` (can land between levels), `legend`, `confidence`

Choice questions take two to 255 options, each with an optional rubric. Score
questions take two to ten ordered levels, lowest first. Both state and each
instructions field accept structured data as well as plain text, so a
question can refer to fields of the record you passed in.

curl -X POST "https://api.llmgateway.io/v1/systemone" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "jev-1.13.0",
    "state": "Help! My payouts have been failing for 3 days.",
    "questions": {
      "department": {
        "type": "choice",
        "instructions": "Which team should handle this?",
        "criteria": {
          "billing": "Payments, invoicing, refunds",
          "technical": "Bugs, outages, integrations",
          "sales": "Pricing, upgrades, new accounts"
        }
      },
      "is_urgent": {
        "type": "noul",
        "instructions": "Does this convey urgency?"
      },
      "impact": {
        "type": "score",
        "instructions": "Rate the operational impact.",
        "criteria": ["None", "Limited", "Critical"]
      }
    }
  }'

{
  "model": "typesafe/jev-1.13.0",
  "answers": {
    "department": {
      "type": "choice",
      "choice": "billing",
      "probabilities": { "billing": 0.88, "technical": 0.12, "sales": 0.0 },
      "confidence": 0.81
    },
    "is_urgent": { "type": "noul", "noul": 0.95 },
    "impact": {
      "type": "score",
      "score": 1.9,
      "legend": { "0": "None", "1": "Limited", "2": "Critical" },
      "probabilities": { "0": 0.0, "1": 0.1, "2": 0.9 },
      "confidence": 0.88
    }
  },
  "usage": { "input_tokens": 318, "output_tokens": 34 }
}

The policy stays in your code

The reason a probability matters more than a label is that the cutoff is a
product decision, not a model decision. A support router and a fraud hold want
very different thresholds on the same question, and neither wants to renegotiate
it by editing a prompt.

With typed answers, the threshold is an if. confidence gives you a second
axis: it is derived from the probability distribution, so you can send an
uncertain case to a human instead of acting on what is effectively a coin flip.

const { answers } = await res.json();

if (answers.department.confidence < 0.7) {
  await queueForTriage(ticket);
} else {
  await route(ticket, answers.department.choice, {
    priority: answers.is_urgent.noul > 0.8 ? "high" : "normal",
  });
}

Because the answer set is fixed by the request, there is no schema drift to
guard against. answers.department.choice is one of the keys you sent, every
time.

Ask everything at once

The state is read once and every question is evaluated against it, so a batch
of questions costs far less than one request each. That changes what is worth
asking. Speculative questions — ones you only read when another answer makes
them relevant — become close to free, so you can resolve a whole decision tree
in a single round trip instead of chaining calls.

The trade-off is that question text counts as input on every call. A large
rubric attached to every request has a real per-request cost, so keep criteria
tight and push detail into the state where it belongs.

Jev 1.13, billed on input only

The first decision model on the gateway is typesafe/jev-1.13.0 from TypeSafe
AI, at $0.042 per million input tokens with output free — the response is a
set of values, not generated text. Moving aliases such as jev-latest are
accepted and resolve to the pinned version, so a request is always billed and
logged against a version whose price you can look up.

The endpoint runs the same machinery as every other route on the gateway: API
key auth, IAM rules, compliance policies, credit gating, provider key rotation,
cancellation, and request logging with a per-model cost breakdown. The rate
limit is 600 requests per minute per organization, and decision requests share
the same fleet-wide concurrency budget as every other inference endpoint. See
the rate limits reference
for the full table.

A few constraints are worth knowing before you build on it:

Decision models only work on /v1/systemone. Requesting one on /v1/chat/completions returns a 400 pointing at the right endpoint.
They cannot generate text or call tools, and they are not in the Playground.
Input is text only. There is no streaming — the response is a single object.

When a decision model is the wrong tool

This is not a replacement for a chat model that happens to classify. If the
answer set is open-ended, if you need the model to explain its reasoning to a
human, or if the classification is one step inside a longer generative task,
stay on /v1/chat/completions — and if you want structured output there,
structured outputs is still
the right mechanism.

Reach for a decision model when the call exists only to produce a value your
code branches on, when you want a probability rather than an assertion, and
when you are making enough of these calls that paying for output tokens on
throwaway prose adds up.

Compliance Alerts for Blocked AI Models

smakosh — Sat, 26 Sep 2026 12:59:07 +0000

Originally published on LLM Gateway. Disclosure: I work on LLM Gateway; this republication was prepared with AI assistance.

A new model lands, your team wants it, and the answer is no — not because the
model is unavailable, but because no provider serving it meets your compliance
requirements. No training on prompts. SOC 2 Type 2 or ISO 27001. Headquarters
in an approved country. Weeks later a provider that does meet all three quietly
starts serving that model, and nobody notices, because the only way to find out
was for someone to re-open the Models page and check by hand.

Compliance alerts close that gap. Pick the models your policy blocks today,
and LLM Gateway tells your team the moment one becomes routable under your
provider compliance policy.

Watch the models your policy blocks

On the Compliance page, the model picker only offers models your saved policy
currently blocks — the ones worth waiting for. Every minute, each watched model
is re-evaluated against that policy, and the first time a compliant provider
serves it, recipients get one alert naming the providers it can now route
through.

The check is a live evaluation, not a catalogue diff. That matters, because a
model becomes usable for several different reasons:

What changed	What you see
A compliant provider starts serving the model	"now available under your policy"
An existing provider earns a certification	Same alert, no catalogue change involved
You relax a requirement yourself	Same alert, so the team hears about it
The model loses its last compliant provider	The watch re-arms for the next opening

Because it is evaluated rather than diffed, a watch survives the messy cases:
providers added and removed, regional variants, and models that briefly appear
and disappear.

Hear about downgrades too

Compliance moves in both directions. When a provider stops meeting your policy
— a certification lapses, or its data policy changes to allow training on API
data — routing silently narrows. A downgrade alert names the failed
requirements, and for owners and admins it lists the models your organization
ran on that provider in the last 30 days that no longer have a compliant
alternative, so you know what to migrate before someone's request 403s.

Edits to your own policy never trigger downgrade alerts. Tightening a
requirement is a decision you already made; only a change on the provider's
side is news.

Recipients by role, not by checkbox

Instead of ticking individual people — unworkable once an org has dozens of
members — you choose the lowest role that should hear about it:

Audience	Who receives alerts
Owners only	Organization owners
Owners and admins (default)	Owners and admins
Everyone in the organization	All members, including project-scoped roles

Each level includes the roles above it, and membership is resolved when the
alert fires, so new hires and role changes are picked up without anyone
revisiting the settings. Individual recipients can still turn off in-app or
email delivery for their own account from the notification bell.

Delivery: in-app, email, and Slack

In-app and email alerts go to each recipient; email only ever goes to verified
addresses. Slack is org-wide: save an incoming webhook once under Preferences
→ Notification Channels, send a test message to confirm it, then enable Slack
in the alert settings. Slack receives one post per event no matter how large
the audience, which makes it the right channel for a shared
#ai-platform room.

The webhook URL is stored encrypted and only ever displayed masked, and only
https://hooks.slack.com/services/… URLs are accepted.

Getting started

Enable a provider compliance policy on the Compliance page — requirements, allowed headquarters countries, and any provider or model restrictions.
Scroll to Compliance Alerts and watch the models you are waiting on. The first watch saves the defaults: owners and admins, in-app and email, downgrade alerts on.
Optionally connect Slack, then adjust the audience.

Compliance alerts are available on the Enterprise plan, alongside the
compliance policy itself. Owners and admins configure them, every change is
recorded in the audit log, and
an organization can watch up to 100 models.

LLM Cost Optimization at $100k a Month

smakosh — Sat, 26 Sep 2026 12:58:31 +0000

Originally published on LLM Gateway. Disclosure: I work on LLM Gateway; this republication was prepared with AI assistance.

If you spend $100,000 a month on inference, the largest line item is usually
not a model that is too expensive. It is a model that is too expensive for
part of your traffic. The same endpoint answers "reword this paragraph" and
"find the race condition in this scheduler", and if you pinned a frontier model
to be safe, you are paying frontier prices for the first one several million
times a month.

This post is a worked model of LLM cost optimization through difficulty-based
routing at that scale. Every price below is the current list price from the
LLM Gateway catalogue; every assumption is
stated so you can substitute your own.

The assumptions, stated up front

This is a model, not a measurement. It rests on four things, and if any of them
are wrong for you the answer changes:

A representative request is 2,000 input tokens and 500 output tokens. Long context shifts the arithmetic toward input price; heavy generation shifts it toward output.
Prices are Anthropic first-party list prices, as served through the gateway.
Traffic mixes are illustrative. The whole result hinges on what fraction of your requests are genuinely easy, and that is the number you should measure rather than borrow from a blog post.
Quality is held constant by assumption. In reality, routing trades some quality risk on the traffic you push down. More on that at the end.

At those token counts the three models cost:

Model	Input / output per 1M	Cost per request
Claude Haiku 4.5	$1.00 / $5.00	$0.0045
Claude Sonnet 4.6	$3.00 / $15.00	$0.0135
Claude Opus 4.6	$5.00 / $25.00	$0.0225

Case 1: you pinned the frontier model

$100,000 a month at $0.0225 a request is about 4.4 million requests. Now
route them by difficulty instead, into Low / Medium / High bands backed by those
three models:

Traffic mix (Haiku / Sonnet / Opus)	Inference	Classifier	Net saving
Conservative — 20 / 50 / 30	$64,000	$467	$35,500 (36%)
Typical — 40 / 40 / 20	$52,000	$467	$47,500 (48%)
Aggressive — 60 / 30 / 10	$40,000	$467	$59,500 (60%)

Even the conservative mix — where 30% of traffic still goes to Opus and only a
fifth drops to Haiku — takes a third off the bill. The classifier is under half
a percent of the saving in every row.

Case 2: you pinned the mid-tier model

Pinning Sonnet is the more common starting point, and the honest one to model,
because the headline number is smaller. $100,000 at $0.0135 is about 7.4
million requests, and now routing cuts both ways: some traffic moves down to
Haiku, but some moves up to Opus.

Traffic mix (Haiku / Sonnet / Opus)	Inference	Classifier	Net saving
Conservative — 30 / 55 / 15	$90,000	$778	$9,200 (9%)
Typical — 45 / 40 / 15	$80,000	$778	$19,200 (19%)
Aggressive — 60 / 30 / 10	$66,700	$778	$32,600 (33%)

Smaller, but note what you are buying alongside it: 15% of your traffic now
reaches a frontier model it previously did not. That is a quality gain you were
not getting before, paid for out of the savings on the easy half.

What the classifier costs

TypeSafe's Jev decision model is
$0.042 per million input tokens, with output priced at zero. One
classification is the rubric plus a bounded slice of the request:

Scenario	Input tokens	Cost per call
Short prompt, 3 configured models	~1,100	$0.000046
Mid prompt, ~10 models	~2,500	$0.000105
Capped state, 30 models	~4,000	$0.000168

About $0.0001 a call, or 10,000 classifications per dollar. Each call is
billed to your account as its own log entry, against the same project and API
key as the request that triggered it — so the line above is something you can
read off your own usage rather than a figure you have to model. The rubric grows
with your model list — each configured model contributes its name, description
and price band to the question — so a 30-model list costs roughly twice a
3-model list per call.

And a request carrying a session id is classified once for the whole
conversation, not once per turn. For agent or chat traffic averaging ten turns,
every classifier figure above divides by ten.

The break-even is about 1%

The cleanest way to think about whether this is worth switching on:

Moving one request from Sonnet to Haiku saves $0.009. A classification costs
$0.000105. The classifier pays for itself if more than about 1.2% of your
requests move down a band.

Below that threshold your traffic is uniformly hard and you should pin a model
and skip the classifier. Above it — and almost all mixed production traffic is
well above it — the classifier is noise against the saving. That asymmetry is
the actual argument for difficulty routing: the downside is bounded at a tenth
of a percent of spend, and the upside is tens of percent.

Where this model is optimistic

Three honest caveats, because a savings estimate with no failure modes is a
sales pitch:

Quality is not free. Every request you push to a cheaper model is a bet that
the cheaper model handles it. A classifier makes that bet better than a coin
flip, but it does not make it free. Measure output quality on the traffic that
moves down, not just the invoice.

Your mix is probably not ours. The 40/40/20 row is an illustration. Run your
own traffic through a classifier in observe-only mode — point the Low and Medium
bands at the model you already use — and read the recorded difficulty
distribution off your logs before you change any routing.

Cheap traffic changes the ratio. These numbers assume a $0.0135–$0.0225
average request. If your average request costs $0.002, the classifier is around
5% of spend rather than 0.5%, and the case is weaker unless sessions amortize it.

Running it

On LLM Gateway this is configuration, not code. Choose the models smart may
resolve to and the classifier, and the gateway sorts them into price bands and
routes each request to the matching one. Every decision — the difficulty, the
task type, the candidates, the model chosen — lands on the request's log entry,
so you can audit the mix you actually got against the mix you assumed here.

Smart routing itself carries no platform fee: you pay for the models it selects
and for each classifier call, at the rates above. A verdict reused from a sticky
session and a classifier call that fails are both charged nothing. It is
available to every organization including pay-as-you-go while it is in beta; it
is not available on DevPass yet. If you want the routing logic explicit instead, a
dynamic route can branch on
the same verdict.

Automatic Model Selection by Request Difficulty

smakosh — Sat, 26 Sep 2026 12:57:58 +0000

Originally published on LLM Gateway. Disclosure: I work on LLM Gateway; this republication was prepared with AI assistance.

Most production traffic is not uniform. The same endpoint gets "reword this
paragraph" and "find the race condition in this scheduler" within a second of
each other, and both are billed at whatever model you hardcoded. Pin a frontier
model and you overpay for the easy half. Pin a cheap one and the hard half
comes back wrong.

The usual workaround is to classify requests in your own application code and
pick a model per branch — which means maintaining a prompt, a model list, and a
fallback path that all drift from the catalogue. LLM Gateway now does
smart routing for you, and reads the request before deciding.

Choose the models, choose the classifier

Sending "model": "auto" picks the cheapest model that fits the request, from
a fixed list. Smart routing is a separate model string, "model": "smart",
where you control the candidate list and how it is ranked — auto keeps its
current behaviour for existing callers. Under Organization settings → Smart Routing, pick up to 30 models
from the catalogue and a classifier. A project
can override the organization default on its own routing settings page.

Classifier	Behavior
None	Pick the cheapest model on your list that can serve the request
Jev (TypeSafe)	Rate the request first, then serve it from the matching price band

With the Jev classifier, your models are sorted by blended average price and
split into Low, Medium and High bands — the dashboard shows which model lands
in which band before you save. Each request is then rated for difficulty, task
type and output type and served from the matching band.

curl https://api.llmgateway.io/v1/chat/completions \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "smart",
    "messages": [{"role": "user", "content": "Prove this queue is linearizable."}]
  }'

Nothing about your request changes. The same call that used to resolve to a
fixed cheap model now resolves to whichever of your models the request actually
warrants.

Your list is a boundary, not a preference

This is the part worth being precise about, because it is what makes the
feature safe to turn on for a whole organization.

If no model on your list can serve a request — an image request against a
text-only list, say — the gateway returns a 400. It does not quietly fall
back to a model you did not approve. free_models_only narrows your list to
its free models rather than replacing it, so a request parameter cannot route
outside what you configured.

The classifier itself fails open in the other direction. A timeout, an outage,
a missing credential, or a compliance policy that blocks the classifier's
provider all fall back to the cheapest eligible model on your list. A
classifier problem slows a request down; it never fails one.

One verdict per conversation

Rating every turn of a long chat would be both wasteful and wrong. Wasteful
because you pay for a classifier call per turn; wrong because a conversation
that hops between models loses the upstream prompt cache it had been building.

So a request carrying a session id is classified once. The rest of the
conversation reuses that verdict and the model it resolved to, for the duration
of the sticky session.
Driving a coding agent through the gateway, a full multi-turn task makes
exactly one classifier call.

Branch on the verdict in a dynamic route

If you want the routing logic explicit rather than implicit,
dynamic routes gain a
classifier node that branches on the same verdict:

{
  "id": "rate",
  "type": "classifier",
  "kind": "jev",
  "on": "difficulty",
  "cases": [
    { "value": "high", "next": "frontier" },
    { "value": "low", "next": "cheap" }
  ],
  "else": "balanced"
}

on also accepts task and outputType, so you can send coding traffic one
way and summarization another. Only graphs containing a classifier node pay for
the call, and else covers both an unmatched answer and no verdict at all —
give it a real model rather than a reject branch.

Every decision is on the log

Routing you cannot audit is routing you cannot trust. Each request records the
classifier used, the candidates it chose between, the difficulty, task and
output type, the band served, the selected model, and whether the verdict was
reused from earlier in the session. It is all on the request's entry in the
activity view, and it is cleared along with the rest of the request payload
when your data retention window elapses.

Getting started

Open Organization settings → Smart Routing.
Pick the models smart may resolve to and check the band preview.
Choose a classifier, save, and send "model": "smart".
Override it per project from that project's routing settings if a workload needs a different list.

Owners and organization admins set the default; project admins can override it.
The routing itself carries no platform fee: you pay for the models it selects,
plus each classifier call at its catalogue rate — about $0.0001, recorded as its
own log entry against the same project and API key.
If you are weighing this against routing by hand, the arithmetic in
cutting LLM costs with request routing
still applies — this is the same idea, with the classification and the fallback
handled for you.

Smart Routing Benchmark: Cost and Quality

smakosh — Sat, 26 Sep 2026 12:57:55 +0000

Originally published on LLM Gateway. Disclosure: I work on LLM Gateway; this republication was prepared with AI assistance.

Routing a prompt to a cheaper model only helps if the answer is useful. We ran a
live Smart Routing benchmark through LLM Gateway to measure both sides:
the complete request cost and whether the output passed its task's checks.

Smart Routing cost 33.2% less than the fixed premium baseline in the primary run. It also cost 35.6× as much as the fixed low-cost baseline. The savings estimate was uncertain, and the cheaper fixed baselines remained competitive on task success.

Compare the same prompts

We fixed 80 prompts before collecting results and sent each through four arms:
Smart Routing, a fixed low-cost model, a fixed mid-priced model, and a fixed
premium model. We ran the full comparison at two output caps, for 640 measured requests.
Prompt order and arm order were shuffled using a recorded seed.

The protocol contains the
exact baseline model IDs, prompts, order, and dataset hash. The results contain the selected
models, responses, scores, token counts, normalized costs, and timing data.
These snapshots make the run auditable. The fixed mid-priced baseline was
outside the configured candidate pool; the low-cost and premium baselines were
in it. Baseline labels describe this comparison, not the router’s price bands.
The results record the actual eligible pool and decision for each routed call.
For current availability, use the model catalogue.

Workload	Prompts	What the check measures
Easy tasks	25	Arithmetic, extraction, sorting, case conversion, sentiment labels
Exact-answer reasoning	15	Reference answers for math, algorithms, logic, and code tracing
Instruction following	40	40 sampled cases from 334 supported IFEval cases

The IFEval subset uses Google Research's dataset.
Its score checks instructions such as avoiding commas or formatting an answer
as JSON. It does not establish factual accuracy or writing quality. We
cross-checked the stored outputs against the official evaluator and independently
verified all 15 exact-answer reference solutions. This is a
subset score, not a full IFEval leaderboard result.

The primary comparison uses the same 8,192-token output limit for every arm.
The initial run used a 4,096-token cap. During that run, we found that this
prevents Smart Routing from raising reasoning effort for high-difficulty
requests. We completed that round and repeated the entire comparison at
8,192 tokens to exercise that behavior. The initial protocol and initial results are also
available. Requests ran serially with
gateway response caching and cross-provider fallback disabled. The fixed-model
arms still use gateway provider selection; they do not pin a provider. Provider-side
prompt caching could still occur. We left reasoning and temperature at their
defaults, including Smart Routing's automatic reasoning adjustment. This tests
the configured service end to end, rather than isolating the classifier.

Read the Smart Routing benchmark results

Arm	Cost index	Strict pass	Scoring sensitivity	Median client time
Smart Routing	66.82	75/80	75/79	6.48s
Fixed low-cost	1.88	74/80	74/79	5.12s
Fixed mid-priced	18.13	76/80	76/79	4.86s
Fixed premium	100.00	74/80	79/79	6.85s

Cost is indexed to the fixed premium arm's total, set to 100. Lower is cheaper.
The total includes the response's inference charge, the separate Jev classifier
charge, and storage for both. Classifier charges came from the actual request
records; they were not inferred from a typical price per call.

Against the premium baseline, the paired 95% interval ranged from 4.3% higher cost to 66.1% lower cost. It includes no savings, so this small sample does not establish a reliable reduction. Against the mid-priced baseline, Smart Routing cost 3.69× as much (paired interval: 1.90× to 5.99×). Against the low-cost baseline, Smart Routing cost 35.57× as much (paired interval: 20.11× to 51.97×).

Classification and its storage accounted for 0.53% of Smart Routing’s total bill. Most of the routed cost came from the selected models.

Latency is measured at the client and includes network and gateway delays.
During the run, some client timings substantially exceeded the logged inference
duration; both are available in the data. These timings describe this run’s
conditions, not a model-speed guarantee. Smart Routing’s client p95 was 66.57s,
compared with 41.09s for the premium baseline.

Check the failures before switching

Arm	Easy tasks	Exact-answer reasoning	IFEval subset
Smart Routing	25/25	12/15	38/40
Fixed low-cost	25/25	12/15	37/40
Fixed mid-priced	25/25	14/15	37/40
Fixed premium	20/25	15/15	39/40

Smart Routing had one provider access error and one client timeout. The low-cost baseline had one output-limit truncation. The other fixed arms completed without request errors or truncations. A timed-out stream counts as a failed, billed request; its recorded inference charge is included even though the client received only part of the answer.

We kept failures and truncated responses in the denominator. We also audited
the scoring rules. One IFEval prompt requires repeating text containing a word
twice while allowing that word fewer than twice in the response. Five easy
case-conversion prompts have ambiguous trailing punctuation. The original
strict scores remain above; a sensitivity check removes the contradictory
prompt for every arm and accepts an optional final period on those five tasks.

The sensitivity check puts Smart Routing at 75/79, compared with 74/79 for the low-cost baseline, 76/79 for the mid-priced baseline, and 79/79 for the premium baseline. The premium arm’s six strict failures were all covered by these ambiguity adjustments; its original 74/80 should not be read as evidence that routing improved answer quality.

The original strict pass-rate difference versus premium was 1.25 percentage points, with a paired 95% interval from -5.00 to 7.50 points. This does not establish equivalent quality.

Check sensitivity to the output cap

Output cap	Smart cost vs premium	Smart strict pass	Low-cost strict pass	Mid-priced strict pass	Premium strict pass
4,096	62.6% lower	71/80	75/80	74/80	73/80
8,192	33.2% lower	75/80	74/80	76/80	74/80

The higher-cap request logs confirm medium reasoning on both high-difficulty routed requests. On the recurrence task, the routed response changed from an incorrect answer in the lower-cap run to the correct answer in the higher-cap run. That routed request cost more in the higher-cap run.

These are separate live runs. Sampling and provider conditions can change
between them, so their difference does not isolate the causal effect of the cap.

Treat this as a pilot

We built Smart Routing and ran this evaluation ourselves. The runner and analysis scripts
are available for inspection and reproduction.

This is a deliberately mixed, single-turn, text-only workload. It does not
represent a particular application's production traffic, long-context work,
tool use, or sessions. Enabling fallback can change failure rates and cost;
session reuse can spread classifier cost across multiple turns. The model pool
and provider conditions are a snapshot
from the run. One response per model and prompt also leaves sampling variation
unmeasured.

The confidence intervals resample paired prompts within the three workload
groups. They quantify uncertainty within this sample; they do not establish
quality equivalence or predict performance after model or routing changes.

Use the model you would actually replace as the baseline. The mid-priced arm cost 72.9% less than Smart Routing and passed 76/80 tasks against its 75/80. The low-cost arm was much cheaper again, with one fewer passing task than Smart Routing. Those fixed baselines deserve consideration alongside routing. Smart Routing can lower a premium-model bill, but this pilot does not justify a blanket claim of equal quality at lower cost. Repeat the paired comparison on representative application tasks and with the candidate pool you plan to deploy.

Smart Routing is available to every organization while in beta, excluding
DevPass. Choose the candidate models and Jev classifier under organization
Smart Routing settings or a project's Routing settings. There is no additional
routing platform fee; inference and successful classifier calls are billed.

Try LLM Gateway
Read the Smart Routing documentation
How automatic model selection works

OpenRouter vs Vercel vs LLMGateway Performance

smakosh — Wed, 22 Jul 2026 17:43:45 +0000

Every AI gateway adds a hop between your app and the model. The question that matters is what that hop costs at the moment your user is staring at a blank chat window: the time to first token. Most gateway latency debates skip the measurement and argue architecture — so we measured it.

We ran an open-source TTFT benchmark against LLM Gateway and OpenRouter, interleaved, from the same machine, on the same model. The medians over 75 runs each: LLM Gateway got the first content token in 906ms on a cold connection and 814ms on a warm one. OpenRouter took 1392ms and 1232ms. That is roughly 35% faster cold and 34% faster warm, with zero errors across the 300 measured runs — 450 HTTP requests in total, counting the throwaway warm-up call that precedes each warm measurement, every one of which returned HTTP 200. The raw per-run data is published in full.

How we measured AI gateway performance

We used ai-gateways-benchmark, an open-source script by Ronny Badilla that recently made the rounds comparing Vercel AI Gateway, OpenRouter, and Cloudflare AI Gateway. It is Python stdlib only — raw sockets, no HTTP library — and it times every phase of a streaming request separately: DNS, TCP connect, TLS handshake, TTFB (request sent to first response byte), and TTFT (request sent to first content token in the SSE stream).

That separation is the point. "Gateway latency" claims usually conflate connection cost, edge proximity, and actual routing overhead. This tool splits them apart.

Our setup, on July 22, 2026:

Same model on both gateways: claude-haiku-4.5, streaming, max_tokens: 16, identical prompt
75 cold + 75 warm runs per gateway, interleaved round-robin so time-of-day drift hits both equally
Cold = a fresh connection paying full DNS + TCP + TLS with a new TLS context (no session resumption)
Warm = a second request on an already-open socket — the connection-pool case your production traffic mostly lives in
One residential vantage point, zero errors on either gateway across all 450 HTTP requests (including warm-ups)

Results: LLM Gateway vs OpenRouter

Medians of n=75 per cell. Cold TTFT is end-to-end: DNS + TCP + TLS + time to first content token — what a short-lived process pays.

Medians	TTFB (cold)	TTFT (cold)	TTFT (warm)
LLM Gateway	201ms	906ms	814ms
OpenRouter (direct)	1369ms	1392ms	1232ms

The spread, as median with p10–p90 range:

Metric	LLM Gateway	OpenRouter
Cold TTFB	201 (194–240)	1369 (979–1643)
Cold e2e TTFT	906 (803–1380)	1392 (1002–1675)
Warm TTFB	176 (171–193)	1229 (924–1500)
Warm TTFT	814 (673–1379)	1232 (924–1501)

Two things stand out beyond the medians. LLM Gateway's p90 cold TTFT (1380ms) came in below OpenRouter's median — the slow tail of one distribution beat the middle of the other. And LLM Gateway's warm TTFB barely moves: 171–193ms across the p10–p90 range, which is the stability you want under a production connection pool.

Where the time goes

The phase breakdown shows the overhead is not in the connection:

Gateway	DNS	TCP	TLS	TTFB	TTFT (request)
LLM Gateway	3.2	20.9	40.2	200.9	829.5
OpenRouter	3.4	7.2	12.7	1369.4	1369.8

OpenRouter's edge actually wins the handshake — 13ms TLS vs our 40ms. It loses everything after the request is sent.

One honest caveat on the TTFB column: it flatters LLM Gateway for an architectural reason. Our gateway starts the response stream in about 200ms, before the first upstream token arrives. OpenRouter holds its first byte until the first token is ready, so its TTFB equals its TTFT on every single run. TTFB tells you who streams headers early; TTFT is the number your users feel, and it is the honest headline of this post.

A second caveat: each gateway ran its default routing for this model. OpenRouter picks its upstream per request (a request we inspected was served via Amazon Bedrock), while our runs were pinned to Anthropic's API. Both are what you get out of the box, but they are different upstreams.

How Vercel AI Gateway compares

We did not benchmark Vercel ourselves. The benchmark's author published his own run of the same script — from his vantage point, on a different date — comparing Vercel AI Gateway, OpenRouter, and Cloudflare AI Gateway proxying OpenRouter:

His run (medians of n=5)	TTFB	TTFT (cold)	TTFT (warm)
Vercel AI Gateway	785ms	1099ms	822ms
OpenRouter (direct)	1100ms	1123ms	986ms
Cloudflare → OpenRouter	1300ms	1420ms	1279ms

These numbers are not directly comparable to ours — different location, different network, different day, n=5 vs n=75. What they are useful for is a sanity check: OpenRouter lands north of a second to first token in his run and in ours, from two different corners of the internet. Against his table, LLM Gateway's 906ms cold and 814ms warm sit at or below the best row — but the only comparison we will state as fact is the one we measured ourselves, interleaved, from one machine.

Location dependence cuts both ways, and the benchmark's README says so explicitly: results are a property of where you measure from, not a global ranking. Which is why the right move is to run it yourself.

Run the benchmark yourself

The whole thing is a config file and two API keys:

git clone https://github.com/rbadillap/ai-gateways-benchmark
cd ai-gateways-benchmark

{
  "runs_cold": 75,
  "runs_warm": 75,
  "prompt": "Reply with the single word: pong",
  "max_tokens": 16,
  "gateways": [
    {
      "name": "llmgateway",
      "host": "api.llmgateway.io",
      "path": "/v1/chat/completions",
      "model": "anthropic/claude-haiku-4-5",
      "auth_value": "Bearer $LLM_GATEWAY_API_KEY"
    },
    {
      "name": "openrouter",
      "host": "openrouter.ai",
      "path": "/api/v1/chat/completions",
      "model": "anthropic/claude-haiku-4.5",
      "auth_value": "Bearer $OPENROUTER_API_KEY"
    }
  ]
}

LLM_GATEWAY_API_KEY=... OPENROUTER_API_KEY=... python3 bench.py config.json

It prints per-run lines while it works, then a medians table, and dumps raw per-run JSON with request-id receipts. If you run it from your region and get different numbers — including ones where we lose — we want to see them.

Latency is the tax you pay on every request

A gateway earns its hop with failover, unified billing, and one API across every model it routes. But you pay its latency on every single request, forever. That makes time to first token one of the few gateway properties worth measuring before you commit — and one of the easiest, since the tooling is open source and takes minutes to run.

If you are on OpenRouter today, LLM Gateway speaks the same OpenAI-compatible API — switching means changing the base URL and swapping in an LLM Gateway API key, covered in the OpenRouter migration guide.

Frequently Asked Questions

What is TTFT and why does it matter more than TTFB?

TTFT (time to first token) is the delay between sending a request and receiving the first piece of model output in the stream. TTFB (time to first byte) only measures when the server starts responding — a gateway can stream headers immediately while the model is still silent. For anything a user watches in real time, TTFT is the latency they actually experience.

What is a good TTFT for an AI gateway?

It depends on the model and your distance from the gateway's edge, so compare gateways against each other rather than an absolute bar. In this AI gateway performance benchmark, first tokens from claude-haiku-4.5 arrived in roughly 800–900ms through LLM Gateway and 1200–1400ms through OpenRouter, measured from the same machine in the same session.

Are these results valid from every location?

No — latency benchmarks are a property of the vantage point, and the benchmark's own README is explicit about it. Our numbers come from one residential connection on one day; the author's Vercel numbers come from another. The script is open source and takes minutes to run, so measure from where your servers actually live.

Does LLM Gateway support the same models as OpenRouter?

LLM Gateway routes an overlapping catalogue — the full list is on the models page, spanning the major closed and open-weight providers on the providers page. Requests use the OpenAI-compatible format either way, so moving a workload between the two is a base URL and key change.

Measure it yourself:

Try LLM Gateway free — one key, streaming from day one
Migrate from OpenRouter — a base URL and API key change
What is an LLM gateway? — where the hop pays for itself

9 Best Open-Source LLMs in 2026 (Compared)

smakosh — Tue, 21 Jul 2026 15:49:11 +0000

Open-source LLMs stopped being the budget option in 2026. Kimi K3 sits level with Claude Opus 4.8 on the Artificial Analysis Intelligence Index (its hosted API is live; the weights themselves are expected by July 27), GLM-5.2 held the top open-model spot before it, and the field behind them is deep enough that the hard part is choosing.

This ranking covers the nine best open-weight models right now — on license, context window, hardware reality, and the per-token price you actually pay. Every one of them is available through LLM Gateway with one key, at each provider's published rate, so you can A/B any two of them by changing one word in a request.

1. Kimi K3 — the open frontier

Moonshot AI · 2.8T params · 1M context · $3.00 / $15.00 per M

The largest open-weight model ever announced — with one caveat: the weights are not downloadable yet. Moonshot expects to release them by July 27, 2026, and the license is still unannounced; the hosted API has been live since July 16. Ranks 4th of 189 models on the Artificial Analysis Intelligence Index — tied with Claude Opus 4.8 and GPT-5.5 — and took first place in Arena's blind Frontend Code testing. Always-on reasoning, vision, tools, and output configurable up to 1M tokens. The open model to beat, priced accordingly. Full breakdown here.

Best for: teams that want closed-frontier quality with open-weight freedom.

2. GLM-5.2 — the value flagship

Z.ai · 744B params · 1M context · $1.40 / $4.40 per M

MIT-licensed, weights on Hugging Face, and the top-ranked open model until K3 arrived. A real 1M-token context, strong agentic-coding results, and built-in web search support — with output at under a third of K3's rate and input at about half. Also the largest model on this list that fits a single 8-GPU node (or one 512 GB Mac Studio) at INT4.

Best for: the best capability-per-dollar in the open field.

3. DeepSeek V4 Pro — frontier scale at commodity prices

DeepSeek · 1.6T params (49B active) · 1M context · $0.435 / $0.87 per M

MIT-licensed MoE with an aggressive sparse design: 1.6T total parameters, 49B active. The result is a 1M-context frontier-class model priced under a dollar per million output tokens — an order of magnitude below K3. Reasoning, tools, JSON output; no vision.

Best for: high-volume production workloads where unit cost decides.

4. Kimi K2.6 — last generation's champion, now cheap

Moonshot AI · 262K context · $0.95 / $4.00 per M

Before K3, Kimi K2.6 topped the open-weight rankings on the Artificial Analysis Intelligence Index. It is still a heavyweight — vision included — and its price dropped into value territory the day its successor shipped.

Best for: near-frontier quality with a smaller bill and a proven track record.

5. MiniMax M3 — the multimodal one

MiniMax · 512K context · $0.60 / $2.40 per M

Open weights, native multimodal understanding (text and images in one model), and MiniMax's sparse-attention design for efficient long-context inference. Posted 59.0% on SWE-Bench Pro at launch — above GPT-5.5 and Gemini 3.1 Pro on that benchmark.

Best for: agents that need to read screenshots, diagrams, and UI alongside code.

6. Nemotron 3 Ultra 550B — the enterprise workhorse

NVIDIA · 550B params (55B active) · 262K context · $0.50 / $2.50 per M

NVIDIA's open-weight flagship, tuned hard for agentic and enterprise workloads, with vision support and weights on Hugging Face. Backed by the deepest deployment tooling in the industry if you ever take it on-prem.

Best for: enterprises standardizing on NVIDIA's stack end to end.

7. Qwen3.6-35B-A3B — the local hero

Alibaba · 35B params (3B active) · 262K context · $0.25 / $1.49 per M

Apache 2.0, multimodal, and genuinely runnable on a 24 GB consumer GPU — 3B active parameters make it fast on modest hardware while it punches far above its size on agentic coding. The model to reach for when "open source" means "runs on my machine."

Best for: local development, edge deployment, and hobbyist hardware.

8. gpt-oss-120b — OpenAI's open one

OpenAI · 120B params · 131K context · $0.15 / $0.75 per M

Apache 2.0 and sized to fit a single 80 GB GPU. Not the newest entry on this list, but the combination of pedigree, permissive license, single-GPU deployment, and rock-bottom hosted pricing keeps it in the rotation.

Best for: cheap reasoning at scale and single-GPU self-hosting.

9. Llama 4 Maverick — the ecosystem incumbent

Meta · 1M context · $0.27 / $0.85 per M

Meta's open-weight line no longer leads benchmarks, but no model family matches its ecosystem: fine-tunes, guardrails, tooling, and institutional familiarity. Vision support and a 1M context at bargain pricing keep it a safe default.

Best for: teams already invested in the Llama toolchain.

Comparison table

#	Model	Lab	License	Context	Input $/M	Output $/M
1	Kimi K3	Moonshot	open weights*	1M	$3.00	$15.00
2	GLM-5.2	Z.ai	MIT	1M	$1.40	$4.40
3	DeepSeek V4 Pro	DeepSeek	MIT	1M	$0.435	$0.87
4	Kimi K2.6	Moonshot	open weights	262K	$0.95	$4.00
5	MiniMax M3	MiniMax	open weights	512K	$0.60	$2.40
6	Nemotron 3 Ultra	NVIDIA	open weights	262K	$0.50	$2.50
7	Qwen3.6-35B-A3B	Alibaba	Apache 2.0	262K	$0.25	$1.49
8	gpt-oss-120b	OpenAI	Apache 2.0	131K	$0.15	$0.75
9	Llama 4 Maverick	Meta	Llama license	1M	$0.27	$0.85

*Weights publish by July 27, 2026; Moonshot has not yet announced the license. Prices are each provider's published rate through LLM Gateway; the live list is on the models page.

How to choose

Maximum capability: Kimi K3, with GLM-5.2 as the value alternative
Production volume: DeepSeek V4 Pro or gpt-oss-120b — pennies per million
Multimodal agents: MiniMax M3, or Qwen3.6-35B-A3B at the small end
Self-hosting: match the model to your hardware — Qwen3.6-35B-A3B for a workstation, gpt-oss-120b for a single 80 GB GPU, GLM-5.2 for a full node
Coding agents: K3 for the hard problems, GLM-5.2 or DeepSeek V4 Pro for the loop — setup guide

One API for all nine

Every model in this list is a model string on the same endpoint:

curl https://api.llmgateway.io/v1/chat/completions \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "glm-5.2",
    "messages": [{"role": "user", "content": "Review this diff for race conditions."}]
  }'

Two ways to pay. Pay-as-you-go: top up from $10, pay the published per-token rates plus a 5% platform fee at top-up — right for shipping products. DevPass: flat $29/$79/$179 a month for coding agents, with roughly 3× your subscription price in model usage at provider rates. On DevPass, every model on this list except Kimi K3 is standard-tier with no weekly cap; K3 crosses the premium price threshold and draws from a weekly premium allowance.

Frequently Asked Questions

What is the best open-source LLM in 2026?

Kimi K3, by benchmark standing: 4th of 189 models on the Artificial Analysis Intelligence Index, level with Claude Opus 4.8 — though its weights are expected by July 27 and it is API-only until then. If price matters, GLM-5.2 delivers most of that capability with output at under a third of K3's price.

What is the best small open-source LLM?

Qwen3.6-35B-A3B — Apache 2.0, multimodal, 3B active parameters, and it runs on a 24 GB consumer GPU while holding its own on agentic coding benchmarks.

Are these models open source or open weight?

Open weight: the trained weights are downloadable and self-hostable under permissive licenses (mostly MIT or Apache 2.0), but training data and code are generally not released. GLM-5.2, DeepSeek V4 Pro, Qwen3.6-35B-A3B, and gpt-oss-120b use standard OSI licenses; Llama uses Meta's community license. Kimi K3 is the exception for now: its weights are expected by July 27, 2026 and are unlicensed until Moonshot announces terms — today it is usable only through hosted APIs.

What is the cheapest way to use open-source LLMs?

Through a gateway at provider list prices. LLM Gateway adds no per-token markup — pay-as-you-go credits carry a 5% platform fee at top-up, and DevPass turns heavy coding-agent usage into a flat monthly rate.

Getting started

Try LLM Gateway free — all nine models, one key
Get DevPass — flat-rate open models in your coding agent from $29/mo
Deep dive on the leader: Kimi K3 and China's Open-Weight Model Wave

Originally published on the LLM Gateway blog.

How to Use Kimi K3 with Claude Code, Cursor, and Cline

smakosh — Tue, 21 Jul 2026 15:48:36 +0000

Kimi K3 took first place in Arena's Frontend Code evaluation the week it launched, and it holds a 1M-token context — but Moonshot doesn't ship a coding agent, and your coding agent doesn't ship Kimi K3. Claude Code is locked to Anthropic's API by default, Cursor to its own backend, Cline to whatever key you hand it.

LLM Gateway bridges that gap. It speaks both the Anthropic and OpenAI API formats, so the tools you already use can run Kimi K3 — or any of 200+ models — with a base-URL change. Here is the exact setup for each tool.

Kimi K3 in Claude Code

Claude Code talks to any endpoint that speaks Anthropic's /v1/messages format, which LLM Gateway does natively. Three environment variables:

export ANTHROPIC_BASE_URL=https://api.llmgateway.io
export ANTHROPIC_AUTH_TOKEN=$LLM_GATEWAY_API_KEY
export ANTHROPIC_MODEL=kimi-k3

claude

That's the whole migration. Every request now routes through LLM Gateway to Kimi K3, and every request shows up in your dashboard with its exact cost, token counts, and cache-hit rate.

One refinement worth adding: Claude Code uses a second, smaller model for routine background work, and you can point it at something cheap — or free:

export ANTHROPIC_SMALL_FAST_MODEL=glm-4.7-flash-free

That puts K3 on the hard reasoning and a $0 model on the housekeeping.

Kimi K3 in Cursor

Cursor routes its chat / plan panel (Cmd/Ctrl + L) through a custom OpenAI-compatible endpoint. Setup:

Open Cursor Settings → Models
Add your LLM Gateway key under OpenAI API Key
Enable Override OpenAI Base URL and set it to https://api.llmgateway.io/v1
Add kimi-k3 as a custom model and select it

Be aware of the boundary: Cursor's Composer, inline edit (Cmd/Ctrl + K), and autocomplete are locked to Cursor's own backend and will not route through any external endpoint. Plan and chat with K3's full 1M context in Cursor; if you want K3 driving the actual agent loop, use Claude Code or Cline instead.

Kimi K3 in Cline

Cline is the straightforward one — it's built to bring your own key:

Open the Cline panel in VS Code and click the settings gear
Set API Provider to OpenAI Compatible
Base URL: https://api.llmgateway.io/v1
API Key: your LLM Gateway key
Model ID: kimi-k3

Cline's full agent loop — file edits, terminal commands, project scaffolding — now runs on K3. If you want a specific upstream, use moonshot/kimi-k3; otherwise the gateway picks a healthy provider and fails over automatically.

Also worth knowing: OpenCode ships LLM Gateway as a built-in provider, so there K3 is a login and a model pick, no URLs at all.

What Kimi K3 costs in your coding agent

Agent loops are token-hungry, which is exactly the case DevPass was built for — a flat monthly rate instead of a per-token bill:

Plan	Price	Model usage included
Lite	$29/mo	~$87 at provider rates
Pro	$79/mo	~$237
Max	$179/mo	~$537

Kimi K3 is a premium-tier model on DevPass (it crosses the $15-per-million-output threshold), so it draws from a weekly premium allowance — roughly $10 per week on Lite, $36 on Pro, $97 on Max. The practical pattern: K3 for planning and the gnarly bugs, a standard-tier model like GLM-5.2 or DeepSeek V4 Pro for the bulk of the loop — both uncapped within your monthly allowance. Pro and Max include one and two Reset Passes per cycle if you burn the premium allowance early.

Prefer straight metering? Pay-as-you-go credits work with the identical setup: top up from $10, pay Moonshot's published rates ($3.00/M input, $0.30/M cached, $15.00/M output) plus a 5% platform fee at top-up. K3's cached-input pricing matters here — agent loops re-send the same context every step, and cache hits bill at a tenth of the fresh rate.

Frequently Asked Questions

Does Claude Code work with non-Anthropic models like Kimi K3?

Yes. Claude Code sends Anthropic-format requests to whatever ANTHROPIC_BASE_URL points at. LLM Gateway accepts that format and translates to each provider behind the scenes, so ANTHROPIC_MODEL=kimi-k3 just works — as does any other model in the catalog.

Can Cursor's Composer or autocomplete use Kimi K3?

No. Cursor only honors a custom endpoint for the chat / plan panel; Composer, inline edit, and autocomplete stay on Cursor's backend regardless of your settings. For a full agent loop on K3, use Claude Code, Cline, or OpenCode.

Is Kimi K3 included in DevPass?

Yes, on every tier, as a premium-tier model with a weekly allowance on top of your monthly credit pool. Standard-tier models — including GLM-5.2 and DeepSeek V4 Pro — have no weekly cap.

Which coding tool is best for Kimi K3?

The ones that route their full agent loop through your endpoint: Claude Code, Cline, or OpenCode. Cursor is fine for K3-powered planning but keeps its agent features on its own models.

Getting started

Get DevPass — flat-rate Kimi K3 in your coding agent from $29/mo
Try LLM Gateway free — one key for K3 and 200+ models
New to K3? Start with Kimi K3 and China's Open-Weight Model Wave, or see how it stacks up in Kimi K3 vs Claude Opus 4.8

Originally published on the LLM Gateway blog.

Kimi K3 vs Claude Opus 4.8: Benchmarks, Price, Verdict

smakosh — Tue, 21 Jul 2026 15:48:00 +0000

Kimi K3 is the first open-weight model that makes the Kimi K3 vs Claude Opus 4.8 question worth asking seriously. On the Artificial Analysis Intelligence Index, K3 ranks fourth of 189 models — level with Opus 4.8 and GPT-5.5, behind only Claude Fable 5 and GPT-5.6 Sol. It also costs 40% less per token, and its weights are expected to be released by July 27.

The honest answer is not "K3 wins" or "Opus wins" — it depends on what you're optimizing for. Here are the numbers, then the verdict.

Benchmarks: Kimi K3 vs Claude Opus 4.8

Benchmark	Kimi K3	Claude Opus 4.8
Artificial Analysis Intelligence Index	57 (4th/189)	statistical tie
GPQA Diamond	93.5%	93.6%
Terminal-Bench 2.1	88.3%*	74.6%*
SWE-bench Verified	not published	88.6%
Arena Frontend Code	1st (1,679)	ranked below

*Reported on different harnesses (Opus 4.8's score uses the Terminus-2 harness), so treat the Terminal-Bench gap as directional, not exact.

Three takeaways. On graduate-level reasoning (GPQA Diamond) the models are statistically tied. On agentic terminal work, K3's published number is well ahead, with the harness caveat above. And in Arena's blind Frontend Code testing, developers ranked K3 first outright — ahead of every model, including Anthropic's.

Where Opus 4.8 keeps an edge: SWE-bench Verified at 88.6% is a published, battle-tested result K3 has no counterpart for yet, and Anthropic's models remain the default target for agent harnesses — Claude Code, MCP tooling, and most agentic scaffolds are tuned against Claude first. K3 is three days old; its ecosystem is not.

Pricing: 40% cheaper across the board

Per-token rates through LLM Gateway (each provider's published pricing):

Per million tokens	Kimi K3	Claude Opus 4.8
Input	$3.00	$5.00
Cached input	$0.30	$0.50
Output	$15.00	$25.00

Both have a 1M-token context window. K3's output limit defaults to 131K tokens but is configurable up to the full 1M in a single response; Opus 4.8 caps output at 128K.

Concrete math: a coding-agent workload of 100M input and 20M output tokens a month runs $600 on Kimi K3 and $1,000 on Claude Opus 4.8. Same ratio at any scale — K3 is 40% cheaper on input, cached input, and output alike.

One operational difference: K3's reasoning is always on at full effort — reasoning_effort currently accepts only max, with lower-effort modes promised in later updates. That is part of why its benchmark numbers are strong, but it means K3 spends thinking tokens even on trivial requests. Opus 4.8 lets you dial reasoning effort up and down per request today — cheaper and faster on the easy 80%.

What each one is for

Pick Kimi K3 when cost dominates, you want frontend-heavy code generation (the Arena result is real), you need single-shot outputs longer than 128K tokens, or open weights matter — self-hosting, fine-tuning, or simply not depending on one vendor's API terms.

Pick Claude Opus 4.8 when you're running mature agent harnesses tuned for Claude, you want configurable reasoning effort, or you need the SWE-bench-class repo-editing reliability that teams have already validated in production.

Or don't pick. The switch between them is a one-word change to the request body, which makes the real answer an A/B test on your own workload:

curl https://api.llmgateway.io/v1/chat/completions \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "kimi-k3",
    "messages": [{"role": "user", "content": "Implement the retry logic described in RFC-42.md"}]
  }'

Run a week on kimi-k3, a week on claude-opus-4-8, and read the answer off your LLM Gateway dashboard — per-model cost, latency, and token counts side by side.

Flat rate or pay-as-you-go

Both models count as premium-tier on DevPass ($29/$79/$179 per month), drawing from the weekly premium allowance — so a flat-rate plan covers the A/B test without a separate Anthropic subscription. On pay-as-you-go credits you pay the per-token rates above plus a 5% platform fee at top-up, from $10.

Frequently Asked Questions

Is Kimi K3 as good as Claude Opus 4.8?

On aggregate intelligence benchmarks, yes — Artificial Analysis ranks them level, and they are statistically tied on GPQA Diamond. On production coding-agent work, Opus 4.8 still has the deeper published record (88.6% SWE-bench Verified) and the more mature harness ecosystem. On frontend code generation, blind developer testing ranked K3 first.

How much cheaper is Kimi K3 than Claude Opus 4.8?

40% on every token class: $3.00 vs $5.00 per million input, $0.30 vs $0.50 cached, $15.00 vs $25.00 output.

Can I switch between Kimi K3 and Claude Opus 4.8 without code changes?

Yes. Through LLM Gateway both are the same OpenAI-compatible endpoint — swap the model field between kimi-k3 and claude-opus-4-8 and nothing else changes. Costs for both land in one dashboard.

Is Kimi K3 open source?

Not yet. Kimi K3's weights are expected by July 27, 2026 and the license has not been announced; until then it is API-only, like Opus 4.8 — the difference is that Opus stays closed. See our Kimi K3 overview for the full release details.

Getting started

Try LLM Gateway free — A/B Kimi K3 against Claude Opus 4.8 with one key
Get DevPass — both models on one flat rate, from $29/mo
Wire K3 into your editor with How to Use Kimi K3 with Claude Code, Cursor, and Cline

Originally published on the LLM Gateway blog.

Kimi K3 and China's Open-Weight Model Wave

smakosh — Tue, 21 Jul 2026 15:47:24 +0000

Moonshot AI released Kimi K3 on July 16, and the benchmarks put an open-weight model next to the best closed ones for the first time. The catch is access. K3 sits on Moonshot's platform, GLM-5.2 on Z.ai's, DeepSeek V4 Pro on DeepSeek's, MiniMax M3 on MiniMax's — four accounts, four billing relationships, four API dashboards, all before you have written a line of code.

LLM Gateway routes every one of them through a single OpenAI-compatible endpoint. One key, one bill, and a switch between Kimi K3 and any of 200+ models is a one-word change to your request.

What is Kimi K3?

Kimi K3 is Moonshot AI's flagship model for long-horizon coding and agentic work. At 2.8 trillion parameters — a mixture-of-experts design that activates 16 of its 896 experts per token — it is the largest open-weight model announced to date. Moonshot has committed to publishing the full weights by July 27, 2026.

The specs that matter in practice:

1M-token context window (1,048,576 tokens), with output configurable up to the same 1M — enough to hold a large repository plus its docs in a single request
Always-on reasoning — K3 thinks before every answer; there is no non-thinking mode
Vision, tool calls, and JSON output supported out of the box
Prompt caching at a 90% discount on repeated input

Early results back up the size. K3 ranks fourth of 189 models on the Artificial Analysis Intelligence Index — level with Claude Opus 4.8 and GPT-5.5 — and took first place in Arena's Frontend Code evaluation in blind developer testing. It posted 93.5% on GPQA Diamond and 88.3% on Terminal-Bench 2.1, the strongest open-weight results published on both at release.

Kimi K3 pricing

Through LLM Gateway you pay Moonshot's published per-token rates:

Tokens	Price per million
Input	$3.00
Cached input	$0.30
Output	$15.00

The cached-input rate is the number to watch. Coding agents re-send the same system prompt, file context, and conversation history on every step, so in a long agent session most of your input tokens are cache hits billed at a tenth of the fresh rate.

The open-weight wave K3 leads

K3 is the headline, but 2026 has produced a full roster of frontier-class open-weight models from Chinese labs — all MIT-licensed or close to it, all available through LLM Gateway at provider list prices:

Model	Lab	Released	Context	Input $/M	Output $/M
Kimi K3	Moonshot AI	Jul 2026	1M	$3.00	$15.00
GLM-5.2	Z.ai	Jun 2026	1M	$1.40	$4.40
MiniMax M3	MiniMax	Jun 2026	512K	$0.60	$2.40
DeepSeek V4 Pro	DeepSeek	Apr 2026	1M	$0.435	$0.87

Each fills a different slot. GLM-5.2 (744B, MIT license) was the top-ranked open model before K3 arrived — its output costs under a third of K3's rate ($4.40 vs $15.00 per million) and its input about 47% ($1.40 vs $3.00). DeepSeek V4 Pro (1.6T total, 49B active, MIT license) delivers a 1M-token context at under a dollar per million output tokens. MiniMax M3 is the multimodal option, taking image input alongside text. And if you want a zero-cost entry point, glm-4.7-flash-free runs through the same endpoint for $0.

One endpoint for all of them

LLM Gateway speaks the OpenAI API, so anything that can call OpenAI can call Kimi K3:

curl https://api.llmgateway.io/v1/chat/completions \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "kimi-k3",
    "messages": [{"role": "user", "content": "Refactor this module to remove the circular import."}]
  }'

Swapping to GLM-5.2 or DeepSeek V4 Pro is a change to the model field — no new SDK, no new key. If you need a specific upstream, pin it with the provider prefix (moonshot/kimi-k3); otherwise the gateway routes to a healthy provider and fails over automatically if one goes down. Kimi K3 is served by both Moonshot and Novita, so a single-provider outage does not take your app with it.

Every request lands in the same dashboard: per-model cost, token counts, cache-hit rates, and latency, across all providers.

Run Kimi K3 on a flat rate with DevPass

If K3 is headed for your coding agent, the cheapest way to run it hard is DevPass — a flat monthly plan that works with Claude Code, OpenCode, Cline, Cursor (chat/plan mode only), and anything else OpenAI- or Anthropic-compatible:

Plan	Price	Model usage included
Lite	$29/mo	~$87 at provider rates
Pro	$79/mo	~$237
Max	$179/mo	~$537

Every dollar of allowance is metered at the provider rates above, so on Pro you get roughly $237 of model usage for $79 — and a runaway agent loop cannot produce a surprise invoice.

One gating detail to know: at $15 per million output tokens, Kimi K3 counts as a premium-tier model on DevPass, so it draws from a weekly premium allowance — roughly $10 per week of premium usage on Lite, $36 on Pro, and $97 on Max, on top of your monthly pool. GLM-5.2, MiniMax M3, and DeepSeek V4 Pro are all standard-tier: no weekly cap, usable until your monthly allowance runs out. A workable pattern is K3 for the hardest tasks and GLM-5.2 or DeepSeek V4 Pro for the bulk of the loop. If you burn through the premium allowance mid-week, a Reset Pass restores it instantly — Pro includes one per cycle and Max includes two.

Or pay as you go

For production apps, PAYG credits skip the subscription entirely. Top up from $10, pay each provider's published per-token price as you go, and LLM Gateway adds a flat 5% platform fee at top-up — the per-token rates themselves are pass-through. You keep the same single endpoint, automatic failover, prompt caching, and cost dashboard, metered to the token.

The two models work together: DevPass for your own coding agents, PAYG for the product you ship.

Frequently Asked Questions

Is Kimi K3 open source?

Kimi K3 is an open-weight model. Moonshot AI announced it on July 16, 2026 and has committed to releasing the full 2.8T-parameter weights by July 27, 2026; the weights license has not been announced yet. The API is live now — self-hosting will be possible once the weights publish, though a 2.8T mixture-of-experts model demands serious hardware, which is why most teams will run it through an API.

How much does Kimi K3 cost?

Through LLM Gateway, Kimi K3 costs $3.00 per million input tokens, $0.30 per million cached input tokens, and $15.00 per million output tokens — Moonshot's published rates. On a DevPass plan it is included in the flat monthly price as a premium-tier model.

What is Kimi K3's context window?

1,048,576 tokens — a full 1M-token context, with output defaulting to 131K tokens and configurable up to the same 1M. The same headline context as GLM-5.2 and DeepSeek V4 Pro.

Can I use Kimi K3 with Claude Code or Cursor?

Yes. LLM Gateway exposes OpenAI- and Anthropic-compatible endpoints, so point your tool's base URL at https://api.llmgateway.io/v1 with your key and select kimi-k3 as the model. In Claude Code and Cline the full agent loop routes through LLM Gateway; Cursor honors an external endpoint only in its chat/plan panel — Composer, inline edit, and autocomplete stay on Cursor's own backend. With DevPass this is covered by the flat monthly rate. Step-by-step setup for each tool: How to Use Kimi K3 with Claude Code, Cursor, and Cline.

Getting started

Try LLM Gateway free — one key for Kimi K3 and 200+ other models
Get DevPass — flat-rate plans from $29/mo for coding agents
Read the API docs or see how the plans compare in 10 Best AI Coding Plans in 2026
Go deeper: Kimi K3 vs Claude Opus 4.8 and the full open-source LLM ranking

Originally published on the LLM Gateway blog.

AI Gateway Fees Compared: Who Marks Up Your Tokens?

smakosh — Wed, 08 Jul 2026 17:09:17 +0000

Comparing AI gateways on price sounds simple until you read the fine print. The number on the pricing page is rarely the number on your invoice, because gateway fees hide in two separate places: a platform fee on the credits you buy, and a markup on the tokens you spend. Miss either one and your "cheap" gateway quietly costs more than the provider you were trying to save money on.

This guide breaks down what the major AI gateways actually charge in 2026, using each vendor's published pricing. LLM Gateway is one of them, so we're biased — but the table below is the real picture, including where others are cheaper for a given workload.

Two fees, not one

Every gateway makes money in one (or both) of two ways:

Token markup — a percentage added on top of the provider's per-token rate. A 10% markup on a $5/1M-token model means you pay $5.50. This is the expensive one, because it scales with every token forever.
Platform / credit fee — a flat percentage taken when you load credits (often with a minimum), or a monthly/usage subscription. This is usually the smaller one.

The healthiest pricing model is zero token markup plus a transparent, predictable platform fee — and the option to drop even that by bringing your own provider keys.

What each gateway charges

Gateway	Token markup	Platform / subscription fee	Bring your own keys	Self-host
LLM Gateway	None	5% on credits (0% with your own keys)	Free	Free (AGPLv3)
OpenRouter	None	5.5% on card credits ($0.80 min; 5% crypto)	Free to 1M req/mo, then 5%	No
Vercel AI Gateway	None	Pay-as-you-go credits + payment fees	Free (paid tier)	No
Cloudflare AI Gateway	None	Free direct; +5% if you use Unified Billing	Yes	No
Eden AI	None	5.5% on credits	Yes	No
Portkey	None	Free tier; Production $49/mo (+$9/100k logs)	Yes	Partial
LiteLLM	None	Free (you pay infra); Enterprise from ~$250/mo	Yes	Yes

The good news: in 2026, none of the major gateways mark up tokens — they all pass provider rates through. The competition has moved to the platform fee, BYOK terms, and whether you can self-host to pay nothing at all.

The costs that don't show up on the pricing page

A few line items are easy to miss when you're comparing:

Payment-processing fees. Several gateways (Vercel, OpenRouter, Cloudflare's Unified Billing) pass through card-processing costs on top of, or as part of, the platform fee. On small top-ups these can dominate — OpenRouter's $0.80 minimum is 8% on a $10 purchase.
Log-retention and seat tiers. Observability-led gateways meter on logs or seats. Portkey's Production plan includes 100k logs/month, then bills $9 per additional 100k. Budget for the volume you'll actually generate.
Self-host infrastructure. "Free and open source" still costs the servers, Redis, and on-call time to run it. That's real money — it's just yours to control.

How to pay the least

Three levers, in order of impact:

Bring your own keys. If you already have provider accounts or volume discounts, BYOK lets you pay the provider directly. With LLM Gateway that's a 0% platform fee; with OpenRouter it's free up to 1M requests a month, then 5%.
Self-host. If data residency or absolute cost control matters, run the gateway yourself. LLM Gateway is AGPLv3 and self-hostable end to end; LiteLLM is a self-hosted proxy. Both drop your platform fee to zero.
Cache and route. The cheapest token is the one you never send. Built-in response caching makes repeat requests free, and routing simple requests to budget models keeps your blended cost down. Run your real traffic through the Token Cost Calculator before you commit.

Frequently Asked Questions

Do AI gateways mark up your tokens?

In 2026, the major ones don't. LLM Gateway, OpenRouter, Vercel AI Gateway, Cloudflare, Eden AI, Portkey, and LiteLLM all pass provider token rates through without a per-token markup. They make money on a platform/credit fee or a subscription instead — so compare those, not the token rates.

What is the cheapest AI gateway?

For most teams, the cheapest path is bringing your own provider keys (0% fee on LLM Gateway) or self-hosting an open-source gateway (LLM Gateway or LiteLLM), which removes the platform fee entirely. On the managed tier, LLM Gateway's flat 5% credit fee is among the lowest, and lower than OpenRouter's 5.5%.

Is a platform fee or a token markup worse?

A token markup is worse. It scales with every token you ever send, so it compounds as you grow. A flat platform fee on credits is a one-time percentage when you load funds — predictable and far smaller over time.

Pay provider rates, not gateway rates

The whole point of a gateway is to save you money and effort, not add a tax. LLM Gateway passes provider token rates through with no markup, charges a flat 5% on credits (or 0% with your own keys), and is free to self-host if you'd rather pay nothing at all.

Try LLM Gateway free | Run the Token Cost Calculator | See the 8 best AI gateways in 2026