ai maya

Posted on Sep 16 Originally published at huggingface.co

The Same Model Can Cost 14x More Depending on Who Serves It

#llm #api #performance #opensource

You picked a model. You are not done.

On a routing platform, one model name maps to many providers, and they do not agree on price.
We pulled per-provider pricing for every model on OpenRouter and compared the cheapest and priciest
endpoint for the same weights.

Of the 405 paid models on the platform, 182 are served by two or more paying providers. Across
all of them: median spread 1.87x, widest 14.47x, and 46% vary by 2x or more.

Same weights. Same model card. Same API call. Up to fourteen times the invoice.

Why the spread exists

Three variables move independently underneath a single model name:

Precision. A provider serving fp4 is not serving the same thing as one serving bf16, even though
the model id is identical. It is cheaper for them and it is a different artifact for you.

Uptime. 89.3% and 99.9% are both "available" on a status page. One of them wakes you up.

Where the data goes. 71 of those 182 models — 39% — are served from providers in more than one
country. If you have a data residency obligation, the model id tells you nothing and the provider
list tells you everything.

None of this is hidden. It is just spread across separate pages, so nobody assembles it before
signing off on a model.

Latency: don't inherit it, measure it

The latency field in the public model data is null. That is the honest answer from an aggregator —
it depends on the provider, the region and the moment.

So we called 329 models on a paid API with one fixed prompt and recorded time-to-first-token and
tokens per second. The harness is unremarkable and that is the point:

t0 = time.perf_counter()
ttft = None
with client.stream("POST", URL, json={"model": mid, "stream": True,
                                      "messages": [{"role": "user", "content": PROMPT}],
                                      "max_tokens": MAX_TOKENS,
                                      # Reasoning models will spend the whole budget thinking
                                      # and return empty content. Turn it off for a speed test.
                                      "reasoning": {"enabled": False}}) as r:
    for line in r.iter_lines():
        if not line.startswith("data: "):
            continue
        if ttft is None:
            ttft = (time.perf_counter() - t0) * 1000   # first token, milliseconds

Two properties of this data will quietly break a harness:

Reasoning budgets consume the whole allowance. With reasoning enabled, a model can return empty
content after spending every token you granted. You measure nothing and still pay for the call.

Undisclosed prices arrive as -1. If you sum projected costs to skip expensive models, those
negatives pull the total down and every model passes the "too expensive" check. Clamp at zero
before summing.

We show our measurement next to OpenRouter's own p50–p99 percentiles, which aggregate thousands of
real requests. They answer different questions: theirs is steady-state across traffic, ours is one
controlled call with a known prompt, provider and moment. Merging them would destroy both.

The column nobody publishes

We also grade Korean-language quality on seven axes — register, knowledge of Korean institutions,
format compliance, honorifics, summary accuracy, terminology, and freedom from translationese — for
330 models.

Two axes collapse across almost the whole field:

Axis	A rate
Honorifics	8.5%
Knowledge of Korean institutions	9.4%
Terminology	31.2%
Format compliance	47.6%
Register	53.3%

This is a failure mode developers should recognise, because it is the same shape as a silent
exception. The model writes fluent, natural Korean and gets the honorific wrong. Fluent output that
is wrong passes review; broken output does not. A test suite that only checks "did it answer in
Korean" will never catch it.

And it does not correlate with the things you would use as a proxy. A 2023 model,
gpt-3.5-turbo-16k, scores a perfect 3.00, above most 2026 flagships. Release date, parameter count
and English benchmarks do not predict it.

Use the data

All of it is open, no key required:

# per-provider price, precision, uptime and HQ for one model
curl .../api/models/deepseek/deepseek-v3.2/endpoints

# latency and throughput percentiles, per provider
curl .../api/models/deepseek/deepseek-v3.2/perf

# our measurements and grades
curl .../api/speed
curl .../api/korean   # includes per-axis detail, not just the band

Base URL is the Space's app URL:
https://huggingface.co/spaces/ginigen-ai/open-router-leaderboard

425 models, 329 measured for latency, 330 graded for Korean, next to price, provider, precision and
uptime. Three languages, three currencies, refreshed daily.

The practical takeaway

Choosing a model is one decision. Choosing an endpoint is a second one, and it moves your bill by up
to an order of magnitude, your reliability by nine percentage points of uptime, and your compliance
posture by a border.

If your procurement checklist stops at the model name, it stops one step early.

Full writeup with methodology: https://huggingface.co/blog/ginigen-ai/openrouter-leaderboard

DEV Community