DEV Community

TapixAi
TapixAi

Posted on

What we learned measuring real token costs across 100+ models

We run a small OpenAI-compatible API relay. To price anything sanely we had to answer a boring question for every model we carry: what does one request actually cost us?

Official price lists did not answer it. So we measured. Over the past two weeks we sent fixed probes to every model on the catalogue and read back the delta on the upstream billing endpoint. A few things came out of that which surprised us, and which apply to anyone building on LLM APIs - even if you never touch a reseller.

1. The sticker price is the least interesting number

Every provider quotes two numbers: price per million input tokens and per million output tokens. Your bill is decided by neither of them directly.

What decided our actual cost, in order:

  • Output length. Output tokens cost roughly 4-8x input tokens. A model advertised at $0.50 / $3.00 is not "a $0.50 model" - the moment answers get long, you are paying the second number.
  • Retries. A 5% failure rate on 100k requests is 5,000 calls you pay for twice, plus the latency.
  • Prompt size drift. A "short" system prompt that quietly grows to 3k tokens changes the bill more than switching between two models in the same tier.

2. Reasoning tokens are billed even when you cannot see them

This one cost us a week. Some upstream relays return the model's thinking in reasoning_content. Others return it in reasoning. Others bill reasoning_tokens in the usage block and return nothing.

If you are building a UI that shows "thinking", you will find that:

  • the field name is not standard across relays
  • some routes silently drop it while still charging for it
  • usage.completion_tokens_details.reasoning_tokens is often the only honest signal that reasoning happened

We ended up normalising upstream field names into reasoning_content on our side, and for models whose route does not expose thinking at all, we say so in the model file instead of pretending.

3. Same model name, different cost

Two relays can serve "the same model" from different account tiers with different prices, and neither is lying. The model is the same; the capacity is not.

This is why per-model comparison pages are worth less than they look. A comparison table compares one quote against another quote. It does not tell you what the bill was after retries, caching and output length.

4. How we measure, concretely

For each model:

  1. Send a fixed prompt with max_tokens set high enough that the answer is not truncated (truncated answers make cost look cheaper than it is).
  2. Read the upstream billing counter before and after.
  3. Divide by the tokens actually reported - not by the tokens we guessed.
  4. Repeat until the sample is stable. If it never stabilises, we do not price the model; we mark the cost as low-confidence and freeze automatic repricing for it.

The ugly part: images, video and speech do not bill per token. Per image, per second, per character - each needs its own unit and its own probe. Mixing those into a token comparison produces numbers that are confidently wrong.

5. What we do with the numbers now

We keep a 20% net margin floor after fees, and two things enforce it:

  • a scheduled job that re-measures cost and raises prices when a model drifts below the floor (it never lowers them automatically)
  • a check at process start that walks the whole catalogue, and if any model is priced below the floor it reprices it before serving traffic

Both write an audit line. Both are boring. Boring is the point: pricing bugs are silent - nothing errors, you just make less money per request until someone notices months later.

6. What this means if you are the buyer

Three practical things:

  • Compare on your own traffic, not on a table. Take the ten prompts you actually send, run them against two providers, compare the invoice. Five minutes of work beats any comparison post (including this one).
  • Watch output length. Capping max_tokens and trimming system prompts is usually a bigger saving than changing vendors.
  • Ask which tier the traffic is served from. "Official price, 70% off" is a real thing; so is "same name, different capacity". A provider that can answer that question is usually the one worth keeping.

We publish our list price next to the official price for every model we carry, including the models where we are not cheaper - those exist, and hiding them would make the rest of the table worthless.

If you want to test the numbers yourself, there is a $1 credit on signup at tapixai.com, enough for a few hundred short replies. Break it if you can; that is the useful outcome.

Top comments (0)