Rebuilding an Internal Cost-Per-Token Lookup Table

#ai #machinelearning #tutorial #llm

The pricing table has two columns, input and output, and it has been right for two years. Adding a second provider to it produces an estimate that diverges from the invoice, and the direction of the error depends on which provider the traffic went to.

Why the two-column table fails

A cost table keyed on (model, direction) encodes an assumption: that every token is one of two kinds and each kind has one price. That was true of the first generation of chat APIs and it is no longer true of any of them. Today a single response can be billed across several dimensions at once, and two of those dimensions are not token counts at all.

The failure is not that the table gives a slightly wrong number. It is that the table has nowhere to put the information, so whoever extends it invents a convention — a synthetic model row called something like “model-cached”, or a multiplier hard-coded in the estimator — and the convention is undocumented and does not survive contact with the next provider. Rebuilding it properly is a day of work that removes a recurring argument about why finance’s number and engineering’s number differ.

Two opposite bugs from one formula

The formula everybody writes is input_count × input_rate + output_count × output_rate. Applied across providers without reading the definition of the fields, it is wrong twice, in opposite directions.

Under-billing. On the Anthropic Messages API, input_tokens does not mean the total input. The rate-limits documentation states it plainly: the field represents tokens after the last cache breakpoint, and total input is cache_read_input_tokens + cache_creation_input_tokens + input_tokens. Its example is a 200,000-token cached document with a 50-token question reporting input_tokens: 50. An estimator that multiplies input_tokens by the input rate on a heavily cached workload therefore reports a small fraction of the real input volume — and because cached reads are billed at a reduced rate rather than free, the cost is understated rather than merely mis-attributed.

Double-counting. On OpenAI’s Chat Completions, prompt_tokens is the total, and prompt_tokens_details.cached_tokens is a breakdown of how many of those were served from cache, with cache_write_tokens reported alongside it. An engineer who has just learned about the first bug and “fixes” it by adding the cached count to the prompt count now bills the cached tokens twice, at full rate.

Both come from the same source: treating a field name as self-explanatory. The rule that prevents them is to store, for every provider in the table, an explicit note of whether the top-level input count is inclusive or exclusive of cached tokens, and to make the estimator read that flag rather than assume.

The dimensions the table needs

Enumerate what actually appears on a bill, keeping each dimension separate even when a given provider does not charge for it.

Uncached input and output. The two you already have.
Cache read. Reported as cache_read_input_tokens or as prompt_tokens_details.cached_tokens. Priced below the base input rate.
Cache write, possibly split by lifetime. This is the field most tables cannot express. The Messages API reports cache_creation_input_tokens and, in a nested cache_creation object, separate ephemeral_5m_input_tokens and ephemeral_1h_input_tokens counts — two cache lifetimes with different write prices. There is no counterpart on a provider whose caching is automatic, so the column exists and is null there.
Reasoning or thinking tokens. Reported as completion_tokens_details.reasoning_tokens in one place and output_tokens_details.thinking_tokens in another. Generally billed as output, but you need them as a separate column because they are the single largest source of unexplained cost growth and cannot be attributed otherwise.
Service tier. The Messages API reports service_tier with values standard, priority and batch, each with its own multiplier. A tier field is not optional: batch and interactive traffic at the same nominal model price are different line items.
Per-request charges. Server-side tool use is billed per invocation, not per token — the usage object carries server_tool_use with counts such as web_search_requests and web_fetch_requests. A table with only per-token rates cannot represent this at all, which is why it is usually the largest unexplained residual.
Audio and other modalities. Reported under their own details fields, priced separately. Include the columns even if you do not use them yet; adding a dimension later means re-deriving history.

Rebuilding the table

Key rows on (provider, model, tier, effective_from). The effective date is what lets you re-cost history correctly after a price change instead of retroactively rewriting last quarter.
Give every dimension a column with an explicit null where the provider has no such charge, and never zero. Zero means free; null means not applicable, and an estimator that cannot tell them apart will silently treat an unpriced dimension as included.
Add the input_count_includes_cache boolean per provider, and make the estimator branch on it. This is the one line that prevents both bugs above.
Store rates per million tokens as integers in the smallest currency unit, not as floats. Float rates times large token counts produce totals that do not reconcile to the cent, and reconciliation to the cent is the entire point.
Write the estimator to consume the usage object rather than re-tokenising. Local re-tokenisation is a second source of truth and it is wrong for exactly the traffic that matters — see why token counts disagree.
Reconcile against one real invoice before you trust it. Take a day of traffic, compute the estimate per dimension, and compare with the provider’s own usage breakdown for that day. A residual over a percent or so means a dimension is missing, not that the arithmetic drifted.

-- One row per price, not one row per model.
CREATE TABLE token_price (
  provider          text    NOT NULL,
  model             text    NOT NULL,
  tier              text    NOT NULL,          -- standard | priority | batch
  effective_from    date    NOT NULL,
  currency          char(3) NOT NULL,
  -- micro-units per million tokens; NULL = dimension not applicable
  input_uncached    bigint,
  output            bigint,
  cache_read        bigint,
  cache_write_short bigint,                    -- shorter cache lifetime
  cache_write_long  bigint,                    -- longer cache lifetime, if offered
  reasoning         bigint,                    -- NULL where billed as output
  -- per-request, not per-token:
  tool_call_request bigint,
  input_count_includes_cache boolean NOT NULL,
  PRIMARY KEY (provider, model, tier, effective_from)
);

Every rate in a table of this shape is a figure a vendor changes without notice, which is why this page contains none. Populate it from the provider’s current published pricing on the day you build it, record the date you read it, and re-check at each reconciliation.

Deriving the numbers that matter

The table is an input; the deliverable is the arithmetic you can now do. Treat every rate below as a placeholder you substitute from your own table.

The cache break-even. Let the base input rate be r, the cache write rate w × r and the cache read rate c × r, with a prefix of P tokens reused across n requests within the cache lifetime. Without caching you pay n × P × r. With caching you pay one write plus n - 1 reads: P × r × (w + (n - 1) × c). Caching wins when n > (w - c) / (1 - c). The whole decision reduces to that one inequality, and the only inputs are the two multipliers from your table. Where two cache lifetimes are priced differently, run it twice: the longer lifetime raises w and is worth it only when your inter-request gap exceeds the shorter lifetime, which is a question about your traffic, not about price.

The reasoning-token sensitivity. If reasoning tokens are billed at the output rate and average k times the visible output, the effective output rate is (1 + k) times nominal. Compute k from your own logs by dividing the reasoning-token field by the visible output count over a day. Two models with the same published output price and different k have different real prices, and this ratio is the reason a migration to a nominally cheaper model can raise the bill.

The tool-call floor. A per-request tool charge is a fixed cost per invocation, so it dominates at short outputs and disappears at long ones. Divide the per-request charge by the output rate to get the number of output tokens at which it equals the generation cost; below that, your cost per request is essentially set by tool usage and no amount of prompt shortening will help.

These dimensions have to be normalised somewhere before a chargeback model can consume them, and doing it per service means every service gets the cached-token definition wrong independently. Multigrid reports usage in one normalised shape across providers, which is the same normalisation this page describes — if you build it yourself, build it once, behind the client, and make the per-service code consume your fields rather than the provider’s. The downstream consequence is on what happens to a chargeback model at a migration.