Re-Baselining Cost Estimates After Switching Providers

#ai #machinelearning #tutorial #llm

The instinct after a switch is to open the spreadsheet and change the price cells. That gives an answer with the right units and the wrong value, because at least two of the other inputs moved at the same time and one of them can move in the opposite direction to the price.

Why substituting the new price fails

A monthly bill for one workload is the product of four things, and a provider switch perturbs all four:

monthly_cost = requests_per_month
             x (input_tokens_per_request  x price_in)
             + requests_per_month
             x (output_tokens_per_request x price_out)

Only price_in and price_out are published numbers you can look up. input_tokens_per_request changes because the new provider counts your identical text with a different tokenizer. output_tokens_per_request changes because a different model writes at a different length for the same instruction — and that is a property of post-training, not of your prompt. requests_per_month can change too, if the new model needs fewer retries for schema compliance or more calls to reach the same answer.

The reason this matters rather than being a rounding concern: a switch to a headline price 30% lower, combined with a model that writes 50% longer answers, is a cost increase on output. Both those numbers are plausible and they point opposite ways. Substituting price alone hides the second one entirely.

The four factors

Price per input token and per output token. Look these up on the day, note the date next to them in your model, and do not hardcode them anywhere a reader will mistake for a fact. Output is normally the dearer of the two by a multiple, so an output-heavy workload is far more sensitive to it.
Input tokens per request. Measured, not assumed. The character count of your prompt did not change; the number of tokens it becomes did.
Output tokens per request. Measured on real traffic, not from a handful of manual tries. This is the factor with the widest spread and the one most likely to surprise you.
The discounts that do not carry. Prompt caching, batch processing and committed-throughput discounts are all provider-specific mechanisms with different eligibility rules. If your incumbent bill was substantially reduced by a cache hit rate on a long shared system prefix, you are not comparing like with like until you know whether the new provider caches at all, at what minimum prefix length, with what time-to-live, and whether cache writes are charged at a premium. A workload can be cheaper on paper and dearer in practice on this factor alone.

Measuring the two you cannot look up

Both providers return the counts they billed you for in the response body. Use those and not your own estimate: the returned number is by definition the number on the invoice, and your local tokenizer count is an approximation of it that omits per-message structural overhead.

On OpenAI’s Chat Completions the object is usage with fields prompt_tokens, completion_tokens and total_tokens; the Responses API names the same quantities input_tokens and output_tokens. On Anthropic’s Messages API it is usage with input_tokens and output_tokens, plus cache_creation_input_tokens and cache_read_input_tokens when prompt caching is in play — and those two are billed at different rates from ordinary input, so a cost model that sums them into one number is wrong.

The procedure that produces a defensible estimate rather than a guess:

Take a sample of real production requests — a few hundred, spanning a full weekday cycle, not the ten you have in a test file. Redact them if they contain customer data; the token counts survive redaction well enough if you replace text with text of similar length.
Replay them against the new provider with the ported prompt and the ported parameters. The ported parameters matter: an output cap that changed name and value will distort output length.
Record the returned usage fields per request. Keep the distribution, not the mean. Report p50 and p95 — a mean output length is dominated by the long tail and will over-estimate the typical request while under-estimating the expensive one.
Compute the same distribution from your incumbent’s logs over the same sample of requests, so the two are measured on identical inputs.
Multiply out. Do it twice: once at p50 for the expected bill, once at p95 for the number you use when setting an alert threshold.

Assembling the estimate

Write the model with the prices as named inputs at the top rather than inline, so the day they change you edit one place and see the effect:

# All four prices are inputs you fill in on the day, per 1M tokens.
P_IN_OLD, P_OUT_OLD = ..., ...
P_IN_NEW, P_OUT_NEW = ..., ...

# Measured from the replay, per request, at p50.
TOK_IN_OLD, TOK_OUT_OLD = ..., ...
TOK_IN_NEW, TOK_OUT_NEW = ..., ...

REQS = 2_000_000   # your own monthly volume

def monthly(p_in, p_out, t_in, t_out):
    return REQS * (t_in * p_in + t_out * p_out) / 1_000_000

delta = monthly(P_IN_NEW, P_OUT_NEW, TOK_IN_NEW, TOK_OUT_NEW) \
      - monthly(P_IN_OLD, P_OUT_OLD, TOK_IN_OLD, TOK_OUT_OLD)

Then run the one analysis that makes the model worth having: vary each of the four measured token figures by plus and minus twenty percent, one at a time, and see which one moves delta most. That factor is the one to go and measure properly on more traffic. Usually it is output length, because output is priced higher and varies more, but on a retrieval-augmented workload with a large context it is often input.

Do not model a switch as a single flag day in the spreadsheet. If you are moving traffic gradually, cost during the overlap is the sum of two bills, and any committed-spend discount on the incumbent is calculated against a volume that is now falling — which can push the effective incumbent unit price up exactly while you are trying to leave. Model the ramp explicitly, month by month, including the overlap.

The measurement step is the awkward one to build, because it means logging normalised usage figures from two providers whose usage objects have different field names and different cache accounting. If you are running requests through a gateway, that normalisation already happened in one place — Multigrid records per-request input and output tokens and attributed cost across providers under one schema, which is the shape the replay above needs. If you are not, log the raw usage object per provider and normalise in your warehouse; the thing to avoid is a per-service cost calculation that each team maintains separately.

Checking it against the first invoice

The estimate is not finished until it has been reconciled once. When the first full invoice arrives, compare it to what the model predicted for the same period. A gap over roughly ten percent is not noise and has a small set of usual causes:

Retries counted once in the model and twice on the bill. A failed request that produced tokens before failing is generally billable.
A background workload nobody included — evaluations, a regression suite in CI, a nightly batch job. These are frequently a double-digit share and are invisible in application logs.
Reasoning tokens, which are billed as output but do not appear in the text you received, so a model built from response lengths misses them entirely.
Cache accounting: cache writes charged above the base input rate, or a hit rate lower in production than in the replay because real traffic has more prefix variation.

Keep the reconciliation. The second switch is much cheaper to estimate than the first, but only if you wrote down where the first estimate was wrong. The single-request version of this arithmetic is worked through in why the same prompt costs a different amount on two providers.