DEV Community

Samuel Ochaba
Samuel Ochaba

Posted on AI-assisted

This LLM cost function looks right. Can you spot the bug?

#ai #openai #llm #python

OpenAI's reasoning models bill you for thinking you never see. So it seems careful to add that thinking to your cost tracking, like this:

from decimal import Decimal

# Made-up prices, in USD per million tokens.
INPUT_RATE = Decimal("0.50")
OUTPUT_RATE = Decimal("2.00")


def estimate_cost(usage):
    """Estimated USD cost of one OpenAI Responses API call."""
    reasoning = usage["output_tokens_details"]["reasoning_tokens"]
    billed_output = usage["output_tokens"] + reasoning
    return (usage["input_tokens"] * INPUT_RATE
            + billed_output * OUTPUT_RATE) / 1_000_000
Enter fullscreen mode Exit fullscreen mode

It runs, and it returns a believable number. It's also wrong for every response that used reasoning. Have a guess at why before you read on.

The answer: the reasoning is already in output_tokens

OpenAI's Responses API reports everything the model generates as output_tokens. Its token counting guide says: "Reported output token usage includes all tokens generated by the model, not only the text visible in a response." The reasoning_tokens value under output_tokens_details breaks that number down: it tells you how many of the output tokens were reasoning. So estimate_cost counts the reasoning twice.

You can see it in the example response in OpenAI's reasoning guide:

{
  "input_tokens": 75,
  "input_tokens_details": { "cached_tokens": 0 },
  "output_tokens": 1186,
  "output_tokens_details": { "reasoning_tokens": 1024 },
  "total_tokens": 1261
}
Enter fullscreen mode Exit fullscreen mode

The total is 75 + 1,186 = 1,261, so the 1,024 reasoning tokens are already inside the 1,186. Add them again and your log says 2,285 tokens for a response that reported 1,261. The same guide confirms the billing: "While reasoning tokens are not visible via the API, they still occupy space in the model's context window and are billed as output tokens."

The fix, and what the bug costs

Use output_tokens as it is. Here's the fixed function, with a made-up response to run both versions on:

def estimate_cost_fixed(usage):
    """Same estimate; output_tokens already includes the reasoning tokens."""
    billed_output = usage["output_tokens"]
    return (usage["input_tokens"] * INPUT_RATE
            + billed_output * OUTPUT_RATE) / 1_000_000


# A made-up response in the shape of OpenAI's Responses API usage object.
usage = {
    "input_tokens": 1200,
    "output_tokens": 900,
    "output_tokens_details": {"reasoning_tokens": 700},
}

for label, cost in [("Counted once: ", estimate_cost_fixed(usage)),
                    ("Counted twice:", estimate_cost(usage))]:
    print(f"{label} ${cost:.4f} per request, ${cost * 10_000:.2f} per 10,000")
Enter fullscreen mode Exit fullscreen mode

Put both blocks in one file and run it:

Counted once:  $0.0024 per request, $24.00 per 10,000
Counted twice: $0.0038 per request, $38.00 per 10,000
Enter fullscreen mode Exit fullscreen mode

On this made-up response, the buggy version is 58% too high. Both numbers look perfectly plausible, which is why the mistake is easy to miss.

Gemini goes the other way

Google's Gemini API splits it differently. In the generateContent reference, thoughtsTokenCount is the "Number of tokens of thoughts for thinking models", and totalTokenCount is "prompt + tool-use prompt + thoughts + response candidates". The thinking tokens sit outside candidatesTokenCount, so if you treat candidatesTokenCount as all of the output, they go missing.

No single formula covers both APIs. Adding the reasoning field counts tokens twice on OpenAI, and leaving it out drops tokens on Gemini. Read each provider's definitions before you write the formula.

Cached tokens hide the same trap

Some providers report cached tokens inside the input count. Gemini's promptTokenCount, for example, "includes the number of tokens in the cached content", so adding the cached count on top counts those tokens twice.

What to do today

  1. Store the raw usage object with each request, field names and all.
  2. For each provider you use, write down which fields contain which. A comment next to your cost function is enough.
  3. Test your cost code against a usage record whose answer you already know, like the one above.
  4. Recheck the provider's usage docs when you change models or providers, because the definitions can change.

Did you spot the bug before the answer? And if you call more than one provider, how do you keep their usage fields straight? Tell me in the comments.

Sources

Top comments (0)