Five ways your LLM cost tracking is lying to you

Your monthly OpenAI or Anthropic invoice tells you how much you spent. It doesn't tell you which feature spent it, which model, or why last Tuesday cost three times as much as Monday. So at some point you (or your team) will build a metering layer: wrap the client, read usage off the response, multiply by a price table, ship it to a database.

I did exactly that over the past few months while building an LLM observability service, and my numbers were wrong in five different ways before they were right. Every one of these failures was silent. No exception, no alert, just numbers that were quietly too low or too high. This is the list I wish someone had handed me.

Pitfall 1: Streaming responses quietly report zero tokens

OpenAI's Chat Completions API returns no usage data at all for streaming requests unless you pass stream_options: { include_usage: true }. No error, no warning. The stream just never contains token counts.

If your metering reads usage off the chunks, every streaming call gets recorded as 0 tokens, $0. And since chat UIs are almost always streaming, that's most of your traffic.

This one bit me twice in the same audit. First finding: all streaming calls in my own dashboard were $0. Second, nastier finding: I had a budget-gate feature that blocks calls once spend crosses a limit, and it waved every streaming call straight through — because as far as it could tell, streaming was free.

The fix is to inject the option in your wrapper when the caller didn't set it:

let injected = false;
if (params.stream && params.stream_options?.include_usage === undefined) {
  params = {
    ...params,
    stream_options: { ...params.stream_options, include_usage: true },
  };
  injected = true;
}

But there's a trap inside the fix. With include_usage on, OpenAI appends one extra chunk at the end of the stream that carries usage and has an empty choices array. Any downstream code that does chunk.choices[0].delta — which is most example code on the internet — will throw on it. So if you injected the option, you also have to swallow that chunk before it reaches the application:

for await (const chunk of upstream) {
  if (chunk.usage) observedUsage = chunk.usage;
  // Usage-only chunk we asked for: record it, don't leak it.
  if (injected && chunk.usage && (chunk.choices?.length ?? 0) === 0) continue;
  yield chunk;
}

If the caller set include_usage themselves, pass the chunk through — they asked for it.

Pitfall 2: "Cached tokens" means something different on every provider

Prompt caching is where the naive formula prompt_tokens × input_rate falls apart, and it fails in opposite directions depending on the provider.

OpenAI: prompt_tokens includes cached reads, and the cached portion is billed at a discount (half the input rate, as of this writing). The cached count is tucked away in usage.prompt_tokens_details.cached_tokens. If you ignore it, you over-count, and the more effective your prompt caching is, the more wrong your numbers get. Which is perverse: the caching you added to save money makes your dashboard say you're spending more.

Anthropic: the opposite layout. input_tokens excludes cache activity; cache reads and writes arrive in separate fields (cache_read_input_tokens, cache_creation_input_tokens). Reads are heavily discounted, but writes cost more than the base input rate (1.25× for the default 5-minute TTL, at current rates). Ignore those fields and you under-count.

Gemini has its own context-caching discount on top of that. There is no shortcut here: you have to normalize per provider.

What worked for me was normalizing everything into three buckets — uncached input, cached reads, cached writes — and pricing each bucket with a per-provider multiplier:

const uncached = Math.max(0, promptTokens - cachedRead - cachedWrite);
const inputCost =
  uncached   * rate.input +
  cachedRead * rate.input * mult.read +   // e.g. 0.5 (OpenAI), 0.1 (Anthropic)
  cachedWrite * rate.input * mult.write;  // e.g. 1.25 (Anthropic 5-min cache write)
const cost = inputCost + completionTokens * rate.output;

The multipliers change (Gemini cut its cache-read rate substantially in 2026), so keep them in one table you can update, not scattered through the code.

Pitfall 3: Serverless drops your records on the floor

Cloudflare Workers, AWS Lambda, and edge runtimes kill outstanding work the moment your handler returns. If your metering sends records fire-and-forget, or buffers them for a batched flush that fires on a size threshold or an idle timer, a short-lived handler exits before either trigger fires. The records evaporate. No error anywhere, because the process that would have logged the error is already gone.

Of everything in this post, this was the failure that scared me most. Not because the data loss was large, but because the metering itself failed and nothing told me. A monitoring system that silently stops monitoring is worse than no monitoring — you still trust it.

Two fixes, use either:

export default {
  async fetch(req: Request, env: Env, ctx: ExecutionContext) {
    const client = wrap(new OpenAI({ apiKey: env.OPENAI_API_KEY }));
    try {
      return await handleRequest(req, client);
    } finally {
      // Keeps the runtime alive past the response without delaying it:
      ctx.waitUntil(flushClient(client));
      // ...or, if you don't have ctx: await flushClient(client);
    }
  },
};

The point is that your metering library must expose an awaitable flush. If you're evaluating an SDK for serverless use and it doesn't document one, assume it loses data there.

Pitfall 4: A cancelled stream bills you and records nothing

Users close tabs mid-stream. Code breaks out of the loop after finding what it needed. The provider bills you for every token generated up to the disconnect. But if your recording logic sits after the consumption loop, it never runs. In JavaScript, breaking out of a for await calls .return() on the generator, and everything after the loop body is skipped.

So: real money spent, zero recorded. Same silent-under-count family as pitfall 1.

The pattern that covers all three exits (completion, error, early break) is a finally with an idempotent record call:

let recorded = false;
const recordOnce = () => {
  if (recorded) return;
  recorded = true;
  save(observedSoFar); // partial usage is better than no record
};

try {
  for await (const chunk of upstream) {
    observe(chunk);
    yield chunk;
  }
  recordOnce();
} catch (err) {
  recorded = true;
  saveErrorRecord(err, observedSoFar);
  throw err;
} finally {
  recordOnce(); // early break / abandoned iterator lands here
}

One more wrinkle a code review caught in mine: openai-node's Stream treats controller.abort() as a normal end of iteration. The loop exits cleanly, so without an extra check you'd record a truncated response as a successful, complete one. Check stream.controller.signal.aborted before marking the record as completed.

If you can afford it, there's a stronger version: tee() the stream and drain an observation branch independently of the user-facing branch. Then even an early break on the user side leaves the observation side to run to completion and capture the final usage chunk. Costs some buffering memory; worth it for accuracy.

Pitfall 5: Your price table started rotting the day you wrote it

Hardcoded per-model rates go stale every time a provider ships a model. The dangerous failure mode isn't the staleness itself. It's what your code does when it looks up a model it doesn't know. If the answer is "silently return $0", then the day someone switches to a new model, your cost graph drops to zero and everyone celebrates the wrong thing.

The floor for handling this:

const entry = lookupPricing(provider, model);
if (!entry) {
  warnOnce(
    `unknown ${provider} model "${model}" — pricing returned 0. ` +
    `Update the pricing table.`,
  );
  return 0;
}

You still record $0 (there's no honest number to put there), but you warn, loudly and once per model, so a human finds out the table needs updating. A silent zero and a zero with a warning look identical in the database. Operationally they are very different things.

(Warn once per unknown model, not per call. The first version of my warning fired on every call and turned the logs into noise nobody read.)

The checklist

If you're building or buying an LLM metering layer, walk through these:

• [ ] Streaming usage: is include_usage handled, and is the usage-only chunk kept away from choices[0] consumers?
• [ ] Cache tokens: does the cost math know that OpenAI includes cached reads in prompt_tokens while Anthropic reports cache activity in separate fields?
• [ ] Serverless: is there an awaitable flush, and is it wired into ctx.waitUntil or a finally?
• [ ] Interrupted streams: do early breaks, aborts, and errors still produce a partial record?
• [ ] Unknown models: warning plus explicit $0, never a silent $0?

None of these is hard on its own. The problem is that they all fail silently, so you tend to find them one production surprise at a time.

Disclosure: I built Argosvix, an LLM observability service whose SDK handles all five of these. Everything above works fine self-rolled too.

Comments

[Xiao Man]

Pitfall 3 hit me in a completely different context — not cost tracking, but agent telemetry. Same root cause: serverless runtime exits before the observation branch finishes. The "monitoring that silently stops monitoring" line is the scariest sentence in this post.

Your tee()-the-stream pattern for pitfall 4 is the same move that agent quality systems need. The user-facing branch can abort, but the observation branch runs to completion and captures the evidence. That separation between "what the consumer sees" and "what the system records" is where most silent failures hide.

One thing I'd add: pitfall 5 (price table rot) has an interesting interaction with provider-level behavioral drift. Even if the price stays the same, the model behind the same endpoint can change — tokenization shifts, context handling evolves, effective cost-per-useful-output moves. The price table tells you the dollar cost but not the value cost. Worth tracking both.