Why the Same AI Request Costs 10 More Some Days (Seeing It in Node)

#typescript #ai #observability #node

Book: AI That Answers
The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

A team I talked to had a summarise endpoint with a fixed prompt and a fixed
model. Same feature, same code, no deploys that week. Their per-request cost
moved by roughly an order of magnitude between quiet days and busy ones.

Nothing about "the AI got more expensive" explains that. Four things move cost
without your prompt changing, and each is measurable in Node, but only if you
record the right fields, which almost nobody does on day one.

1. Cache hits, which are invisible unless you log them

Prompt caching bills cached input differently from fresh input. A request that
hits a warm cache and an identical request that misses it are the same request
to you and different line items to the provider.

The usage object separates them:

const res = await client.messages.create(params);
const u = res.usage;

logger.info("model_call", {
  inputTokens: u.input_tokens,
  outputTokens: u.output_tokens,
  cacheWriteTokens: u.cache_creation_input_tokens ?? 0,
  cacheReadTokens: u.cache_read_input_tokens ?? 0,
});

Sum input_tokens + output_tokens into one number and you have thrown away
the distinction that explains your bill. Cost has to be computed per category:

export function costOf(model: string, u: Usage): number {
  const r = RATES[model];           // read from provider pricing; check current
  if (!r) throw new UnknownModel(model);
  return (
    u.input_tokens * r.input +
    u.output_tokens * r.output +
    (u.cache_creation_input_tokens ?? 0) * r.cacheWrite +
    (u.cache_read_input_tokens ?? 0) * r.cacheRead
  ) / 1_000_000;
}

Why it swings with traffic: caches have a TTL. At high traffic your prefix
stays warm and most requests read from it. Overnight, each request re-warms it
— paying the write rate instead of the read rate. Same code, different bill,
purely as a function of request spacing.

Track the ratio directly:

metrics.gauge("llm.cache_hit_ratio",
  (u.cache_read_input_tokens ?? 0) /
  Math.max(1, u.input_tokens + (u.cache_read_input_tokens ?? 0)));

A ratio that collapses at night is normal. One that collapses permanently
means something made your prefix unstable — see below.

2. A prefix that stops being stable

Caching requires a byte-identical prefix. Anything that varies early in the
prompt destroys every hit after it.

// quietly uncacheable
const system = `You are a helpful assistant. Today is ${new Date()}.`;

That timestamp changes per request, so nothing after it can be reused. A date
belongs in the user turn, after the stable block:

const system = STABLE_INSTRUCTIONS;                    // cacheable
const user = `Today is ${isoDate()}.\n\n${question}`;  // varies, and that is fine

Other prefix-breakers that look harmless: tool definitions built by iterating
a Set or object (order varies), a user's display name interpolated into the
system prompt, and a "context" object serialised with JSON.stringify whose
key order depends on how it was constructed.

3. Output length, which you do not control directly

max_tokens is a ceiling, not a setting. Actual output varies with input, and
output is billed at several times the input rate on most models, so output
length dominates cost far more than people expect.

A summarise endpoint fed a 200-word email and a 40-page thread produces very
different outputs from the same prompt. Aggregate "cost per request" hides
that completely.

Segment by input size:

const bucket = (n: number) =>
  n < 1_000 ? "s" : n < 10_000 ? "m" : n < 50_000 ? "l" : "xl";

logger.info("model_call", {
  inputBucket: bucket(u.input_tokens),
  outputTokens: u.output_tokens,
  costUsd: +costOf(model, u).toFixed(6),
});

Now "cost went up" becomes answerable: either the mix shifted toward larger
inputs, or cost within a bucket moved. Those have completely different fixes,
and without the bucket you cannot tell which you are looking at.

If output length is your driver, the lever is the prompt — asking for a
bounded form ("at most five bullet points") does more than lowering
max_tokens, which just truncates mid-sentence and wastes the whole call.

4. Retries and agent turns, counted as one request

Your metric probably says "requests". Your bill counts model calls. One
inbound HTTP request can be many calls: a retry, a validation repair turn, an
agent loop.

export const ledger = new AsyncLocalStorage<{ cost: number; calls: number }>();

export function meter(model: string, u: Usage) {
  const e = ledger.getStore();
  if (e) { e.cost += costOf(model, u); e.calls += 1; }
}

app.use((req, res, next) => {
  const e = { cost: 0, calls: 0 };
  res.on("finish", () => logger.info("request", {
    route: req.route?.path,
    calls: e.calls,
    costUsd: +e.cost.toFixed(6),
    status: res.statusCode,
  }));
  ledger.run(e, next);
});

calls per request is the field that catches this. A p99 of 7 calls where you
expected 1 means retries or an agent loop, and that alone can be the whole
tenfold difference — no per-token change required.

Putting it together

Five fields per call, four per request. That is the whole instrumentation:

// per model call
{ model, promptVersion, inputTokens, outputTokens,
  cacheReadTokens, cacheWriteTokens, costUsd }

// per inbound request
{ route, userId, calls, costUsd, status }

With those, the tenfold swing decomposes in one query each time: cache ratio
dropped, input mix shifted, output grew, or call count grew. It is always one
of the four.

Without them you have a monthly total and a theory.

The check to run first

Before optimising anything, sort last week's requests by cost and look at the
top one percent. In every case I have seen described, that tail is not "the
model is expensive" — it is a handful of requests with enormous inputs, or
runaway call counts.

Fixing the tail is usually a bounded input and a call cap, which are both
afternoon-sized changes. Fixing "the model is expensive" is not a task anyone
can start.

If this was useful

AI That Answers covers the cost side
of a first LLM app properly — what each token category means, why caching
changes the arithmetic, and building the accounting in before the invoice
rather than after.