Your Bill Doubled Overnight: A Triage Runbook

#ai #debugging #llm #machinelearning

An LLM bill that doubles overnight has one of about eight causes, and the fastest route to it is not reading code. It is six queries over your request log, run in order, each of which eliminates a branch. The first one takes thirty seconds and settles whether you are looking for more requests or dearer ones.

Before the queries: stop the bleeding

If spend is still climbing while you investigate, put a ceiling on it first. A provider-side spending limit, a lowered rate limit on your own gateway, or disabling the newest feature flag all buy you time, and none of them require knowing the cause. Diagnosis is cheaper when the meter is not running.

Resist the urge to change several things at once to make it stop. If you disable three suspects simultaneously and the spend falls, you have solved the incident and learned nothing, and it will return.

What you need logged

The runbook assumes one row per request. If you do not have this, building it is the first fix, and it is a day of work that pays for itself the first time this happens.

CREATE TABLE llm_requests (
  ts                timestamptz NOT NULL,
  request_id        text,
  model             text NOT NULL,   -- from the RESPONSE, the resolved one
  route             text,            -- which feature or endpoint
  caller            text,            -- service, job, or user id
  tenant            text,            -- customer, if multi-tenant
  prompt_tokens     int  NOT NULL,
  cached_tokens     int,             -- prompt tokens served from cache
  completion_tokens int  NOT NULL,
  reasoning_tokens  int,
  cost_usd          numeric(12,6),   -- computed at write time
  status            int,
  attempt           int,             -- 1 for the first try, 2+ for retries
  duration_ms       int
);

Two columns do disproportionate work. attempt is what makes a retry storm visible instead of looking like organic traffic. And model taken from the response rather than the request is what makes an alias move visible — the request said one thing and the provider served another.

The six queries, in order

Volume or unit cost? Everything downstream depends on this answer, and it is one query.
```
SELECT date_trunc('day', ts) AS day,
       count(*)                          AS requests,
       sum(cost_usd)                     AS spend,
       sum(cost_usd) / count(*)          AS cost_per_request
FROM llm_requests
WHERE ts > now() - interval '14 days'
GROUP BY 1 ORDER BY 1;
```
Requests doubled and cost per request is flat: a traffic or loop problem, go to step 4. Requests flat and cost per request doubled: something about each call changed, go to step 2. Both moved: usually two causes, and you should chase them separately.
Which model? A shift in the mix is the commonest cause of a unit-cost jump, and it frequently happens without a deploy.
```
SELECT date_trunc('day', ts) AS day, model,
       count(*), sum(cost_usd) AS spend
FROM llm_requests
WHERE ts > now() - interval '14 days'
GROUP BY 1, 2
ORDER BY 1, 4 DESC;
```
A model name that appears for the first time on the day the bill moved is your answer. So is a familiar name whose share jumped — a fallback route that started firing, or an alias that resolved somewhere new.
Which token type? Input, output, reasoning and cached tokens have different prices, and the ratios tell you what changed.
```
SELECT date_trunc('day', ts) AS day,
       avg(prompt_tokens)                              AS in_avg,
       avg(completion_tokens)                          AS out_avg,
       avg(reasoning_tokens)                           AS reasoning_avg,
       avg(cached_tokens::numeric / nullif(prompt_tokens,0)) AS cache_hit_rate
FROM llm_requests
WHERE ts > now() - interval '14 days'
GROUP BY 1 ORDER BY 1;
```
Input grew: the prompt got bigger — more retrieved documents, longer history, more tools. Output grew: verbosity or a raised max_tokens. Reasoning grew: an effort setting moved. Cache hit rate collapsed: something made the prefix vary, and this one is invisible in every other view.

Which caller?

SELECT route, caller,
       count(*) AS n,
       sum(cost_usd) AS spend,
       sum(cost_usd) FILTER (WHERE ts > now() - interval '2 days')
         - sum(cost_usd) FILTER (WHERE ts BETWEEN now() - interval '4 days'
                                            AND now() - interval '2 days')
         AS delta
FROM llm_requests
WHERE ts > now() - interval '4 days'
GROUP BY 1, 2 ORDER BY delta DESC NULLS LAST LIMIT 20;

Sorting by the change rather than by the total is the point. The biggest spender is usually the biggest spender every week; the one that moved is the one you are looking for.

Retries and loops.

SELECT date_trunc('hour', ts) AS hour,
       count(*) FILTER (WHERE attempt = 1) AS first_tries,
       count(*) FILTER (WHERE attempt > 1) AS retries,
       sum(cost_usd) FILTER (WHERE attempt > 1) AS retry_spend
FROM llm_requests
WHERE ts > now() - interval '3 days'
GROUP BY 1 ORDER BY 1;

A retry share that is normally 1% and is now 30% is a retry storm, and the underlying trigger is usually a provider error that your code treats as retryable when it is not. For agents, count steps per session as well: a session with two hundred steps is a loop, not a user.

The expensive tail. Cost is almost always concentrated, and the top 1% of requests frequently is the entire story.
```
SELECT request_id, route, caller, model,
       prompt_tokens, completion_tokens, cost_usd
FROM llm_requests
WHERE ts > now() - interval '2 days'
ORDER BY cost_usd DESC
LIMIT 50;
```
Read the actual requests behind the top rows. This is the step that most often ends the investigation, because a pathological request is recognisable on sight.

What it usually turns out to be

A retry storm. An upstream blip, retries with no cap and no jitter, and every retry paying full price for a long prompt.
An agent loop with no step limit. One session producing hundreds of calls because a tool keeps failing and the model keeps trying. Cost per session is the metric that catches this; stopping conditions and agent cost control cover the prevention.
A prompt that grew. Retrieval top-k raised from 5 to 10, a tool added, unbounded conversation history. Small change, multiplied by every request.
Prompt caching stopped hitting. A timestamp, a session ID or a shuffled tool order in the cached prefix means every request is a cache miss. The bill can move sharply with no other visible change — see what prompt caching saves and ordering context for the cache.
A model or effort change. An alias moved, a fallback fired, or someone raised a reasoning effort setting for quality and did not model the cost.
A backfill or a batch job. Someone reprocessed a corpus. Legitimate, one-off, and worth confirming so you stop looking.
A leaked key. Rarer than people fear, but check: traffic from unfamiliar callers, unusual hours, or a request mix that does not match any of your features. If so, rotate immediately rather than investigating further.

The arithmetic that bounds the answer

Before accepting any explanation, check it is arithmetically possible. The maximum a route can spend is bounded by its request count times its worst-case tokens times the price.

Worked example. Assume 4 USD per million input tokens and
16 USD per million output tokens, and a route doing 50,000 requests/day:

  ceiling per request = (8,000 input  x 4 / 1e6)
                      + (2,000 output x 16 / 1e6)
                      = 0.032 + 0.032 = 0.064 USD

  ceiling per day     = 50,000 x 0.064 = 3,200 USD

If the bill for that route is 9,000 USD, the request count is wrong,
the token counts are wrong, or requests you are not logging exist.
The discrepancy IS the finding.

Those prices are placeholders for the arithmetic, not quotations. Substitute the current published rates for the models you actually call; the method is what transfers, not the numbers.

What to change so it cannot recur

A hard spending limit at the provider or the gateway. Not an alert — a limit. An alert relies on somebody being awake.
A per-session and per-tenant cost cap. Cost per session is the single most useful derived metric here, because almost every runaway is one session behaving unlike the others.
A daily anomaly check on cost per request. The ratio moves before the total does, which buys a day.
A retry budget, not just a retry count. Cap total retries per minute across the process, so one bad minute cannot become one bad hour.
Log attempt and the resolved model. If step 2 or step 5 was unanswerable this time, fix that before you close the incident.

The controls themselves are covered in more depth in budget controls, cost attribution and denial of wallet.

Steps 2 and 4 both depend on per-request cost being recorded at the moment of the call rather than reconstructed from a monthly invoice. Multigrid computes cost per request and attributes it to a key and a label, which is what makes the “which caller moved” query answerable on the day rather than at the end of the month.