The page that mattered never fired. Finance sent it instead, on a Thursday, in Slack: "Projected LLM spend this month is 41% over. Is that expected?" It was not expected. It took me four hours to find out why, and every one of those hours was a monitoring failure, not a model failure.
Here is the short version. One customer shipped a change that put an LLM call inside a polling loop. Their integration went from roughly 2,000 calls a day to about 140,000 calls a day, overnight. We paid for all of it. It ran for six days before anyone looked, because every dashboard I owned was aggregate, and the aggregate looked boring.
The hot take I keep repeating in incident reviews: cost blowups are a metering gap. The bill breaks down per tenant, and if you only watch the total you miss the tenant that is on fire.
What happened
The customer's feature polled a downstream job for completion. Someone refactored it and, instead of polling the job status, the loop re-issued the full LLM summarization call on every tick. Tick interval was five seconds. Do the math on a handful of concurrent sessions running most of the day and you land around 140k calls. Their code, our endpoint, our invoice.
None of it errored. That is the part that stings. Every one of those calls returned a clean 200. Latency was fine. Our SLO burn was zero. The system was, by every signal we alerted on, perfectly healthy. It was also setting money on fire at a steady rate.
What it cost
Our average cost per call for that feature is about $0.011 (short prompt, capped output). Normal footprint for this tenant:
- 2,000 calls/day
- about $22/day
After the loop bug:
- about 140,000 calls/day
- about $1,540/day
The delta is about $1,518 a day. It ran six days before finance flagged it. That is roughly $9,100 of spend we had no budget line for, on one tenant, on one feature, for output nobody consumed (they were throwing away 69 of every 70 responses).
For context, our whole platform's LLM spend was averaging around $1,800/day. This one tenant nearly doubled it. And I still did not see it, because our daily total already swung 30% on a normal week (weekend dips, batch backfills, onboarding spikes). A jump from $1,800 to $3,300 read like "big customer doing something legitimate." We had genuinely had 2x days before that were fine. So the signal sat inside the noise band, and the noise band was wide because I had never bothered to narrow it per tenant.
What it missed at scale
The scary math is not the $9,100. It is the slope. At $1,518/day this was a $45k/month leak from a single misbehaving integration, and we have hundreds of tenants. Any one of them can do this to us on any day. Our exposure was never "how expensive is the model." It was "how long can one runaway tenant run before a human happens to look at a bill." Six days, apparently. That is the actual SLO I had, and I had never written it down or defended it.
The fix
The refactor was easy: tag every call, attribute cost per tenant and per feature, and alert on spend rate the same way I alert on error rate. We already emitted tenant_id and feature on the request span. We just were not multiplying tokens by price and summing by tenant anywhere a human or an alert would see it.
The core alert is a spend-rate anomaly per tenant against that tenant's own recent baseline. Rolled up hourly, it looks like this:
-- hourly spend per tenant vs its trailing 7-day median
WITH hourly AS (
SELECT tenant_id,
date_trunc('hour', ts) AS hr,
SUM(calls * cost_per_call) AS spend
FROM llm_calls
WHERE ts > now() - interval '8 days'
GROUP BY 1, 2
),
baseline AS (
SELECT tenant_id,
percentile_cont(0.5) WITHIN GROUP (ORDER BY spend) AS med_spend
FROM hourly
WHERE hr < date_trunc('hour', now()) -- exclude current window
GROUP BY tenant_id
)
SELECT h.tenant_id, h.spend, b.med_spend,
round(h.spend / nullif(b.med_spend, 0), 1) AS ratio
FROM hourly h
JOIN baseline b USING (tenant_id)
WHERE h.hr = date_trunc('hour', now())
AND h.spend > b.med_spend * 3 -- 3x its own median
AND h.spend > 5 -- ignore trivial tenants
ORDER BY ratio DESC;
Two guards matter. Compare a tenant to itself, not to the fleet (a big customer is allowed to be big, they just are not allowed to suddenly 70x). And put a floor on absolute spend so you do not page at 3am because a tiny tenant went from 4 cents to 15. On our data, this alert would have fired inside the first hour of the loop bug, at a ratio near 70x, six days before finance did.
I also wired a projection: current month-to-date spend, extrapolated to month end, compared against budget. That is the check that would have caught it even if the per-tenant alert had a gap. It is cheap and it maps directly to the number the business actually cares about.
What I'd page on
This is the dashboard and alert set I now run for anything that spends money per request. If you take one thing, take the fact that none of these are latency or error signals. Cost needs its own alerts, the same way latency and errors do.
- Per-tenant spend rate. Page when any tenant's hourly spend exceeds 3x its trailing 7-day median, with a floor (we use $5/hour) so trivial tenants stay quiet. (The current hour is partial, so this catches a 60-70x spike fast and lags a subtle drift; pair it with the projection below.)
- Top-tenant concentration. Page if a single non-whitelisted tenant crosses 40% of total hourly spend. One customer owning the bill is an incident until proven otherwise.
- Calls/min per (tenant, feature). Page on step changes (greater than 5x hour over hour). This catches the loop before the dollars pile up, because request-count moves before the invoice does.
- Cost per call, p99, per feature. Page when it climbs. Rising cost per call with flat volume means prompt or context bloat, a different leak with the same symptom.
- Month-end spend projection vs budget. Page if the linear projection exceeds budget by more than 15%. This is the backstop that speaks finance's language.
- New entrants in the top-10 spenders. Not a page, a daily digest. A (tenant, feature) pair you have never seen in the top 10 is worth 30 seconds of a human's attention.
The uncomfortable lesson: a green dashboard is not proof that nothing is wrong. It is proof that nothing you decided to measure is wrong. I measured latency and errors because those page loudly and customers complain. Nobody complains about a bill that is quietly too high, so nobody measured it, so it ran for six days.
Top comments (0)