TL;DR. Our LLM-judge eval suite had no cost ceiling. It ran the full judge over 1,200 cases on every CI trigger. On 08/07 a dependency bot opened 41 pull requests between 01:00 and 04:00, and our merge queue re-ran the whole suite on every push and every rebase: roughly 270 full runs at about $14.40 each. That one window cost $3,900 against a $1,730 monthly eval budget. No dashboard fired, because ours watched request rate and 5xx, not tokens or dollars. We found out from the invoice, because nothing we monitored watched spend. What I shipped was four guards under the suite: a pre-flight cost cap (estimate tokens with tiktoken, multiply by price, refuse the run if it would breach a daily ceiling), a result cache keyed on the candidate answer, sampling on non-main branches, and an alert on token-spend rate. Code is below.
I run reliability for a small ML platform team. We ship an LLM feature and gate it with an offline eval suite: 1,200 graded cases, each scored by a separate judge model against a rubric. Standard setup. It has caught real regressions. It also carried, for four months, a failure mode I built and did not see until it cost us most of a monthly budget in nine hours.
This is the writeup. Numbers are from our incident, rounded. The prices are the ones we paid at the time, not a benchmark.
How I found out
The first signal came in as an email.
At 09:40 the next morning, finance forwarded a cost-anomaly notice from our model provider: the prior day's spend on one API key was 68x its trailing average. My first reaction was that the anomaly detector was wrong. We had shipped nothing overnight. No incident channel, no 5xx, no latency alarm. Green board.
Then I read which key. It was ci-eval, not prod. I had no dashboard for ci-eval, because eval traffic never paged anyone, so I had never built one. I pulled the provider's usage export for that key and sorted by hour. Between 01:00 and 04:00 it had billed just over a billion tokens. A normal day for that key is about 14 million.
I checked the deploy log twice, expecting a runaway retry loop or a stuck worker hammering the API. There was neither. The requests were clean, sequential, and successful. Whatever had done this had done it deliberately, at 200 OK, which meant the provider was behaving correctly and the bug was in my own cost arithmetic.
That is how I learned my eval suite had a cost bug: about twelve hours late, from a finance email, in dollars rather than in the tokens or request counts I was actually watching.
What it cost
The arithmetic is the whole story, so here it is.
One judged case, at our rubric size, costs:
- input: about 2,600 tokens (rubric, question, candidate answer, reference answer)
- output: about 550 tokens (a score plus a short justification)
At the prices we paid ($2.50 per million input, $10.00 per million output), that is $0.0065 plus $0.0055, so roughly $0.012 per case. The full suite is 1,200 cases, so one run is 1,200 x $0.012 = $14.40. On a normal day CI fires it three or four times: a couple of merges to main, a manual rerun or two. Call it $57 a day, about $1,730 a month. That budget line had been flat for a quarter. So I stopped looking at it. That was mistake one.
On the night of 08/07, a dependency bot opened 41 pull requests in three hours. Each PR triggered the full suite on its first push. Each then got a lockfile follow-up commit, which triggered it again. Then the merge queue rebased each PR onto main before merging and ran the suite once more. Six to seven full runs per PR. About 270 runs total.
270 x $14.40 = $3,888. Just over a billion tokens. In one overnight window we spent 2.25x our entire monthly eval budget, and every single run passed. Nothing was broken. That is the part that still bothers me. The suite did exactly what I had configured it to do, 270 times, and nothing malfunctioned. The entire bill came from correct runs firing far more often than they ever should have.
Split the billion tokens and it maps straight back to the bill: about 842 million input tokens at $2.50 per million is $2,106, and about 178 million output tokens at $10.00 per million is $1,782. Sum $3,888. Output was 17% of the tokens and 46% of the cost, which is worth internalizing. On a rubric-heavy judge, the short justification you ask it to write is nearly half the spend. Trimming the justification moves the bill more than trimming the rubric does.
Root cause, and my part in it
Two settings did this. Both were mine.
Our CI ran the eval job on the GitHub synchronize event, which fires on every push to an open PR. I wrote that trigger in March so a PR's eval status stayed fresh as you pushed fixes. Reasonable for a human pushing three commits to one branch. Pathological for a bot pushing 41 branches with lockfile follow-ups.
The merge queue had "re-run required checks after rebase" turned on. I enabled that in May, after a stale check let a bad merge through. Also reasonable in isolation. Combined with the synchronize trigger and a bot storm, it meant every PR paid for the full suite at least three times before it even cleared the queue.
Neither setting is wrong on its own. Together, with no cost ceiling underneath them, they are a spend amplifier. I built both halves, two months apart, and never once did the multiplication. That combination is the actual root cause. The dependency bot was only the trigger: it exercised a gap I had left open, 41 times overnight.
What the dashboards missed
Three things were watching this system. All three were structurally blind to a cost spike.
The provider billing console. Aggregated by calendar day, updated on a lag of roughly 24 hours. At 04:00, while the spend was actually happening, it still showed the previous day's $58. It is accurate but far too slow for a spike that starts and finishes inside one night.
Our Grafana panels. Built on request count and error rate, scoped to the prod key. Two separate problems. First, count is the wrong unit. 270 eval runs is not a remarkable request volume; one prod minute makes more requests than that. The signal was never in the count, it was in tokens-per-request times price. Second, the panels were scoped to prod, and the runaway was on ci-eval, which had no panel at all.
The one alert we did have. A 5xx and rate-limit alarm on the provider. It never fired, because the provider was perfectly happy. It served all billion tokens without a single error. This is the trap: a cost runaway is not an error condition. The provider will keep billing at 200 OK and never surface an error for you to alert on.
So the real failure was not that an alert broke. It is that I had never expressed "dollars per hour on the eval key" as a number that anything watched. I monitored availability and errors, which are what page you at 3am. Spend was only ever reviewed in a monthly budget meeting, so nothing watched it continuously.
The fix
Four changes, ordered by how much they mattered.
- A pre-flight cost cap. Before any eval run, estimate its token cost with tiktoken, multiply by the price, and refuse to run if it would push the key past a daily ceiling. This is the guard that stops run number nine, not run number 270.
import tiktoken
# Judge pricing, USD per 1M tokens (from the provider's public pricing page).
PRICE_IN_PER_M = 2.50
PRICE_OUT_PER_M = 10.00
DAILY_BUDGET_USD = 120.00 # hard ceiling for the ci-eval key
def encoding_for(model: str):
try:
return tiktoken.encoding_for_model(model)
except KeyError:
# Newer judge models aren't in tiktoken's registry yet; fall back.
return tiktoken.get_encoding("o200k_base")
def projected_cost(prompts: list[str], est_output_tokens: int, model: str) -> float:
enc = encoding_for(model)
in_tokens = sum(len(enc.encode(p)) for p in prompts)
out_tokens = est_output_tokens * len(prompts)
return (in_tokens / 1e6) * PRICE_IN_PER_M + (out_tokens / 1e6) * PRICE_OUT_PER_M
def guard(prompts, est_output_tokens, model, spent_today):
run_cost = projected_cost(prompts, est_output_tokens, model)
if spent_today + run_cost > DAILY_BUDGET_USD:
raise SystemExit(
f"eval blocked: projected ${run_cost:,.2f} would push today's "
f"ci-eval spend past the ${DAILY_BUDGET_USD:,.2f} ceiling"
)
return run_cost
The edge case that bit us while testing this: encoding_for_model raises KeyError for judge models tiktoken has not shipped a mapping for yet. If you let that propagate, the guard crashes, and depending on how you wired it that either blocks all evals or (worse) gets swallowed by a broad except so the cap silently stops applying. Fall back to a known encoding: o200k_base for recent models, cl100k_base for older ones. The estimate is then approximate, which is fine here. This is a budget fuse, not an invoice. tiktoken is the actual tokenizer for this model family and it is open source (github.com/openai/tiktoken), so the estimate lands within a couple of percent of billed tokens in practice.
A result cache. Key the judge result on (rubric_version, case_id, sha256(candidate_answer)). On a dependency-bump PR, the model under test emits byte-identical answers, so the candidate hash does not change and the judge never runs twice on the same input. Rerunning 270 times over the same 1,200 unchanged candidates should have been about 1,200 judge calls and 322,800 cache hits: a hit rate near 99%. Cost with the cache in place: about $14, paid once.
Sampling off main. The full 1,200-case suite runs on merges to main and in the merge queue's final gate. Every other trigger (feature pushes, draft PRs, bot PRs) runs a fixed 10% stratified sample, 120 cases, about $1.44. You still catch gross regressions on a branch. You stop paying full freight for a lockfile bump.
Spend-rate alerting. This one gets its own section below, because it is the part I most want you to copy.
We shipped all four in two days. The cap and the cache did the heavy lifting. Together they take the worst case of a storm like this from $3,900 down to about $120, because once the ceiling is hit the cap simply stops starting new runs.
What it misses at scale
I am not going to pretend this is finished.
The pre-flight estimate uses a fixed est_output_tokens. Judges that write longer justifications on hard cases will be under-estimated, so the fuse is loosest on exactly the runs that cost the most. We set the constant to our 90th-percentile output length, which over-charges most runs on purpose. Safer, less precise. That is the trade, and it is deliberate.
The cache is only correct while the rubric and the judge model are pinned. Bump either one and every cache key changes, so the first run after a judge upgrade pays full price, and 270x full price if the same bot storm lands that same morning. The cap catches that case. The cache does not.
Sampling trades cost for coverage. A 10% branch sample will miss a regression living in the 90% you skipped, and you only see it at the main-merge gate. For us that is acceptable, because branch evals are advisory and the main gate is the one that blocks release. For a team that ships straight off branches, it is not, and they should not copy the 10%.
And a daily ceiling is a blunt instrument. Set it too low and you block legitimate work at 16:00 on a heavy release day. Set it too high and it would not have stopped this incident. We landed on 2x a normal day and accept that a genuinely large legitimate day needs a human to raise it by hand. The ongoing work here is tuning that threshold, not writing more code.
What I'd page on
The lesson was not "eval is expensive." It was that I had been monitoring the wrong quantities. I watched request counts and error rates; the failure only ever showed up in tokens and dollars, which nothing tracked. So here is the dashboard I built afterward, and the four things it pages on. Copy the metrics, not the numbers. The numbers are ours.
- Token-spend rate, per API key, per hour. The primary signal. Alert when any key crosses 3x its own trailing 7-day hourly median. This is the one that would have paged me at 01:20 instead of routing through finance at 09:40. Per-key matters: ci-eval and prod have different baselines and have to alarm independently, or the loud one masks the quiet one.
- Cost per eval run. Emit it as a metric from the harness itself, tagged by trigger (main, merge-queue, branch, bot). Page if a single run exceeds $20 (our full run is $14.40) or if runs-per-hour exceeds 12. A run-count spike is the earliest sign of a trigger loop, and it shows up before the dollars land in any billing export.
- Budget burn, daily and month-to-date. Track spend against the daily ceiling as a percentage: warn at 70%, page at 100% (by which point the cap has already blocked runs, so the page is really telling a human to decide whether to raise it). Track month-to-date against the $1,730 line and warn at 80% before the 20th, so an expensive first half of the month is visible while there is still time to react.
- Cache hit rate on the judge. A drop below 60% means either a rubric or model change invalidated the cache (expected, transient, no page) or the cache key is broken (not expected, page). On a normal week we sit near 40% from genuinely new candidates. A sudden collapse to single digits during a PR storm is the tell that the cache is not absorbing what it should, which is the early warning I did not have on 08/07.
None of these watch availability. Availability was fine the entire night. The provider returned 200 to every one of a billion tokens' worth of requests, and that was precisely the problem. Reliability for an LLM system has to include a spend SLO, and a spend SLO needs continuous monitoring rather than a monthly budget review.
If you run an LLM-judge eval suite and you cannot answer, from a dashboard, right now, "what does one run cost, and what stops it running a thousand times tonight," that is your next on-call ticket. We paid $3,900 to find that out.
Top comments (0)