DEV Community: Jasmine Park

Trace sampling at 10 percent: a $238 invoice against my $82 estimate

Jasmine Park — Tue, 11 Aug 2026 16:26:53 +0000

A product manager asked what one feature had cost us in April. I pulled the number from our trace backend, divided by the sample rate, and gave her a figure just over eighty dollars. She came back a week later with the provider invoice broken out by API key, which we had wired per feature the previous year. That feature had cost $238.

Mine was the wrong one, by nearly 3x. The invoice is ground truth and my estimate was extrapolated from 10 percent of requests, uniformly sampled, decided at the head of each trace. For that feature, in that month, 10 percent was not enough.

I had set that sample rate myself, eighteen months earlier, for a good reason. Full-fidelity traces on 4.2 million LLM requests a month were not affordable. Ten percent was a round number that made the bill work. What I had never done is ask what 10 percent uniform sampling does to the specific questions we point observability at, which turn out to be almost entirely questions about tails.

Why the estimate was that bad

LLM request cost is heavy-tailed and my sampled estimator did not care.

For a high-volume feature the estimate is fine. Six hundred thousand requests a month, 10 percent kept, the mean of the sample converges and the extrapolation is within a couple of percent. The feature in question did about 9,000 requests in April, so the sample held roughly 900 of them. Its cost distribution had two populations: 36 requests that dragged a full contract into context, around 400,000 input tokens on a premium tier, at about $6.00 each, and 8,964 that cost around a quarter of a cent. Two hundred and sixteen dollars of document calls, twenty-two dollars of everything else.

The sample contained one of the 36. Scaled up by ten, that produced my $82. The expected count was 3.6, and the chance of drawing exactly one is 9.0 percent, so this was not a freak month. Drawing a 6 sits at 8.3 percent, near enough the same, and it would have handed the same PM $382, sixty percent above the invoice, with exactly the same confidence. The estimator overshoots as readily as it undershoots. April happened to undershoot.

The arithmetic is worth doing once. Per-request cost across those two populations has a mean near 2.6 cents and a standard deviation near 38 cents, so the coefficient of variation is about 14. Relative standard error on a scaled-up total is roughly that CV divided by the square root of the sampled count, which at n equals 900 is around 48 percent. In dollars, the standard error on the monthly total was about $114 against a true total of $238. My miss was $156, which is under one and a half standard errors. That is the part that should worry anyone reading this: a 3x error was not an anomaly. It was an ordinary draw from a number I had been reporting as a fact.

Nothing about this is specific to LLMs. It is the ordinary behavior of a ratio estimator on a skewed population. What is specific to LLMs is that the skew is enormous and the thing we most want to count sits in the tail.

Head sampling decides before it knows anything

Our OpenTelemetry collector made the keep-or-drop decision on the first span, hashing the trace ID. That is what head-based sampling is, and it is why it is cheap.

It also means the decision cannot depend on cost, latency, token count, or whether the request errored, because none of those exist yet. The six-dollar request and the quarter-cent request are equally likely to be kept. That is fair in the sense a coin is fair, and it is exactly backwards for an investigation. I am not looking for a representative request. I am looking for the one that cost six dollars or took forty seconds, and head sampling keeps those at exactly the same rate as everything else.

The third problem followed from the second and was the one that actually made people stop trusting the system. Support escalates a bad answer with a request ID. Nine times in ten the trace was gone. We measured it at 91 percent rather than the 90 the sample rate implies, and I never chased the extra point. We had built an observability platform with a one-in-ten chance of answering the only question anyone ever asked it, and the team's rational response was to stop asking.

What it costs, honestly

The reason nobody had revisited the sample rate is that everyone assumed the alternative was 100 percent traces at 30-day retention, which is expensive and obviously so. That framing is the mistake. Three different questions were being served by one pipeline, and they have different shapes.

At 4.2M requests a month, our mean full trace with prompt and completion bodies is 34 KB. That is 143 GB a month at full fidelity. A metrics-only record, no payloads, holding request ID, feature, model, input and output token counts, latency, status and a cost field, is 210 bytes. At 100 percent that is under a gigabyte.

10% uniform, head-based (what we had): traces per month 420,000, ingested per month 14.3 GB, retention 30 days, error traces kept 10%, requests over $0.05 kept 10%
Metrics census, no payloads: traces per month n/a, ingested per month 0.9 GB, retention 13 months, error traces kept n/a, requests over $0.05 kept n/a
Full fidelity, short retention: traces per month 4,200,000, ingested per month 143 GB (9.5 GB standing), retention 48 hours, error traces kept 100%, requests over $0.05 kept 100%
Tail-based keep-rules: traces per month 248,000, ingested per month 16.0 GB, retention 30 days, error traces kept 100%, requests over $0.05 kept 100%

The census answers every cost, volume, latency and error-rate question by counting rather than estimating. The April question now has an exact answer and it reconciles against the invoice to within rounding. This is the path that removed the problem this post opens with, and it costs less than a gigabyte a month.

The 48-hour tier is the one I would argue hardest for, because it is the one I nearly missed. Escalations arrive fast. Ours had a median age of about four hours from the user hitting the problem to a request ID landing in our queue, and the ninetieth percentile was inside a day. Holding every trace for 48 hours in cheap storage is 143 GB of ingest a month but only about 9.5 GB standing, because everything expires. Trace-lookup on an escalation went from 9 percent to essentially 100 percent inside the window, and the residual is escalations older than two days, which fall back to the rules below.

The obvious question, and the one I would ask of anyone else's diagram, is why both trace tiers exist. If you already hold 100 percent of completed traces for 48 hours, you can evaluate the keep-rules as a scheduled query over data you already have and extend the TTL on whatever matches, which deletes the in-flight sampler and everything downstream of it in this post. The reason we did not is that the two tiers are different storage. The 48-hour tier is flat unindexed objects, which is why it is cheap; the 30-day tier lives in the queryable backend. Promoting a trace between them after the fact means paying the indexing ingest a second time, per trace, forever. Deciding in flight costs a buffer. Deciding later costs a second write into the expensive store, and the second one is the larger bill at our volume. If your long-term store is cheap to write, invert this without hesitation.

The keep-rules decide what earns 30-day retention, after the trace completes: all errors, 0.7 percent of requests; everything over 8 seconds, 1.1 percent; everything over 5 cents, 2.3 percent; plus a 3 percent uniform baseline so "normal" stays visible. Those overlap, and the union is 5.9 percent.

The row I got wrong in my first version of this analysis is the last one, and I want to show the error rather than the corrected table alone, because it is the same error as the one in the opening. I originally costed tail-based at 8.4 GB by multiplying 248,000 traces by the 34 KB corpus mean. But the rules deliberately select long multi-step agent runs and large-document requests, so a rule-kept trace is not an average trace. Measured, they mean 96 KB. Costing the baseline at 34 KB and the rule-selected 2.9 percent at 96 KB gives 16.0 GB, not 8.4.

So tail-based is not cheaper than the 10 percent uniform sampling it replaced. It is about 12 percent more expensive on the 30-day tier, and it takes error and expensive-request coverage from 10 percent to 100. I had briefly believed the better story, which was that we would pay less and see more, and I believed it because I had used a mean trace size on a population selected for being large. Same mistake, one layer down.

And 12 percent is the number for one row, so I should give the aggregate before anyone else does. Across all four paths this pipeline ingests 0.9 plus 143 plus 16.0, about 160 GB a month against the 14.3 it replaced. That is 11x the bytes through the collector and the export path. Standing bytes are the gentler number, roughly 37 GB against 14.3, because the big tier expires in two days. That is the actual shape of the trade: storage at rest stays close to rounding error, and what genuinely went up 11x is the write path, so what you are really buying is collector capacity and egress. Size that before you turn it on. We did not, and our collector pool was the first thing to fall over.

The part that bit us

Tail-based sampling has to buffer. The collector holds the spans of an in-flight trace until the trace ends, then decides, and that buffer has two limits: a time limit and a count limit. I assumed both were the problem. Only one was.

Start with the limit that was not the problem. I assumed we were overflowing the trace-count ceiling and that adding capacity would fix it. The arithmetic says otherwise: 4.2M requests a month is 1.6 traces a second, and that is a ceiling rather than a count, because most of our traffic is one call per trace while an agent session bundles several into one. So at a 30-second decision wait we held at most about 49 traces in flight against a num_traces default of 50,000. You cannot fill a fifty-thousand-trace ceiling with forty-nine traces. Nothing was overflowing, and at this volume nothing on this pipeline has ever been dropped for space.

The real defect is in what a policy sees when the timer fires. The latency policy measures a trace's duration as the earliest span start to the latest span end among the spans it is holding at that moment. For a trace that has finished, that is the trace's duration, which is what you want. For a trace still running, it is however long the collector has been holding it, which is the decision wait.

So at decision_wait: 30s, every session still open at thirty seconds reported a duration of about thirty seconds, cleared our 8-second threshold, and was kept. Not because it was slow. Because it was still going. The keep-rule was measuring my own timeout. And that kept nothing extra. A session still open at thirty seconds runs past thirty seconds, so it is over the 8-second threshold whenever you evaluate it, and the slow rule keeps it either way. All the early decision changed was when.

What it changed instead is which rule did the keeping, and that is the real defect. Every policy is evaluated against the spans the processor is holding at decision time. status_code scans those spans for an error status; numeric_attribute scans them for the cost attribute. Decide at thirty seconds and neither can see an error that fires at minute four, or the cost of the eleven model calls that had not happened yet. Those sessions were retained regardless, because the latency rule had already caught them for being long.

So on every session over thirty seconds, two of my four keep-rules were dead, and the pipeline looked completely healthy because a third rule was quietly covering for them. Nothing in the retention numbers moves, which is exactly why it survives review. It only surfaces the day somebody tightens the latency threshold, or drops the rule because the traces it keeps look redundant, and error and cost coverage on long sessions falls to the 3 percent baseline without a single alert firing.

The fix is to stop deciding on unfinished traces. The processor can trigger the decision off the root span closing instead of off a fixed timer, which is what tail sampling was supposed to be doing in the first place.

processors:
  tail_sampling:
    decision_wait: 600s                    # a ceiling, longer than any session we run
    decision_wait_after_root_received: 5s  # decide 5s AFTER the session's root span closes, so the
                                           # latency rule sees a finished trace instead of my timeout
    num_traces: 20000                      # ~1.6 traces/sec x ~305s = ~490 in flight. headroom, not a
                                           # real constraint at our volume. size from YOUR arrival rate.
    policies:
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow
        type: latency
        latency: {threshold_ms: 8000}
      - name: expensive
        type: numeric_attribute
        numeric_attribute: {key: llm.cost_micros, min_value: 50000}   # 50,000 micros = $0.05.
                                         # llm.cost_micros is ours: the GenAI semantic conventions
                                         # define token-usage attributes and no cost attribute at all.
      - name: baseline
        type: probabilistic
        probabilistic: {sampling_percentage: 3}

The decision now lands a few seconds after the session actually ends, on a complete trace, so every rule is evaluated against the whole thing and the error and cost policies can see what they were written to see. Buffer occupancy goes to roughly 490 traces, which is still nothing against 20,000, but work that out for your own arrival rate before copying either number: at a hundred times our volume the same config is the difference between a comfortable pool and an out-of-memory loop.

The other thing this pipeline needs, and the first thing I look for in anyone else's setup, is that every span of a trace reaches the same collector instance. Tail sampling cannot decide on a trace it only partly holds, and the processor's own documentation is explicit that all spans for a given trace must arrive at the same instance. Behind a normal load balancer they do not. That means a two-layer deployment: a front tier running the load-balancing exporter, routing by trace ID, and a back tier running the tail sampler. We ran one tier for most of a sprint and could not work out why the error policy kept firing on fragments.

We also stopped relying on the trace being whole. Whatever the sampler does, the census record for that request still exists on path one, keyed by request ID, so cost and latency questions never depend on a trace surviving intact. That separation is what makes the rest of this survivable.

What I'd page on

Sampled-total drift: alert when a feature's trace-derived monthly cost and its provider-invoice cost differ by more than 15 percent. It is the check that would have caught this eighteen months earlier, and it costs one scheduled query.

Late-span age, as a histogram, not the dropped-too-early counter. This is the correction I most want to pass on. sampling_trace_dropped_too_early is a trace-count overflow metric, so at our volume it reads zero permanently and I spent a week treating that zero as reassurance. The late-span age histogram records how long after a decision the rest of a trace turns up, which tells you how far past decision_wait your sessions actually run, which is the number you needed in order to set it. Alert on the upper percentiles moving, and read the histogram when you tune.

Split-trace rate: spans arriving at a collector instance for a trace whose root landed elsewhere. Non-zero means your load-balancing tier is misrouting and every keep-rule below it is deciding on fragments.

And a warn rather than a page: trace-lookup miss rate on support escalations, bucketed by escalation age. Inside 48 hours it should be near zero now. Outside 48 hours it will not be, and knowing where that line sits is more useful than a single number. Before any of this, ours was 91 percent and nobody had measured it.

A 4% cache hit rate was costing us money. Here is the arithmetic I should have run first.

Jasmine Park — Thu, 06 Aug 2026 17:12:38 +0000

We turned on prompt caching for our document-QA service and the invoice went up.

Not dramatically. About 5%. Enough that I assumed it was traffic growth for the first two weeks, and it was not.

This post is the calculation I now run before anyone staffs a caching project. It takes ten minutes, it needs three numbers you already have, and in our case it would have told us what hit rate the cache had to clear before it was worth anything at all, and that the best possible outcome was smaller than anyone in the room believed.

What our contract actually charges

Two terms matter and only one of them gets discussed.

A cache read costs a tenth of the normal input rate. That is the term everybody quotes.

A cache write costs 1.25 times the normal input rate, and a miss writes the entry. So a request that misses pays a surcharge on its prefix for the privilege of storing it for the next one. Not every contract has this term. Ours does, and if yours does not, you should be able to point at the rate schedule that says so.

Put those together and there is a hit rate below which caching is a net loss:

h x 0.10 + (1 - h) x 1.25 = 1

which solves to h = 21.7%. Below a 21.7% hit rate, our cache was more expensive than no cache at all.

We shipped at 4%.

Why 4% and not zero

The system prompt opened with Current date and time: 2026-08-01T09:41:07.284Z, injected per request so the service could answer questions about document deadlines. Prefix caching keys on a prefix, so one variable field at position nine invalidates every token after it. Roughly 1,900 tokens of stable system prompt and tool schemas were being priced as novel, more than 40,000 times a day.

The residual 4% was our own retry logic. A retry re-sends the prompt object the client already built, timestamp string and all, so a retried request is byte-identical to its first attempt and lands on a warm prefix, provided the first attempt got far enough to write one. A retry after a 429 rejected at the edge finds nothing warm, because the attempt it is retrying never processed the prefix. That is why the hit rate sat a little below our retry rate for the month, which ran just under 5%, rather than on top of it. Nothing else was hitting the cache at all.

Moving the timestamp below the tool schemas, into the part of the message that was already variable, took the hit rate to 71%. One line of code.

It did not take it to 100%, and the gap is worth naming because it is the part you cannot engineer away either. Cache entries expire, our traffic has quiet stretches overnight and at weekends, and a request that arrives on a cold shard pays full price whatever the prompt looks like. Seventy-one percent was roughly what the traffic shape allowed.

The fix, and the most it could ever have been worth

A request looks like this for us:

System prompt + tool schemas: tokens 1,900, cacheable yes
Retrieved context: tokens 4,600, cacheable no, different every request
User turn: tokens 120, cacheable no
Total input: tokens 6,620, cacheable 28.7% of it
Output: tokens 350, cacheable no

Three states, all measured against a bill with no caching at all:

4% (where we shipped): effective input price per million $3.18, bill vs no cache 4.8% higher
71% (after the fix): effective input price per million $2.51, bill vs no cache 13.4% lower
100% (unreachable): effective input price per million $2.23, bill vs no cache 21.3% lower

The invoice moved 17.4% between the first row and the second, which is the number our dashboard celebrated. It is larger than the 13.4% we were actually saving, because the starting state was worse than not caching at all. The 17.4% is what the invoice did. The 13.4% is what the work was worth. A business case takes the second.

The row that matters most is the third. A perfect cache saves us 21.3%, and that was knowable on day zero. That number should have opened the planning conversation. We computed it at the end instead.

The ten-minute calculation

Three factors, multiplied:

Cacheable prefix, as a share of input tokens. Ours: 1,900 of 6,620, so 28.7%. Not tokens you feel are repetitive. Tokens that are byte-identical and sit before the first variable byte.

The discount on a cache read. Ours is 90% off, so 0.90.

Input spend as a share of the total bill. Output tokens never cache, and ours are billed at four times the input rate, which makes output 17.5% of our baseline invoice. So input is 82.5% of it, and that is the fraction the caching work can touch at all.

0.287 x 0.90 x 0.825 = 21.3%

That is the ceiling. Nothing in the caching work gets past it, because all three factors are properties of the prompt and the contract rather than of the implementation. If you want more, you have to change one of them: make the stable prefix a larger share of the request, negotiate the read rate, or shift the input/output balance. Adding cache infrastructure does none of those.

For what it is worth, the sprint did land. $1,059 a day became $875, so the invoice fell about $5,500 a month, of which about $4,100 is saving measured against never having cached at all. I would still have staffed it. I would have described it differently, and I would not have had to walk anyone back from a number I had implied but never computed.

The failure I had seen before, and why this one is different

I wrote a post a few weeks ago about a response cache running a 90% hit rate while the bill climbed, where the problem was that the hit rate counted requests and the requests it was hitting were the cheap ones (the write-up is here). Same headline metric, different failure, and the difference is worth spelling out because I initially conflated the two.

I should also own the embarrassing part. A timestamp in the system prompt was on the list of key-busters in that post. I found it in one cache, fixed it there, and shipped it into another one.

The distinction that matters: that post was about which requests hit. This one is about how much of a request can hit. Our prefix is 28.7% of the input on every single request, including the ones that hit perfectly, so a flawless cache still leaves 71.3% of the input tokens at full price. The traffic mix has no say in that; the prompt does. We fixed the first cache and the second one still had the same field sitting in it.

What I'd page on

Cached-token share of input tokens, dropping more than 10 points below its 7-day median. This is cached_input_tokens / total_input_tokens from the usage payload, not requests. Ours sits at 20.4% at a 71% hit rate, so the alert fires below about 10%. It catches someone putting a variable field back near the top of a prompt, which has now happened to us twice.
Hit rate falling below the break-even point. For us that is 21.7%, computed from the read and write terms in the contract. Below it the cache is a cost center, and unlike a drifting hit rate this threshold means something specific.
Have you ever checked what your provider actually billed you, divided by the tokens you actually sent? That is the third alert: effective input price per million above $2.70 for two consecutive hours, against $2.51 healthy and $3.18 broken. It is derived from billing rather than from a feature dashboard, so it catches caching, tier changes, and anything else the provider does.
Stable-prefix share of input tokens is the fourth thing I watch, and it is the only one that is not really an alert. When retrieval starts returning more chunks, the ceiling drops, quietly and without any deploy. Ours is a weekly review line rather than a page, but somebody should be looking at it before the next caching proposal, because it is the first of the three factors and the one most likely to have moved.

Three days of overage after a four-minute revert

Jasmine Park — Thu, 06 Aug 2026 13:15:06 +0000

The change went out at 12:00 on a Monday. Nothing paged. Our spend-rate alert fires at 3x the trailing hourly median, which is built for a tenant running away with the bill, and this was a steady 50 percent above normal. It surfaced at 16:36 because somebody was reading a per-model cost breakdown for an unrelated reason. We reverted and the deploy was green four minutes later.

Our normal token spend is about $1,420 a day. Monday finished about $480 over. Tuesday came in $674 over. Wednesday was $136 over. It was flat again on Thursday, and only because we shipped something else on Wednesday night, not because of the revert.

Nothing was broken. The revert did exactly what a revert does. My mistake was in what I thought that guaranteed.

The single largest term was not in the repository at all, so git revert was never going to touch it. That is the part I want to spend this post on, because it is the one I did not have a name for and now check first.

The rule that was not in the deploy

The Monday change was two things shipped together. A prompt edit, which added a few examples to a system prompt. And a routing change, moving one class of requests to a larger model.

The prompt lives in the repository. The routing rule does not. It lives in a runtime config store, so we can change routing without a deploy, which is a feature we asked for and use constantly. Steady-state we run several deliberate non-default routes.

So the revert took back the prompt and left the routing rule pointing at the larger model.

That class is about 6.3 percent of our calls, and the larger model is roughly 7.4 times the unit price, so the class goes from about $89 a day to about $660: call it $570 a day extra while the rule is live. (I am treating its share of calls and its share of spend as the same, which flatters me slightly, since a class routed to a bigger model is usually the heavier work.) It stayed live for another 19 hours, until someone looking at a per-model cost breakdown for an unrelated reason asked why that route was still there. Across the whole incident that rule cost about $562, more than any other single term, and just the Tuesday portion of it is $277, which is 41 percent of Tuesday's overage on its own.

I had confirmed the diff. I had watched the deploy. Neither of those was ever going to show me a rule that lives in a different system.

Three smaller terms, and why they mattered more than their size

The other $397 of Tuesday is three things, and I list them because each one had already committed work that the revert could not recall.

The queue was already full. The same deploy kicked off a re-summarisation pass over documents touched in the previous 30 days, because the new prompt changed the summary format and we wanted consistency. It enqueued its whole work list, 61,000 documents, in the first few minutes. Our summarisation workers are a separate deployment from the API, and the revert pipeline does not roll them, so they kept running with the new prompt held in memory. The pass cost $336, of which $214 landed on Tuesday.

Those summaries are an input cost now. This is the term I had never thought about. The new format produced longer summaries, roughly 520 tokens against the 180 we had before. Those summaries are stored, and they get retrieved as context by downstream requests. So 61,000 documents now carry an extra 340 tokens each into every request that touches them, and about 142,000 retrievals a day land on one of those documents, because the recent set is what people actually work on. That is roughly $145 a day. A revert does not shorten text that is already written to a table. This term had no expiry at all: it was going to keep billing until somebody re-summarised those documents, which is what we shipped on Wednesday night. Left alone it would have been about 10 percent of our token spend in perpetuity, near $53,000 a year, which is a much larger number than the Tuesday everybody was looking at.

A dead-letter queue kept firing the old prompt. Failed calls go to a dead-letter queue with a scheduled re-drive that runs every six hours for two days. The payload we persist includes the rendered prompt rather than a pointer to a prompt version, which is a reasonable choice for reproducibility and means a re-drive fires the expensive prompt long after the revert. $38 on Tuesday, $9 on Wednesday. Trivial money. It mattered because a thin tail of expensive calls kept the graph from going cleanly flat, and for most of Tuesday afternoon I read that as the revert not having worked, which sent me looking in the wrong place.

Wednesday is the detail I would want a reader to take away. It was 10 percent over baseline, and of the two terms still running, the one that mattered had no expiry at all. Nobody escalates a 10 percent day, which is why this one ran until Wednesday night.

Reversibility has to be designed in

Here is the question I now ask before shipping anything that touches prompts, models or routing. Not "can we roll this back", because the answer is always yes and it is not the useful question. Instead: what will this change write, enqueue, or set outside the repository, and how long does each of those live?

That has a concrete answer every time, and it is usually a number of hours.

For any such change I write down four things:

What it writes, with a lifetime. Stored summaries, tags, embeddings, conversation state. Anything with no lifetime, like our summaries, gets a corrective backfill planned before the change ships, not after.
What it enqueues, and how long that queue takes to drain if we stop feeding it. Including which deployment owns the consumers, because that is what decides whether a revert stops them.
What lives outside the deploy. Feature flags, routing rules, model aliases, rate-limit tiers. This is the list I did not have.
The longest of the above, in the change ticket, instead of "revertible: yes".

For that Monday an honest entry would have read: revertible in minutes for the prompt, 19 hours for the routing rule if nobody checks it, about a day for the queue, and open-ended for the stored summaries. Writing that down would have changed how we shipped it. Probably behind a percentage rollout, which caps every one of those terms proportionally, including the one with no expiry.

The cheap version of the fix

Two things, both small.

The first is stamping the version into everything the pipeline writes and enqueues. Not because stamping makes the revert clean, it does not, but so that afterwards I can answer how much of what is being served or queued came from the version I just removed. Before this I had no way to answer that except by watching the spend graph and guessing, which is how Tuesday afternoon went.

import hashlib

cache_key = f"{hashlib.sha256(normalised_request.encode()).hexdigest()}:{prompt_version}"
retry_payload["prompt_version"] = prompt_version
summary_row["written_by_prompt_version"] = prompt_version

hashlib rather than the builtin hash() matters here: hash() on a string is salted per process, so it is silently wrong for anything shared across workers or restarts.

Then the query that tells you whether the revert actually landed, which is the one I did not have:

-- how much of what we are still serving came from the version we removed?
-- No time window on purpose: derived rows have no TTL, so a trailing-7-day
-- filter goes blind on exactly the thing this query exists to find.
SELECT written_by_prompt_version,
       count(*)                        AS rows_total,
       sum(summary_tokens)             AS tokens_carried,
       max(written_at)                 AS last_written
FROM   document_summaries
GROUP  BY 1
ORDER  BY tokens_carried DESC;

If last_written for the reverted version is later than your revert timestamp, something is still running. That one column would have caught the workers on Monday evening.

The second is an inventory of every LLM-affecting setting that is not in the repository, with an owner. Ours had eleven entries when we wrote it down, about eight more than I would have guessed. Three of them can change which model serves a request.

What I'd page on

Roughly in the order they would have helped that Monday:

Live routing config diverging from a declared snapshot. Check the running config against a snapshot committed to the repository, and alert on any rule that has been live longer than four hours without a matching commit. This is the one that catches the $277. Note that alerting on "traffic to a non-default model" would be useless for us, because deliberate non-default routes are our steady state; the divergence from a declared expectation is the signal, not the routing itself.
Any row written by a prompt version that is no longer deployed. A single counter off the query above. This is the alert that would have told me the workers were still going while I was staring at the graph.
Stored-context token volume per retrieval, as a trend. We already watch average input tokens per request, and this is the same signal from the artifact side rather than the request side. The term with no expiry was invisible in every cost view we had, because it does not show up as a spike. It shows up as slightly worse unit economics forever. Watch the size of what you retrieve, not only the count.
Queue depth on any job a deploy can trigger, weighted by mean cost per item rather than depth alone. Six hundred expensive items and sixty thousand cheap ones want different responses, and in this incident the sixty thousand were the expensive ones.
Age of the oldest item in the dead-letter queue. A re-drive carrying a rendered prompt is invisible in every other view, and it is the thing most likely to make you distrust a revert that worked.

We already page on spend rate against the same hour last week. That alert is not on this list, and not because it did its job. An alert tuned for a tenant spiking 70x does not fire on a routing change worth 50 percent, and a human reading an unrelated breakdown beat it. Everything above exists because of the four and a half hours that cost us, and the two days after the revert that cost more.

The thing I got wrong was smaller than any of this. I had a rollback plan, and I had never asked which parts of the system it covered.

Our p50 latency SLO was green all quarter. Nearly 1 in 10 sessions hit a wall anyway.

Jasmine Park — Wed, 29 Jul 2026 10:23:31 +0000

The dashboard said 1.9s p50 against a 2.5s target. Green. It stayed green the entire quarter. Meanwhile churn in one segment crept up and the support inbox filled with "the assistant is so slow" from people whose traces I could not find, because on average we were fine. The average was the problem. We were measuring the middle of the distribution and shipping the tail to a subset of users who never saw a fast response.

What a p50 SLO actually promises

A p50 target says "half of requests are at least this fast." Read that back slowly. It says nothing about the other half. For an LLM app, where a single request can fan out into a retrieval call, a rerank, a generation, and sometimes a retry, the other half is where the pain lives. Our p50 was 1.9s. Our p99 was 11s. One in a hundred requests took eleven seconds, and because a session is many requests, a lot more than one in a hundred sessions hit at least one eleven-second wait.

That compounding is the part people miss. If a user makes 10 requests in a session and each has a 1% chance of being a p99 dog, the chance that session contains at least one bad request is not 1%. It's about 1 minus 0.99^10, roughly 9.6%. So a problem that hits 1% of requests shows up in nearly 10% of sessions. "The tail at scale" (Dean and Barroso, CACM 2013) made this argument for datacenter services a decade ago and it is even truer for chained LLM calls, because we fan out more per user action than a typical web request ever did.

Why the tail is fat for LLM apps specifically

Three reasons our tail was worse than a normal API's:

Token-count variance. Response latency scales with output length. A request that generates 800 tokens takes far longer than one generating 80. The distribution is not a tight bell, it has a long right tail by construction.
Retries and fallbacks. When a provider 429s and we fall back, that request now carries two round-trips. Those land squarely in the tail, and they correlate with load, so the tail fattens exactly when traffic is high.
Cold cache and cold routes. The rarely-hit path (a long-context request, an unusual tool call) is both slower and rarer, so it never moves the median and always lives in p99.

None of these show up in a p50. All of them are what a real user in the wrong segment experiences every time, because "the wrong segment" often means "the users whose requests are systematically longer."

What I measure now

I stopped trusting a single percentile and a single global number. The changes that mattered:

Track p95 and p99, and alert on them, not just p50. The p50 tells you the common case is fine. The p99 tells you how bad the bad case is. You need both.
Percentiles per route, not global. Our global p99 hid that one endpoint (long-context summarization) had a p99 of 24s while everything else was single-digit. A global number averages your worst route into invisibility.
Session-level, not just request-level. Report the fraction of sessions that contained at least one request over threshold. That 9.6% number is the one that correlates with churn, and it's the one a request-level dashboard will never show you.
Watch the retry rate as a leading indicator. Our tail got worse before latency alarms fired, because retries climbed first. Retry rate is the canary.

A quick note on computing these honestly: do not average percentiles across shards or time buckets. p99 of the union is not the mean of the per-bucket p99s. If you're on Prometheus, histogram_quantile over the raw buckets is right; averaging pre-computed quantiles is a common and silent error that makes your tail look better than it is.

# right: quantile over aggregated raw histogram buckets, per route
histogram_quantile(0.99,
  sum by (le, route) (rate(llm_request_duration_seconds_bucket[5m])))

# wrong: averaging a pre-computed per-instance p99 (understates the real tail)
avg by (route) (llm_request_p99_seconds)

What it costs to ignore this

The dollar version, since that's usually what unlocks the work: the churned segment was small in headcount and above-average in spend, because heavy users make more requests and therefore hit the tail more often. We were losing the users who used the product most, and the SLO that was supposed to protect experience was structurally blind to them. A p50 SLO is tuned to the median user, but the heaviest users are not the median user, and they are where a lot of the revenue sits. So the number the SLO protects is not the number that pays the bills.

What I'd page on

p99 per route breaching target, not just global p50. A global green with one red route is still an incident for that route's users.
Fraction of sessions with any request over the user-perceptible threshold (we use 5s) climbing week over week. This is the churn-correlated number.
Retry rate rising ahead of latency, as the leading edge of a fattening tail.
p99/p50 ratio widening. If the median holds but the ratio grows, the tail is pulling away from the median, and a single average number will hide it.

Cost per action: the number your LLM spend dashboard cannot produce

Jasmine Park — Fri, 24 Jul 2026 13:30:00 +0000

TL;DR. Cost dashboards for LLM systems report spend per model, per provider, per thousand calls. Finance asks what the support copilot costs per resolved ticket, and the dashboard cannot answer because no call carries the ID of the action it served. One user action fans out to many calls, the fan-out multiplies when providers degrade, and per-call metrics hide the whole mechanism. The fix is attributing cost to the user action through a propagated trace ID, then alerting on cost per action instead of daily spend.

The question the dashboard cannot answer

The cost dashboards that come with LLM tooling all answer the same three questions. Spend by model. Spend by day. Tokens by endpoint. Useful for exactly one conversation: the one with the provider.

The conversation that actually happens is with a product owner, and it goes: this feature has a budget of X per month, are we inside it, and which change blew it up on Tuesday. Per-call metrics cannot answer that, because no call knows which feature it served or which user action it belonged to.

What one action costs

Take a support-copilot answer as a worked example. The numbers below are a model of the shape, not a bill I am quoting; substitute your own.

One "answer the ticket" action, healthy path:

1 routing call on a mini-tier model
1 retrieval rerank
1 main answer call on a frontier-tier model
2 tool round-trips (order lookup, refund policy)
1 log-summary call

Six calls for one action. The per-call dashboard renders this as six unrelated rows in three model buckets. The thing with a budget, the action, exists nowhere.

Now degrade the provider. Latency climbs until a quarter of calls blow through your 30-second client timeout, and your retry policy allows two retries. Expected attempts per call become 1 + 0.25 + 0.0625, about 1.31. Six calls become roughly 7.9 attempts on average, and an abandoned attempt is not free: depending on provider and streaming mode, it can still bill some or all of the tokens it generated before you hung up, so a degraded action pays for failures and their replacements both.

The average hides the tail. Even with timeouts modeled as independent coin flips, the p95 action in this setup bills 10 attempts, up two-thirds from the healthy six. Real incidents are worse: timeouts cluster in time, so the actions inside the degradation window take nearly all their calls through the retry ladder at once. The mean moved a little. The tail moved a lot. Every spend spike I have personally chased started life as a provider incident, not a traffic bump, and daily spend alerting noticed hours after the retry ladder had been billing the whole time.

The join key

The fix is unglamorous: one action ID, stamped on every call the action causes, all the way down through retries and fallbacks.

The instrumentation half already has a standard. The OpenTelemetry GenAI semantic conventions define token accounting on spans as gen_ai.usage.input_tokens and gen_ai.usage.output_tokens (the conventions moved into their own semconv repository this year and are still evolving, so pin the version you adopt). Put your action ID on the enclosing trace, and the rollup becomes a GROUP BY:

SELECT
  trace_id,
  feature,
  SUM(input_tokens)  * :in_price  +
  SUM(output_tokens) * :out_price AS action_cost,
  COUNT(*)                        AS calls_in_action
FROM llm_calls
GROUP BY trace_id, feature;

Prices are placeholders; take yours from the provider's sheet. From this table, three numbers per feature: p50 cost per action, p95 cost per action, and calls per action. The last one is your amplification factor, and it is the earliest signal you have. Calls per action rises the moment a dependency starts timing out, minutes before spend-per-hour looks unusual and hours before the invoice does.

Two honest gaps

Unattributed spend never reaches zero. Batch jobs, warmup calls, evals in CI. Give them synthetic action IDs and report the unattributed share explicitly. When that share creeps up, someone shipped a code path outside the instrumentation, and that is worth knowing regardless of cost.

And per-action pricing does not tell you whether the action was worth it. A resolved ticket at 4 cents and an abandoned session at 4 cents cost the same. Joining cost to outcome is the next join, and it needs the same trace ID, which is one more reason to lay it now.

What I'd page on

p95 cost per action, per feature, against a budget line, because daily spend alerting pages you only after the damage is done.
The amplification factor (billed attempts per action) drifting above its baseline. That is a retry ladder or a fallback loop announcing itself, minutes before spend charts move.
Unattributed spend share rising week over week. A new code path escaped the trace.
Actions that exhausted their retry budget. Each one paid the full fan-out and still failed the user, which makes them the most expensive failures you have.

One tenant, one polling loop, and $9,100 we didn't budget for

Jasmine Park — Tue, 21 Jul 2026 17:28:17 +0000

The page that mattered never fired. Finance sent it instead, on a Thursday, in Slack: "Projected LLM spend this month is 41% over. Is that expected?" It was not expected. It took me four hours to find out why, and every one of those hours was a monitoring failure, not a model failure.

Here is the short version. One customer shipped a change that put an LLM call inside a polling loop. Their integration went from roughly 2,000 calls a day to about 140,000 calls a day, overnight. We paid for all of it. It ran for six days before anyone looked, because every dashboard I owned was aggregate, and the aggregate looked boring.

The hot take I keep repeating in incident reviews: cost blowups are a metering gap. The bill breaks down per tenant, and if you only watch the total you miss the tenant that is on fire.

What happened

The customer's feature polled a downstream job for completion. Someone refactored it and, instead of polling the job status, the loop re-issued the full LLM summarization call on every tick. Tick interval was five seconds. Do the math on a handful of concurrent sessions running most of the day and you land around 140k calls. Their code, our endpoint, our invoice.

None of it errored. That is the part that stings. Every one of those calls returned a clean 200. Latency was fine. Our SLO burn was zero. The system was, by every signal we alerted on, perfectly healthy. It was also setting money on fire at a steady rate.

What it cost

Our average cost per call for that feature is about $0.011 (short prompt, capped output). Normal footprint for this tenant:

2,000 calls/day
about $22/day

After the loop bug:

about 140,000 calls/day
about $1,540/day

The delta is about $1,518 a day. It ran six days before finance flagged it. That is roughly $9,100 of spend we had no budget line for, on one tenant, on one feature, for output nobody consumed (they were throwing away 69 of every 70 responses).

For context, our whole platform's LLM spend was averaging around $1,800/day. This one tenant nearly doubled it. And I still did not see it, because our daily total already swung 30% on a normal week (weekend dips, batch backfills, onboarding spikes). A jump from $1,800 to $3,300 read like "big customer doing something legitimate." We had genuinely had 2x days before that were fine. So the signal sat inside the noise band, and the noise band was wide because I had never bothered to narrow it per tenant.

What it missed at scale

The scary math is not the $9,100. It is the slope. At $1,518/day this was a $45k/month leak from a single misbehaving integration, and we have hundreds of tenants. Any one of them can do this to us on any day. Our exposure was never "how expensive is the model." It was "how long can one runaway tenant run before a human happens to look at a bill." Six days, apparently. That is the actual SLO I had, and I had never written it down or defended it.

The fix

The refactor was easy: tag every call, attribute cost per tenant and per feature, and alert on spend rate the same way I alert on error rate. We already emitted tenant_id and feature on the request span. We just were not multiplying tokens by price and summing by tenant anywhere a human or an alert would see it.

The core alert is a spend-rate anomaly per tenant against that tenant's own recent baseline. Rolled up hourly, it looks like this:

-- hourly spend per tenant vs its trailing 7-day median
WITH hourly AS (
  SELECT tenant_id,
         date_trunc('hour', ts) AS hr,
         SUM(calls * cost_per_call) AS spend
  FROM llm_calls
  WHERE ts > now() - interval '8 days'
  GROUP BY 1, 2
),
baseline AS (
  SELECT tenant_id,
         percentile_cont(0.5) WITHIN GROUP (ORDER BY spend) AS med_spend
  FROM hourly
  WHERE hr < date_trunc('hour', now())      -- exclude current window
  GROUP BY tenant_id
)
SELECT h.tenant_id, h.spend, b.med_spend,
       round(h.spend / nullif(b.med_spend, 0), 1) AS ratio
FROM hourly h
JOIN baseline b USING (tenant_id)
WHERE h.hr = date_trunc('hour', now())
  AND h.spend > b.med_spend * 3            -- 3x its own median
  AND h.spend > 5                          -- ignore trivial tenants
ORDER BY ratio DESC;

Two guards matter. Compare a tenant to itself, not to the fleet (a big customer is allowed to be big, they just are not allowed to suddenly 70x). And put a floor on absolute spend so you do not page at 3am because a tiny tenant went from 4 cents to 15. On our data, this alert would have fired inside the first hour of the loop bug, at a ratio near 70x, six days before finance did.

I also wired a projection: current month-to-date spend, extrapolated to month end, compared against budget. That is the check that would have caught it even if the per-tenant alert had a gap. It is cheap and it maps directly to the number the business actually cares about.

What I'd page on

This is the dashboard and alert set I now run for anything that spends money per request. If you take one thing, take the fact that none of these are latency or error signals. Cost needs its own alerts, the same way latency and errors do.

Per-tenant spend rate. Page when any tenant's hourly spend exceeds 3x its trailing 7-day median, with a floor (we use $5/hour) so trivial tenants stay quiet. (The current hour is partial, so this catches a 60-70x spike fast and lags a subtle drift; pair it with the projection below.)
Top-tenant concentration. Page if a single non-whitelisted tenant crosses 40% of total hourly spend. One customer owning the bill is an incident until proven otherwise.
Calls/min per (tenant, feature). Page on step changes (greater than 5x hour over hour). This catches the loop before the dollars pile up, because request-count moves before the invoice does.
Cost per call, p99, per feature. Page when it climbs. Rising cost per call with flat volume means prompt or context bloat, a different leak with the same symptom.
Month-end spend projection vs budget. Page if the linear projection exceeds budget by more than 15%. This is the backstop that speaks finance's language.
New entrants in the top-10 spenders. Not a page, a daily digest. A (tenant, feature) pair you have never seen in the top 10 is worth 30 seconds of a human's attention.

The uncomfortable lesson: a green dashboard is not proof that nothing is wrong. It is proof that nothing you decided to measure is wrong. I measured latency and errors because those page loudly and customers complain. Nobody complains about a bill that is quietly too high, so nobody measured it, so it ran for six days.

Our LLM service had no backpressure. The provider got 7x slower and our p99 got 25x worse.

Jasmine Park — Fri, 17 Jul 2026 19:21:07 +0000

TL;DR. Our summarization endpoint holds a p99 under 3 seconds. On 21/05 our provider degraded: median call time went from 620ms to about 4.3 seconds, roughly 7x. Our p99 went to 35 seconds, roughly 25x, and it stayed there for 52 minutes. The provider was slow for 12 of those minutes. The other 40 were us draining a queue we had never bounded and could not see. Every arrival became an in-flight task, in-flight peaked near 9,000 across 12 pods, and the latency budget went on queue wait rather than on the provider. CPU sat at 7%, so the autoscaler never moved: the workers were I/O-bound, parked on a socket, burning no CPU at all. More pods were never the answer. What worked: a bounded queue, a concurrency semaphore, a wait budget that drops requests whose caller already left, and two metrics where we previously had one. Queue wait and provider latency are different numbers. Report them as one and you will blame your provider for your own queue.

Our summarization endpoint has one SLO anyone cares about: p99 under 3 seconds, 99% of the month. It held for two quarters. On 21/05 it missed for 52 minutes and burned about 13% of a 28-day error budget in an afternoon.

The provider was slow for 12 minutes.

That gap is the whole post. A 12-minute problem upstream became a 52-minute problem for us, and the extra 40 minutes were self-inflicted. All numbers here are ours, rounded, from our incident. The code at the bottom runs.

The shape of it

14:14, latency alert: p99 over 3s for five minutes. I opened the provider's status page, which was green, and our dashboard, which said p99 31s and climbing.

Nothing had deployed since 09:40. Error rate was 0.02%, which is normal. The endpoint was returning 200s. Slowly, but returning them.

Our provider-latency graph had gone from 620ms median to about 4.3s. Real degradation, and 7x. I filed a ticket with them and started the incident note with "upstream provider degradation" as the cause, because that is exactly what it looked like.

Then the arithmetic stopped working.

7x on a 620ms call gets you 4.3s. Our p99 was 35s. Even if every request paid the full degraded latency twice, that is 9s, not 35. Something was adding twenty-five seconds that was not the provider.

At 14:24 the provider recovered. Median went back to 640ms. I watched our p99 and it did not move. It sat at 35s for another forty minutes, on a healthy provider, with a green status page, while the alert kept firing.

At 15:06 I restarted the deployment. That dropped every queued request on the floor, and p99 was back under 3s within ninety seconds. A rolling restart is a crude, indiscriminate load-shed, and it is the only thing that ended the incident. I did not enjoy learning that.

A service still broken forty minutes after its dependency is healthy is not suffering from its dependency. It is suffering from what it did while its dependency was unhealthy. We had built a backlog, and the backlog had to clear before anybody got a fast answer. The provider's 12 bad minutes bought us 40 of our own.

Why the autoscaler sat still

First thing I checked was the HPA. CPU: 7%. Target: 70%. Nothing was computing, so the autoscaler had never had a reason to act and by its own logic it was right: every pod was sitting on await, holding a socket open to a provider taking four seconds to answer.

I wrote up the general version of this a couple of weeks ago, the week we found our autoscaler tracking request rate while the bill tracked tokens, so I will spare you the re-derivation and give you the line that transfers: scaling on a signal your incident cannot move is the same as not scaling at all. Here it would also have hurt, because more pods means more concurrent calls into a provider that was already saturating for us.

The saturation signal for an I/O-bound LLM service is in-flight requests against a concurrency limit, and queue depth against a queue bound. We had neither number, because we had neither limit.

The queue I could not see

Here is what I shipped, simplified to the shape that matters:

@app.post("/summarize")
async def summarize(req: Request):
    body = await req.json()
    resp = await client.post(PROVIDER_URL, json=build(body))   # httpx.AsyncClient
    return parse(resp)

No queue in that code. That is the problem: there is no queue in that code that I can name, measure, or bound. There are three, and I wrote none of them.

The event loop's task list is the first. Uvicorn accepts a connection, schedules a coroutine. At 40 requests per second with each parked for 4.3 seconds, you are holding about 172 at once. Nothing in that handler stops the number growing.

The second lives inside httpx. A default AsyncClient carries Limits(max_connections=100, max_keepalive_connections=20) (documented here; I also checked httpx._config.DEFAULT_LIMITS on 0.28.1 to be sure). The 101st concurrent request does not fail and does not reach the provider. It waits for a free connection, inside the pool, with no metric on it. We were timing await client.post(...), which includes that wait. So our "provider latency" graph was never measuring the provider. It measured the provider plus however long we sat in our own connection pool, and during the incident the second term dominated.

Third is the kernel accept queue, which I will not pretend I looked at.

None of these are bugs. Each is a sensible default doing exactly what it documents. Together, with 40 rps arriving and no admission control anywhere, they add up to a queue that grows without bound and reports itself to you as "the provider is slow."

What it cost

Close to nothing on the invoice, so this section is short: the provider bills tokens, and a slow call costs the same as a fast one. Financially this incident was a rounding error. That is precisely why it ran for 52 minutes without anyone escalating on cost.

It was expensive in the budget that applied. At 40 rps, a 28-day window is roughly 97 million requests, so a 99% latency SLO permits about 970,000 breaches of the 3-second line. We put roughly 125,000 requests over it in 52 minutes: about 13% of a month's error budget, in under an hour, caused by a dependency that was unhealthy for 12 minutes.

Around 100,000 of those had already timed out at 30 seconds and gone. We computed responses in full and wrote them to closed sockets.

Little's Law, and the number I did not have

The arithmetic that explains all of this is 65 years old and fits on one line.

L = λW. The average number of items in a queuing system equals the average arrival rate times the average time each item spends in it. John Little published the proof in 1961 (A Proof for the Queuing Formula: L = λW, Operations Research 9(3), 383-387) and wrote a genuinely readable retrospective on its fiftieth anniversary: Little's Law as Viewed on Its 50th Anniversary, Operations Research 59(3), 2011. It assumes almost nothing. No distribution, no independence, no particular queue discipline, and it holds when arrivals are nonstationary, which is exactly what an incident is. It describes your service whether or not you have thought about it.

Healthy: λ = 40 rps, W = 0.62s, so L = 25 concurrent calls. Comfortable. It is why nobody had ever needed a concurrency limit to exist.

Degraded: W = 4.3s, so holding 40 rps requires L = 172 concurrent calls. Twelve pods at 100 pooled connections each gave us room for 1,200, so we could physically open 172. We did. And that is what did the damage, because the provider would not serve 172 of our calls at once. Their effective throughput for us fell to about 27 rps.

Now the law runs the wrong way. Arrivals 40. Departures 27. The queue grows at 13 per second, 780 per minute, and over 12 minutes that is about 9,400. We measured a peak near 9,000 in-flight. That agreement is the only reason I trust the reconstruction at all.

Then invert it. L = 9,000, λ = 40, so W = L/λ = 225 seconds. Our clients time out at 30. At the peak we were producing answers that were, on average, 195 seconds too late to be wanted, and we kept producing them for forty minutes after the provider was fine. Nothing recovers from that on its own, which is why a restart was the only lever left.

The fix: a queue you can name

More pods would not have helped. A longer timeout is the other reflex, and it is strictly worse: a longer timeout means the caller waits longer before leaving, which raises W, which raises L. Timeouts are not a capacity strategy.

Four things went in that week. Roughly in the order I would put them back:

A concurrency semaphore. A hard cap on simultaneous provider calls, set below the point where the provider's latency curve bends. This is the one that stops 172 from ever happening. (asyncio.Semaphore is the whole implementation.)
A bounded queue with fail-fast admission. Past the bound, refuse immediately: 429 plus Retry-After. That is the same contract as the token-budget admission gate from that same write-up, moved down a layer and re-denominated. That one bounded tokens in flight, because tokens were what saturated the GPU. This one bounds calls in flight, because concurrency is what saturates a provider you do not own. asyncio.Queue(maxsize=N) raises QueueFull from put_nowait the moment it is full, which is the behaviour you want (docs). A 429 in one millisecond is a kinder answer than a 200 in 225 seconds.
A wait budget. When a worker picks a job up, check how long it sat. If it sat longer than the caller will wait, drop it and never call the provider. Calling on behalf of someone who has already hung up spends provider capacity that the callers still waiting need.
Two timers instead of one. One extra perf_counter() call. Cheapest thing on this list and the reason I understood any of the rest, which is why I have given it room further down rather than a bullet.

The whole thing, runnable, no dependencies:

"""Bounded concurrency + admission control for LLM calls.  python3 pool.py"""
import asyncio
import time
from dataclasses import dataclass, field

CONCURRENCY = 8    # simultaneous provider calls
QUEUE_MAX = 11     # admission buffer: drain rate x WAIT_BUDGET, not a vibe
WAIT_BUDGET = 2.0  # sec. waited longer than this and the caller is gone


class Shed(Exception):
    """Queue full at admission. -> 429 + Retry-After."""


class Stale(Exception):
    """Waited past budget. -> 503. Never reaches the provider."""


@dataclass
class Job:
    rid: int
    fut: asyncio.Future
    enqueued_at: float = field(default_factory=time.perf_counter)


class BoundedPool:
    def __init__(self, call, concurrency=CONCURRENCY, queue_max=QUEUE_MAX,
                 wait_budget=WAIT_BUDGET):
        self.call, self.wait_budget, self._n = call, wait_budget, concurrency
        self.q = asyncio.Queue(maxsize=queue_max)
        self.sem = asyncio.Semaphore(concurrency)
        self.wait_ms, self.provider_ms = [], []
        self.ok = self.shed = self.stale = 0

    async def __aenter__(self):
        self._workers = [asyncio.create_task(self._worker()) for _ in range(self._n)]
        return self

    async def __aexit__(self, *exc):
        for t in self._workers:
            t.cancel()
        await asyncio.gather(*self._workers, return_exceptions=True)

    async def submit(self, rid):
        job = Job(rid, asyncio.get_running_loop().create_future())
        try:
            self.q.put_nowait(job)        # QueueFull once maxsize is reached
        except asyncio.QueueFull:
            self.shed += 1
            raise Shed(f"req {rid}: {self.q.qsize()} already queued, refusing")
        return await job.fut

    async def _worker(self):
        while True:
            job = await self.q.get()
            try:
                async with self.sem:
                    # Everything above this line is our fault, not theirs.
                    waited = time.perf_counter() - job.enqueued_at
                    self.wait_ms.append(waited * 1000)
                    if waited > self.wait_budget:
                        self.stale += 1
                        job.fut.set_exception(Stale(f"req {job.rid}: {waited:.1f}s"))
                        continue
                    t0 = time.perf_counter()
                    try:
                        result = await self.call(job.rid)
                    except Exception as e:
                        job.fut.set_exception(e)
                    else:
                        self.ok += 1
                        job.fut.set_result(result)
                    finally:
                        self.provider_ms.append((time.perf_counter() - t0) * 1000)
            finally:
                self.q.task_done()


def pct(xs, p):
    s = sorted(xs)
    return s[min(int(round(p / 100 * (len(s) - 1))), len(s) - 1)] if s else 0.0


# A stand-in provider with a real ceiling: serves 12 at a time, never errors,
# and goes 7x slower two seconds in. Swap in your own client here.
_sem, START = None, 0.0


async def provider(rid):
    async with _sem:
        await asyncio.sleep(1.40 if time.perf_counter() - START > 2.0 else 0.20)
        return f"resp-{rid}"


async def main():
    global _sem, START
    _sem, START = asyncio.Semaphore(12), time.perf_counter()
    async with BoundedPool(provider) as pool:
        async def one(rid):
            try:
                await pool.submit(rid)
            except (Shed, Stale):
                pass  # in a request handler, return 429 / 503 here

        tasks = []
        for rid in range(400):
            tasks.append(asyncio.create_task(one(rid)))
            await asyncio.sleep(1 / 40)   # 40 rps of arrivals
        await asyncio.gather(*tasks, return_exceptions=True)

        print(f"served={pool.ok}  shed={pool.shed}  stale={pool.stale}")
        print(f"queue wait  p50={pct(pool.wait_ms, 50):6.0f}ms  p99={pct(pool.wait_ms, 99):6.0f}ms")
        print(f"provider    p50={pct(pool.provider_ms, 50):6.0f}ms  p99={pct(pool.provider_ms, 99):6.0f}ms")


if __name__ == "__main__":
    asyncio.run(main())

The stand-in provider has a ceiling (12 concurrent) and goes 7x slower two seconds in. It is not our incident in miniature, and I want to be precise about the gap: the stand-in never recovers, where ours did after 12 minutes and the backlog outlived it by forty. So the demo shows a backlog outliving its arrivals. It does not show one outliving the degradation that caused it, and that second thing is what made 21/05 confusing enough that restarting pods looked like the only lever. On my machine:

served=136  shed=255  stale=9
queue wait  p50=     0ms  p99=  2574ms
provider    p50=   201ms  p99=  1401ms

Run the same 400 arrivals with no pool, one task per request, the way we had it, and you get this instead, again on my machine:

served=400/400  peak in-flight=263  drained in 40s
end-to-end  p50=  12273ms   p99=  29696ms

Compare the provider numbers. The bounded run says the provider's p99 is 1,401ms, which is the truth: 1.4s is literally the constant inside the sleep. The unbounded run says 29,696ms. Same provider, same degradation, and a 21x difference in the number you would paste into a support ticket.

Watch the drain, too. Arrivals stop at 10 seconds; the unbounded run does not finish until 40. Three quarters of that run is backlog burning down after the last caller has arrived. Ours was forty minutes of drain on twelve minutes of degradation.

Timing our wait separately from their call

The highest-leverage line in that file is where the second timer starts.

async with self.sem:
    waited = time.perf_counter() - job.enqueued_at   # ours
    ...
    t0 = time.perf_counter()
    result = await self.call(job.rid)                # theirs

Two timers, split at the moment we actually begin talking to the provider. Everything before is our queueing: admission, queue, semaphore. Everything after is theirs.

We had been exporting one histogram, llm_request_duration_seconds, wrapped around the whole handler. That histogram is worse than having none, because it is confidently wrong. It read 35 seconds while pointing at a provider that was answering in 4.3. I took that graph into a support ticket and asked a vendor to explain a number my own service had manufactured. That is embarrassing, and one extra perf_counter() call would have prevented it.

We now export three: llm_queue_wait_seconds, llm_provider_duration_seconds, and end-to-end. The first two should roughly sum to the third, and when they stop summing, the difference is time being spent somewhere neither timer covers. That divergence is its own signal.

The rule I would hand my past self: if a request waits inside your process before you do the thing, that wait is a metric. Any queue you are not measuring is unbounded, because you cannot bound what you cannot see.

What it misses at scale

Four things still wrong, in descending order of how much they bother me.

I cannot defend the queue depth. I derived QUEUE_MAX = 11 from Little's Law: useful depth = drain rate x wait budget, and the degraded drain rate is 8 permits / 1.4s = 5.7 rps, so 5.7 x 2.0 ≈ 11. But drain rate is not a constant, so the derivation is thinner than it looks. Healthy, it is 8/0.2 = 40 rps, so the right depth would be 80. A fixed number is wrong in one direction at all times. I ran the demo at both depths, several times each, on my machine: served never moved outside noise (135 to 141 at depth 32, 131 to 141 at depth 11) while stale dropped from the seventies to around ten. Your absolute numbers will move with load; the gap between the two columns does not. Depth buys you no throughput whatsoever. It only decides whether you refuse a request in one millisecond or waste two seconds of its life first. The wait budget is what actually protects the caller. The depth only decides how much memory the backlog is allowed to occupy while it waits.

The SRE book disagrees with my ratio and may well be right. Chapter 22 of the Google SRE book, Addressing Cascading Failures, recommends keeping queue size small relative to pool size, on the order of 50% or less, so that a server rejects early under sustained overload. Mine is 11 against 8 permits, about 1.4x. I picked that deliberately for bursty arrivals: a short burst that clears inside the wait budget should be absorbed rather than refused. If your traffic is steady rather than bursty, take their ratio over mine.

The concurrency limit is static. Ours is a number I chose by watching where the provider's latency curve bends, then re-chose twice. The honest version measures it continuously, because a provider's capacity for you is neither constant nor published. That is real work, and we have not done it.

Shedding is not free, and the demo is blunt about the bill: 255 shed, 9 stale, 136 served out of 400. A 66% rejection rate. No amount of engineering fixes that, because a pool draining at 5.7 rps cannot absorb 40 rps of arrivals. Physics gets a vote. The only decision available is who finds out, and how quickly. Shedding does not rescue the requests it drops. It moves the failure to a moment you picked in advance, while you can still answer in one millisecond with a status code the caller can act on. If you read one thing before this post, read Handling Overload, chapter 21 of the same book.

What I'd page on

I had four dashboards for this service on 21/05. Every one of them was green while the SLO burned, because all four watched CPU, error rate, request count, and provider latency: the two that cannot see queueing, and two that actively lie about it. Here is what replaced them. Copy the metrics. The thresholds are ours.

Queue wait p99, alone, as its own series. Page when it exceeds half the wait budget. Not end-to-end, not provider latency: the wait, by itself. Had this existed on 21/05 it would have read 200 seconds while the provider graph read 4.3, and nobody would have spent nineteen minutes reading a status page.
Shed rate and stale rate, separately, never summed. They mean opposite things. Shed climbing means admission control is working and arrivals exceed capacity: expected, page only if sustained past five minutes. Stale climbing means we admitted work we could not finish in time, which is an admission-control bug, and the queue is too deep for the budget. That distinction is what QUEUE_MAX = 11 is for, and stale is the metric that proves the number is right.
Provider latency timed around the call and nothing else. Not the handler. Not the pool acquire. The call. This is the only number worth taking to a vendor, and until 21/05 we did not have it, which is why my support ticket was fiction.
Backlog drain time, derived: queue depth divided by observed drain rate. L/λ, on a graph. It answers "if arrivals stopped right now, how long until we are clear," and it is the number that tells you a restart is the only remaining lever, roughly forty minutes before you work it out by hand.

I have written before about a green dashboard hiding a real problem, so rather than run that argument again I will mark what is different here. That one was a bill: every run passed, nothing errored, and the damage showed up on an invoice. This one is time. Our error rate on 21/05 sat at 0.02%, normal for us, and the provider's status page stayed green from the first alert to the last. Neither number was wrong. Neither number was about where the 195 seconds went, because we were measuring how much time a request took and never which queue it spent it in.

Three questions decide whether you are running my 21/05 service. What happens to the 173rd concurrent call? How long did the last request sit before we dialed anyone? If arrivals stopped this second, when would the backlog be clear? We could not answer any of the three, and it took a provider having a bad twelve minutes to show us.

Self-hosted LLM observability: six stacks, weighed by what they cost you to run

Jasmine Park — Tue, 14 Jul 2026 21:24:23 +0000

TL;DR: If your LLM traffic sends personal data, or your obs invoice scales with token volume, self-hosting the trace pipeline is worth costing out. Storage is the cheap part (roughly 60 GB and about 18 dollars a month for a million spans a day at 30-day retention). The expensive part is that you now run it: retention, cardinality, ingest lag, and the pager. Here are six open-source stacks you can run yourself, what each is actually good at, and what each costs you to operate.

Why self-host at all

Two reasons show up in real incidents. The first is data residency: if a trace payload contains a prompt with a customer's PII, sending it to someone else's cloud is a compliance conversation you do not want to have after the fact. Self-hosting keeps the payload inside your VPC. The second is cost shape. Managed LLM observability tends to price on events or token volume, so the invoice grows exactly when your product succeeds. I inherited a managed bill that had quietly tripled over two quarters because traffic tripled. We moved the high-volume traces in-house. The invoice went down. The number of dashboards I now own went up.

That is the trade. You are swapping a predictable invoice for operational surface area. Before you make it, cost out the operational side honestly, because "open-source" is a license, not a free lunch. The question is never "is it free," it is "what does it take to keep it green at 3x the traffic."

What "self-hostable" has to mean here

I only included stacks you can actually run on your own infrastructure without a sales call, that speak a standard trace format (most of these are built on OpenTelemetry), and that a small team can operate. Every option below is open-source and self-hostable today. I have put them in no particular ranking order, because the right pick depends on whether you want pure tracing, a full platform, or just instrumentation you point at a backend you already run.

The six stacks

Langfuse. The one most teams reach for first. It is open-source and self-hostable, and it combines tracing with evals and prompt management, so it is more than a trace viewer. Strong at the prompt-iteration loop: versioned prompts, scores, and a clean trace UI. What it costs to run: a Postgres plus a ClickHouse-backed deployment for the self-hosted stack, which is a real database to operate at volume. What it can miss at scale: high-cardinality trace attributes will punish your storage if you do not sample. github.com/langfuse/langfuse

Arize Phoenix. Open-source, self-hostable, and built on OpenTelemetry. Like Langfuse it is more than tracing: it pairs trace collection with an evaluation layer, and it is notebook-friendly, which makes it strong for offline analysis and debugging a RAG pipeline span by span. What it costs to run: light to stand up for a single team, heavier once you want durable multi-tenant storage rather than an analysis session. What it can miss at scale: it started life analysis-first, so treat the production, always-on deployment as the part you validate under load. github.com/Arize-ai/phoenix

Helicone. Proxy-first. You route model calls through it and get logging, cost, latency, and caching almost immediately, which is the fastest path to a spend dashboard. Open-source and self-hostable. What it costs to run: low to start, but a proxy in the request path is now a thing on your critical path, so its availability is your availability. What it can miss at scale: proxy-based capture is excellent for cost and latency and less focused on deep multi-span agent traces than the OTel-native tools. github.com/Helicone/helicone

Future AGI. Open-source and self-hostable, and OpenTelemetry-native for tracing through its traceAI framework. It sits at the platform end of this list rather than the pure-tracing end: the same stack also carries evaluation, simulation, and a model gateway, so the draw is running one self-hosted system instead of several. Honest placement: it is younger at pure tracing than Langfuse or Phoenix, so if all you want is the most battle-tested trace viewer, it is not your first pick. What it costs to run: a full platform, so you operate more surface than a single-purpose tracer. What it can miss at scale: validate the tracing path at your volume before you retire the incumbent. github.com/future-agi/future-agi

SigNoz. The general-purpose APM option. It is open-source, self-hostable, and OpenTelemetry-native, and it treats LLM spans as spans inside your broader application traces. Strong if you already want one observability backend for services and models rather than an LLM-only tool. What it costs to run: a ClickHouse-backed APM, which your infra team may already know how to operate, which is a point in its favor. What it can miss at scale: it is not LLM-specialized, so prompt-level ergonomics (diffing prompt versions, judge scores) are not its focus. github.com/SigNoz/signoz

Traceloop OpenLLMetry. Not a backend at all, and that is the point. It is an open-source set of OpenTelemetry instrumentations for LLM apps that you point at whatever OTel-compatible backend you already run. Strong if you have an observability stack and just want your model calls to show up in it without vendor lock-in on the storage side. What it costs to run: almost nothing on its own, because it is instrumentation, but you still need a backend, so its true cost is whatever you send the data to. What it can miss at scale: it gives you spans, not opinions, so the eval and prompt-management layers are on you. github.com/traceloop/openllmetry

Cost out the storage before the on-call

The invoice you escaped was the easy number. Here is the one people skip. This is a rough hot-storage estimate, not a full TCO, but it anchors the conversation.

def trace_storage_gb(spans_per_day, bytes_per_span, retention_days):
    return spans_per_day * bytes_per_span * retention_days / 1e9

def monthly_cost(spans_per_day, bytes_per_span=2_000, retention_days=30,
                 usd_per_gb_month=0.10, replication=3):
    gb = trace_storage_gb(spans_per_day, bytes_per_span, retention_days)
    return gb, gb * usd_per_gb_month * replication

for spd in (1_000_000, 20_000_000):
    gb, cost = monthly_cost(spd)
    print(f"{spd:>12,} spans/day -> {gb:8.1f} GB hot, ~${cost:7.2f}/mo storage (x3 repl)")

   1,000,000 spans/day ->     60.0 GB hot, ~$  18.00/mo storage (x3 repl)
  20,000,000 spans/day ->   1200.0 GB hot, ~$ 360.00/mo storage (x3 repl)

Storage is cheap. Eighteen dollars a month for a million spans a day is not what makes self-hosting expensive. What makes it expensive is the ClickHouse or Postgres you are now running, the retention job you have to get right, the cardinality that blows up when someone puts a UUID in a span attribute, and the fact that when ingest lags, that is your page. Plug your real numbers in. If the storage line is trivial and the operator line is not, you have learned the actual shape of the decision.

What I'd page on

If you self-host any of these, do not page on "the dashboard is down." Page on the things that mean you are silently losing data:

Ingest lag over threshold (your traces are minutes behind reality, so every incident you debug is stale).
Dropped or refused spans greater than zero (the exporter queue is full, and you are blind to exactly the traffic spike you most need to see).
Storage growth rate outrunning your retention budget, or trace attribute cardinality spiking (a UUID landed in a label, and your index is about to hurt).
Sampling rate drift (someone changed the sampler, and your tail-latency traces quietly stopped being collected).

A $3,900 overnight bill from our LLM eval suite: the incident, and the spend guard I shipped after

Jasmine Park — Tue, 14 Jul 2026 21:21:13 +0000

TL;DR. Our LLM-judge eval suite had no cost ceiling. It ran the full judge over 1,200 cases on every CI trigger. On 08/07 a dependency bot opened 41 pull requests between 01:00 and 04:00, and our merge queue re-ran the whole suite on every push and every rebase: roughly 270 full runs at about $14.40 each. That one window cost $3,900 against a $1,730 monthly eval budget. No dashboard fired, because ours watched request rate and 5xx, not tokens or dollars. We found out from the invoice, because nothing we monitored watched spend. What I shipped was four guards under the suite: a pre-flight cost cap (estimate tokens with tiktoken, multiply by price, refuse the run if it would breach a daily ceiling), a result cache keyed on the candidate answer, sampling on non-main branches, and an alert on token-spend rate. Code is below.

I run reliability for a small ML platform team. We ship an LLM feature and gate it with an offline eval suite: 1,200 graded cases, each scored by a separate judge model against a rubric. Standard setup. It has caught real regressions. It also carried, for four months, a failure mode I built and did not see until it cost us most of a monthly budget in nine hours.

This is the writeup. Numbers are from our incident, rounded. The prices are the ones we paid at the time, not a benchmark.

How I found out

The first signal came in as an email.

At 09:40 the next morning, finance forwarded a cost-anomaly notice from our model provider: the prior day's spend on one API key was 68x its trailing average. My first reaction was that the anomaly detector was wrong. We had shipped nothing overnight. No incident channel, no 5xx, no latency alarm. Green board.

Then I read which key. It was ci-eval, not prod. I had no dashboard for ci-eval, because eval traffic never paged anyone, so I had never built one. I pulled the provider's usage export for that key and sorted by hour. Between 01:00 and 04:00 it had billed just over a billion tokens. A normal day for that key is about 14 million.

I checked the deploy log twice, expecting a runaway retry loop or a stuck worker hammering the API. There was neither. The requests were clean, sequential, and successful. Whatever had done this had done it deliberately, at 200 OK, which meant the provider was behaving correctly and the bug was in my own cost arithmetic.

That is how I learned my eval suite had a cost bug: about twelve hours late, from a finance email, in dollars rather than in the tokens or request counts I was actually watching.

What it cost

The arithmetic is the whole story, so here it is.

One judged case, at our rubric size, costs:

input: about 2,600 tokens (rubric, question, candidate answer, reference answer)
output: about 550 tokens (a score plus a short justification)

At the prices we paid ($2.50 per million input, $10.00 per million output), that is $0.0065 plus $0.0055, so roughly $0.012 per case. The full suite is 1,200 cases, so one run is 1,200 x $0.012 = $14.40. On a normal day CI fires it three or four times: a couple of merges to main, a manual rerun or two. Call it $57 a day, about $1,730 a month. That budget line had been flat for a quarter. So I stopped looking at it. That was mistake one.

On the night of 08/07, a dependency bot opened 41 pull requests in three hours. Each PR triggered the full suite on its first push. Each then got a lockfile follow-up commit, which triggered it again. Then the merge queue rebased each PR onto main before merging and ran the suite once more. Six to seven full runs per PR. About 270 runs total.

270 x $14.40 = $3,888. Just over a billion tokens. In one overnight window we spent 2.25x our entire monthly eval budget, and every single run passed. Nothing was broken. That is the part that still bothers me. The suite did exactly what I had configured it to do, 270 times, and nothing malfunctioned. The entire bill came from correct runs firing far more often than they ever should have.

Split the billion tokens and it maps straight back to the bill: about 842 million input tokens at $2.50 per million is $2,106, and about 178 million output tokens at $10.00 per million is $1,782. Sum $3,888. Output was 17% of the tokens and 46% of the cost, which is worth internalizing. On a rubric-heavy judge, the short justification you ask it to write is nearly half the spend. Trimming the justification moves the bill more than trimming the rubric does.

Root cause, and my part in it

Two settings did this. Both were mine.

Our CI ran the eval job on the GitHub synchronize event, which fires on every push to an open PR. I wrote that trigger in March so a PR's eval status stayed fresh as you pushed fixes. Reasonable for a human pushing three commits to one branch. Pathological for a bot pushing 41 branches with lockfile follow-ups.

The merge queue had "re-run required checks after rebase" turned on. I enabled that in May, after a stale check let a bad merge through. Also reasonable in isolation. Combined with the synchronize trigger and a bot storm, it meant every PR paid for the full suite at least three times before it even cleared the queue.

Neither setting is wrong on its own. Together, with no cost ceiling underneath them, they are a spend amplifier. I built both halves, two months apart, and never once did the multiplication. That combination is the actual root cause. The dependency bot was only the trigger: it exercised a gap I had left open, 41 times overnight.

What the dashboards missed

Three things were watching this system. All three were structurally blind to a cost spike.

The provider billing console. Aggregated by calendar day, updated on a lag of roughly 24 hours. At 04:00, while the spend was actually happening, it still showed the previous day's $58. It is accurate but far too slow for a spike that starts and finishes inside one night.

Our Grafana panels. Built on request count and error rate, scoped to the prod key. Two separate problems. First, count is the wrong unit. 270 eval runs is not a remarkable request volume; one prod minute makes more requests than that. The signal was never in the count, it was in tokens-per-request times price. Second, the panels were scoped to prod, and the runaway was on ci-eval, which had no panel at all.

The one alert we did have. A 5xx and rate-limit alarm on the provider. It never fired, because the provider was perfectly happy. It served all billion tokens without a single error. This is the trap: a cost runaway is not an error condition. The provider will keep billing at 200 OK and never surface an error for you to alert on.

So the real failure was not that an alert broke. It is that I had never expressed "dollars per hour on the eval key" as a number that anything watched. I monitored availability and errors, which are what page you at 3am. Spend was only ever reviewed in a monthly budget meeting, so nothing watched it continuously.

The fix

Four changes, ordered by how much they mattered.

A pre-flight cost cap. Before any eval run, estimate its token cost with tiktoken, multiply by the price, and refuse to run if it would push the key past a daily ceiling. This is the guard that stops run number nine, not run number 270.

import tiktoken

# Judge pricing, USD per 1M tokens (from the provider's public pricing page).
PRICE_IN_PER_M = 2.50
PRICE_OUT_PER_M = 10.00
DAILY_BUDGET_USD = 120.00  # hard ceiling for the ci-eval key

def encoding_for(model: str):
    try:
        return tiktoken.encoding_for_model(model)
    except KeyError:
        # Newer judge models aren't in tiktoken's registry yet; fall back.
        return tiktoken.get_encoding("o200k_base")

def projected_cost(prompts: list[str], est_output_tokens: int, model: str) -> float:
    enc = encoding_for(model)
    in_tokens = sum(len(enc.encode(p)) for p in prompts)
    out_tokens = est_output_tokens * len(prompts)
    return (in_tokens / 1e6) * PRICE_IN_PER_M + (out_tokens / 1e6) * PRICE_OUT_PER_M

def guard(prompts, est_output_tokens, model, spent_today):
    run_cost = projected_cost(prompts, est_output_tokens, model)
    if spent_today + run_cost > DAILY_BUDGET_USD:
        raise SystemExit(
            f"eval blocked: projected ${run_cost:,.2f} would push today's "
            f"ci-eval spend past the ${DAILY_BUDGET_USD:,.2f} ceiling"
        )
    return run_cost

The edge case that bit us while testing this: encoding_for_model raises KeyError for judge models tiktoken has not shipped a mapping for yet. If you let that propagate, the guard crashes, and depending on how you wired it that either blocks all evals or (worse) gets swallowed by a broad except so the cap silently stops applying. Fall back to a known encoding: o200k_base for recent models, cl100k_base for older ones. The estimate is then approximate, which is fine here. This is a budget fuse, not an invoice. tiktoken is the actual tokenizer for this model family and it is open source (github.com/openai/tiktoken), so the estimate lands within a couple of percent of billed tokens in practice.

A result cache. Key the judge result on (rubric_version, case_id, sha256(candidate_answer)). On a dependency-bump PR, the model under test emits byte-identical answers, so the candidate hash does not change and the judge never runs twice on the same input. Rerunning 270 times over the same 1,200 unchanged candidates should have been about 1,200 judge calls and 322,800 cache hits: a hit rate near 99%. Cost with the cache in place: about $14, paid once.
Sampling off main. The full 1,200-case suite runs on merges to main and in the merge queue's final gate. Every other trigger (feature pushes, draft PRs, bot PRs) runs a fixed 10% stratified sample, 120 cases, about $1.44. You still catch gross regressions on a branch. You stop paying full freight for a lockfile bump.
Spend-rate alerting. This one gets its own section below, because it is the part I most want you to copy.

We shipped all four in two days. The cap and the cache did the heavy lifting. Together they take the worst case of a storm like this from $3,900 down to about $120, because once the ceiling is hit the cap simply stops starting new runs.

What it misses at scale

I am not going to pretend this is finished.

The pre-flight estimate uses a fixed est_output_tokens. Judges that write longer justifications on hard cases will be under-estimated, so the fuse is loosest on exactly the runs that cost the most. We set the constant to our 90th-percentile output length, which over-charges most runs on purpose. Safer, less precise. That is the trade, and it is deliberate.

The cache is only correct while the rubric and the judge model are pinned. Bump either one and every cache key changes, so the first run after a judge upgrade pays full price, and 270x full price if the same bot storm lands that same morning. The cap catches that case. The cache does not.

Sampling trades cost for coverage. A 10% branch sample will miss a regression living in the 90% you skipped, and you only see it at the main-merge gate. For us that is acceptable, because branch evals are advisory and the main gate is the one that blocks release. For a team that ships straight off branches, it is not, and they should not copy the 10%.

And a daily ceiling is a blunt instrument. Set it too low and you block legitimate work at 16:00 on a heavy release day. Set it too high and it would not have stopped this incident. We landed on 2x a normal day and accept that a genuinely large legitimate day needs a human to raise it by hand. The ongoing work here is tuning that threshold, not writing more code.

What I'd page on

The lesson was not "eval is expensive." It was that I had been monitoring the wrong quantities. I watched request counts and error rates; the failure only ever showed up in tokens and dollars, which nothing tracked. So here is the dashboard I built afterward, and the four things it pages on. Copy the metrics, not the numbers. The numbers are ours.

Token-spend rate, per API key, per hour. The primary signal. Alert when any key crosses 3x its own trailing 7-day hourly median. This is the one that would have paged me at 01:20 instead of routing through finance at 09:40. Per-key matters: ci-eval and prod have different baselines and have to alarm independently, or the loud one masks the quiet one.
Cost per eval run. Emit it as a metric from the harness itself, tagged by trigger (main, merge-queue, branch, bot). Page if a single run exceeds $20 (our full run is $14.40) or if runs-per-hour exceeds 12. A run-count spike is the earliest sign of a trigger loop, and it shows up before the dollars land in any billing export.
Budget burn, daily and month-to-date. Track spend against the daily ceiling as a percentage: warn at 70%, page at 100% (by which point the cap has already blocked runs, so the page is really telling a human to decide whether to raise it). Track month-to-date against the $1,730 line and warn at 80% before the 20th, so an expensive first half of the month is visible while there is still time to react.
Cache hit rate on the judge. A drop below 60% means either a rubric or model change invalidated the cache (expected, transient, no page) or the cache key is broken (not expected, page). On a normal week we sit near 40% from genuinely new candidates. A sudden collapse to single digits during a PR storm is the tell that the cache is not absorbing what it should, which is the early warning I did not have on 08/07.

None of these watch availability. Availability was fine the entire night. The provider returned 200 to every one of a billion tokens' worth of requests, and that was precisely the problem. Reliability for an LLM system has to include a spend SLO, and a spend SLO needs continuous monitoring rather than a monthly budget review.

If you run an LLM-judge eval suite and you cannot answer, from a dashboard, right now, "what does one run cost, and what stops it running a thousand times tonight," that is your next on-call ticket. We paid $3,900 to find that out.

We alerted on errors. The silent failure was a truncated answer.

Jasmine Park — Wed, 08 Jul 2026 18:31:42 +0000

Every monitor was green. HTTP 200s, latency inside SLO, zero exceptions. Meanwhile users were getting half an answer and cut-off JSON, and nothing on our side had noticed, because a truncated completion is not an error.

The page never fired. That was the whole problem. Support forwarded a handful of complaints that answers were "getting cut off," and I opened the dashboards expecting to see the cause. Error rate flat at zero. Latency well under budget. No exceptions in the logs, no failed calls, no 5xx. By every signal we watched, the service was healthy. The signals were wrong. It was handing back broken output with a 200 stamped on it, and none of our monitors were built to notice.

What the monitors were watching, and what they missed

We had inherited the standard web-service monitoring shape: watch HTTP status, watch latency, watch exceptions. That shape assumes failure looks like an error. For a normal API that holds. For an LLM call it does not, because the most common user-facing failure throws nothing at all.

The failure was truncation. When a completion hits the max_tokens ceiling, the model stops mid-thought and hands back what it had so far. The provider returns a perfectly valid response object with a 200. The only tell is a field in the payload: finish_reason comes back as length instead of stop. No exception, no error code, no latency anomaly. A response that stopped because it ran out of room looks identical, at the transport layer, to one that finished naturally.

So the answer that got cut in half was a 200. The JSON blob that ended after {"status": "app was a 200. Our completeness was failing and our monitors were structurally blind to it, because they were counting the wrong kind of failure.

What it cost before we caught it

This ran for the better part of two weeks before the pattern was clear enough to act on. Best I can reconstruct, somewhere around 3 to 4 percent of completions were truncating, and a big share of those were the long-output ones: multi-step answers, long summaries, and the structured JSON responses another service consumed downstream.

The JSON case was the expensive one. A downstream service parsed those completions. A truncated blob is invalid JSON, so that service caught a parse exception, swallowed it, and fell back to a default. No page there either. So one silent failure (truncation with a 200) fed a second silent failure (a swallowed parse error and a default value), and the only place reality surfaced was a slow trickle of user complaints. Two weeks of that is a lot of quietly wrong answers.

Treat finish_reason as a first-class signal

The fix starts with promoting finish_reason from a field nobody reads to a metric you alert on. Every completion, check why it stopped, and emit that. A rising rate of length finishes is a truncation incident in progress, and it will show up here long before support forwards the first complaint.

from collections import Counter

_finish_reasons = Counter()

def record_finish_reason(response) -> str:
    # OpenAI-style: response.choices[0].finish_reason
    # "stop" = natural end, "length" = hit max_tokens (truncated), plus others
    reason = response.choices[0].finish_reason or "unknown"
    _finish_reasons[reason] += 1
    return reason

def truncation_rate() -> float:
    total = sum(_finish_reasons.values())
    return _finish_reasons["length"] / total if total else 0.0

That is the signal we never had. truncation_rate() is the number that should have paged us on day one.

Monitor completeness, not just finish_reason

Finish reason tells you the model ran out of room. It does not tell you whether the answer the user got was usable. For that, check the output itself. For structured responses, the sharpest signal is whether the payload parses and carries the fields you require. A valid-JSON-parse rate and an expected-fields-present rate turn "the answer looks complete" into a number you can alert on.

import json

_completeness = {"parse_ok": 0, "parse_fail": 0, "fields_ok": 0, "fields_missing": 0}
REQUIRED_FIELDS = ("status", "summary", "items")

def check_completeness(text: str, structured: bool) -> dict:
    result = {"truncated_json": False, "missing_fields": []}
    if not structured:
        return result
    try:
        obj = json.loads(text)
        _completeness["parse_ok"] += 1
    except json.JSONDecodeError:
        # a cut-off JSON blob lands here: valid 200, unusable payload
        _completeness["parse_fail"] += 1
        result["truncated_json"] = True
        return result
    missing = [f for f in REQUIRED_FIELDS if f not in obj]
    if missing:
        _completeness["fields_missing"] += 1
        result["missing_fields"] = missing
    else:
        _completeness["fields_ok"] += 1
    return result

def json_parse_rate() -> float:
    total = _completeness["parse_ok"] + _completeness["parse_fail"]
    return _completeness["parse_ok"] / total if total else 1.0

Now a truncated JSON response is not a downstream default swallowed in silence. It is a parse_fail, counted, and it moves a rate you can wake someone up over.

Set max_tokens from the data, not a guess

The last piece is the setting that caused it. Our max_tokens was a round number somebody picked once and never revisited. It was too low for the long-output traffic, which is exactly the traffic that truncates. The right value is not a guess, it is the measured distribution of actual output lengths with headroom on top.

def recommend_max_tokens(observed_output_lengths, headroom=1.2, pct=99):
    # size the ceiling off the p99 real output, not a round number
    s = sorted(observed_output_lengths)
    p99 = s[min(len(s) - 1, int(pct / 100 * len(s)))]
    return int(p99 * headroom)

Feed it a few days of real completion lengths and it hands back a ceiling that clears your p99 output with room to spare, instead of a number that clips your longest and most important answers. Reserving output budget you rarely use costs a little on the concurrency side, so this is a trade to make with eyes open, not a free lever. But clipping real answers costs correctness, and correctness is not a thing to save money on quietly.

Once truncation rate, JSON parse rate, and expected-fields-present were on the wall, the incident stopped being a support-ticket archaeology exercise. The next time a prompt change pushed outputs longer and truncation started climbing, the rate moved within minutes and we caught it before a single user did.

What I'd page on

Different checklist from the token-cost write-up. This one is about a failure that never throws.

Truncation rate (finish_reason == "length"). The headline. Fraction of completions that stopped because they hit max_tokens. Warn at a low single-digit percent, page on a sharp climb. This fires on the exact failure that green error dashboards hide.
JSON parse-success rate. For any structured output, the fraction that parses. A drop is truncated or malformed payloads reaching consumers. Page on it, because the downstream service will swallow the failure and you will not hear about it otherwise.
Expected-fields-present rate. Of the payloads that do parse, the fraction carrying every required field. Catches the answer that parsed but came back half-filled.
Output-length distribution vs max_tokens. Plot p50 / p95 / p99 output length against the ceiling. When p99 marches toward the ceiling, truncation is about to start. This is the leading indicator.
Downstream default-fallback rate. How often the consuming service fell back to a default because it could not use the response. A silent failure feeding a silent failure is the worst case, so surface the second one too.
Completeness by route. Truncation is not uniform. Break the truncation rate out by endpoint or prompt template so the long-output routes, the ones that actually clip, are not averaged into invisibility by the short ones.

Our cache hit rate was 90 percent and the bill still climbed

Jasmine Park — Wed, 08 Jul 2026 18:22:36 +0000

The dashboard said we were serving nine of every ten requests from cache. The invoice said we were paying more every week. Both numbers were correct. The hit rate was counting requests and the bill was counting dollars, and those are not the same thing once your traffic is lopsided.

The response cache went in for exactly the reason you would expect. Repeated prompts were hitting the model over and over, so we keyed on the prompt, stored the completion, and served the stored answer on a match. Within a week the hit-rate panel settled around 90 percent and stayed there. Nice flat line. Everyone moved on.

Then the monthly spend kept creeping. Not a spike, a slope. Up a bit, up a bit more, up again. The cache was clearly working (90 percent of requests were free), so nobody looked at it as the culprit. That was the mistake. The cache was doing exactly what the panel measured. We had picked the wrong thing to measure.

What the hit rate was actually counting

A hit rate is a ratio of requests. Ninety percent hit means nine of ten requests were served from the store and one went to the model. What that number never told us is which requests were in which bucket.

Our traffic was not uniform. The bulk of it was short, cheap, repetitive calls: the same handful of classification and lookup prompts, hammered thousands of times an hour. Those cached beautifully. They were also nearly free per call. Meanwhile a thin slice of traffic was long-context work, tens of thousands of tokens of document stuffed into the prompt, and those were mostly unique. They missed the cache almost every time.

So the 90 percent hit rate was 90 percent of the cheap requests and almost none of the expensive ones. The cache was serving the traffic that barely cost anything and passing through the traffic that cost real money. Counted by request, the cache looked like it was carrying us. Counted by dollars spent, it was barely touching the bill. The panel only knew how to count requests, so it never showed us the second number.

What it actually cost

Put rough numbers on it. Say a cheap call is 200 tokens and a long-context call is 30,000 tokens, so the expensive one costs roughly 150 times more. If 95 percent of requests are cheap and 5 percent are long-context, and the cheap ones hit cache 94 percent of the time while the long ones hit 8 percent of the time, the request-weighted hit rate still lands around 90 percent. The dollar-weighted hit rate, dollars served from cache over total dollars, was under 15 percent.

That is the whole gap. Ninety percent of requests free, and we were still on the hook for something like 85 percent of the money. The panel we trusted only counted requests, and requests were not what showed up on the invoice.

Two more things the cache was quietly doing wrong

Once we started looking, two smaller problems fell out of the same investigation.

The keys were too strict. We keyed on the raw prompt string. A trailing space, a reordered pair of retrieved chunks, a timestamp injected into the system prompt, any of these produced a different key and a forced miss on what was semantically the same request. We were busting the cache on prompts that should have collided. Normalizing the key (strip whitespace, sort the parts that are order-insensitive, drop the volatile fields that do not change the answer) recovered a real chunk of hits on the traffic that mattered.

The cheap calls were inflating the number we bragged about. Because the tiny repetitive calls dominated the request count and cached almost perfectly, they dragged the headline hit rate up toward 90 no matter what the expensive traffic did. The one metric on the wall was structurally incapable of showing us the miss cost, because the misses were rare by count and huge by dollar.

The metric we should have had from day one

The fix is to weight the hit rate by cost instead of by request. Every lookup carries a dollar estimate. Sum the dollars you served from cache, sum the total dollars you would have spent, and divide. That number moves when an expensive request misses, which is exactly when you want it to move.

from dataclasses import dataclass, field

# price per 1k tokens for the model you are caching in front of.
# use your real numbers; these are placeholders.
INPUT_PER_1K = 0.0005
OUTPUT_PER_1K = 0.0015

def dollar_cost(prompt_tokens: int, output_tokens: int) -> float:
    return (prompt_tokens / 1000) * INPUT_PER_1K + (output_tokens / 1000) * OUTPUT_PER_1K

@dataclass
class CostWeightedCacheStats:
    dollars_from_cache: float = 0.0   # cost we avoided by serving a hit
    dollars_total: float = 0.0        # cost we would have paid with no cache
    hits: int = 0
    misses: int = 0
    miss_costs: list = field(default_factory=list)  # per-miss dollar cost

    def record_hit(self, prompt_tokens: int, output_tokens: int) -> None:
        c = dollar_cost(prompt_tokens, output_tokens)
        self.dollars_from_cache += c
        self.dollars_total += c
        self.hits += 1

    def record_miss(self, prompt_tokens: int, output_tokens: int) -> None:
        c = dollar_cost(prompt_tokens, output_tokens)
        self.dollars_total += c
        self.miss_costs.append(c)
        self.misses += 1

    @property
    def request_hit_rate(self) -> float:
        total = self.hits + self.misses
        return self.hits / total if total else 0.0

    @property
    def dollar_hit_rate(self) -> float:
        # the number that actually tracks the bill
        return self.dollars_from_cache / self.dollars_total if self.dollars_total else 0.0

    def miss_cost_percentiles(self):
        # where the missed money lives: is it the long tail?
        if not self.miss_costs:
            return {}
        s = sorted(self.miss_costs)
        def pct(p):
            return s[min(len(s) - 1, int(p / 100 * len(s)))]
        return {"p50": pct(50), "p90": pct(90), "p99": pct(99), "max": s[-1]}

Wire it around the cache lookup and both numbers fall out of the same code path.

stats = CostWeightedCacheStats()

def cached_call(req, cache):
    key = normalize_key(req.prompt)     # strip, sort order-insensitive parts, drop volatile fields
    hit = cache.get(key)
    if hit is not None:
        stats.record_hit(req.prompt_tokens, hit.output_tokens)
        return hit
    resp = call_model(req)
    stats.record_miss(req.prompt_tokens, resp.output_tokens)
    cache.set(key, resp)
    return resp

# emit both, side by side, so nobody trusts the wrong one again
def emit_metrics():
    return {
        "cache.request_hit_rate": round(stats.request_hit_rate, 3),
        "cache.dollar_hit_rate": round(stats.dollar_hit_rate, 3),
        "cache.miss_cost": stats.miss_cost_percentiles(),
    }

When we put those two lines on the same panel, the gap was undeniable. Request hit rate near 90, dollar hit rate scraping the bottom. The miss-cost percentiles made it concrete: the p99 miss was hundreds of times the median miss. All the money was in the tail, and the tail was invisible on the old graph.

Normalizing the keys pulled the dollar hit rate up, because some of that expensive traffic was more repetitive than the strict key let us see. But the honest conclusion was that a response cache in front of mostly-unique long-context calls was never going to save much. Knowing that stopped us from over-investing in a cache that could not move the bill, and pointed us at prompt trimming and shorter retrieved context instead, which did.

What I'd page on

Different checklist from the autoscaling write-up. This one is about a cache lying to you by measuring the wrong axis.

Dollar-weighted hit rate. Dollars served from cache over total dollars. This is the headline. Warn if it drops below your target while request hit rate stays high, because that divergence is the exact failure in this post. Request hit rate alone is a vanity metric here.
Request hit rate vs dollar hit rate, on one panel. Never show one without the other. The wider the gap between them, the more spend the request number is hiding.
Miss cost distribution, p50 / p90 / p99. Where the unserved money lives. A p99 miss orders of magnitude above the median means your expensive traffic is bypassing the cache, and that is where to spend engineering time.
Key-collision rate. Fraction of lookups where a normalized key would have hit but the raw key missed. A rising value means your keys are too strict and you are busting cache on semantically identical prompts. Alert when it climbs.
Cost per served request, hourly. Total spend over requests served. Flat is healthy. Rising while the request hit rate holds steady is the surprise invoice forming, and it is invisible on any request-count panel.
Cache dollar savings, absolute per day. Not a rate, a number: dollars the cache actually kept off the bill today. If it stops growing while traffic grows, the cache has stopped earning its keep and it is time to look upstream at prompt and context size.

What wakes you at 2am when an enterprise operator deploys your agent

Jasmine Park — Thu, 02 Jul 2026 08:13:02 +0000

Month one of our enterprise rollout: three 3am pages. None of them were code bugs. None of them were model quality issues. All of them were operational surprises we had not accounted for before the handoff.

We were production-ready. We were not operator-ready. Those are different things.

Here's what the three incidents were, what they cost, and the alert set I now require before any enterprise agent deployment.

Incident one: rate limits hit at 6pm EST

The operator's usage pattern had a daily spike between 5pm and 7pm EST. Their team was closing out the day, reviewing agent outputs, running follow-up queries. Peak usage was 4x average.

Our OpenAI rate limit was sized for average load. At 6:04pm on day eight, we hit it. Every request after that returned 429 errors for eleven minutes. The agent returned a generic error message. Eleven minutes of silent failure during peak business hours.

Nobody had tested for burst traffic. We had tested for average load. The cost of finding this in production instead of before handoff: one executive complaint, two support tickets, and an emergency weekend call.

The fix was multi-provider routing with automatic failover: when provider A returns 429, route to provider B. We had this wired up within 48 hours. We should have had it before day one.

What this incident taught us about operator-readiness: size for the operator's peak, not your average. The operator's usage pattern will not match your test environment.

Incident two: $11,800 overage in 18 days

The operator had one very active team. Four engineers running batch analysis jobs against our agent at hours that made no operational sense (11pm local, Saturdays). Within 18 days, their usage represented 68% of total API cost.

We had a total cost ceiling. We did not have a per-tenant ceiling. When we hit the total ceiling, the entire deployment slowed down. The heavy team's cost was invisible to us until the month-end bill.

The question from the customer's VP of operations on day 20: "Can you break down the cost by team?"

We could not.

This is a FinOps problem that looks like a monitoring problem. Per-tenant cost tagging needs to be in the request headers before the first request goes out. Not added after month-end when someone asks the question.

Cost of finding this in production instead of before handoff: $11,800 overage, one executive conversation, two weeks of retroactive tagging work to approximate the breakdown.

Minimum viable cost attribution setup: tag every request with operator ID, team ID, and use-case ID at the gateway layer. Aggregate daily by tag. Alert when any single tag hits 70% of its monthly budget by day 15.

Incident three: no audit trail for a compliance request

Six weeks in, the operator's compliance officer needed to reconstruct a decision the agent made on a specific document at a specific time. Customer data question. The agent had made a routing decision that the compliance team needed to audit.

We had trace spans in our observability system. We did not have an immutable per-request audit log that showed: which document, which agent version, which model version, which prompt version, what the output was, what confidence score.

Trace spans are not audit logs. They're operational data. An audit log needs to be write-once, timestamped, and correlated with the business object (the document, the customer record, whatever the operator's domain object is).

We spent four days building a retroactive audit log approximation. It was not what the compliance officer needed, but it was the best we could do. The real audit log went live in week eight.

Cost of finding this in production instead of before handoff: one compliance near-miss, four days of engineering time, one uncomfortable meeting.

What operator-ready means vs. what production-ready means

Production-ready is about your system. Uptime, latency, error rates. Metrics you control.

Operator-ready is about what happens when your system runs inside someone else's organization, with their usage patterns, their cost constraints, their compliance requirements, their data.

The three incidents above were all foreseeable. The operator's usage pattern is discoverable before handoff (just ask them). Per-tenant cost attribution is a gateway configuration decision that takes a day to implement. Audit log requirements for regulated industries are documented in their compliance frameworks.

We didn't discover any of these things before the handoff because we didn't ask the right questions before the handoff.

The pre-operator readiness checklist I run now

Before any enterprise agent deployment, I go through five operational questions. These are not code reviews. They're operational configuration checks.

Rate limit sizing: What is the operator's expected peak usage, and what is our per-provider rate limit at that peak? If peak usage exceeds 60% of the rate limit, configure burst handling or multi-provider routing before go-live.

Cost attribution: Is every request tagged with the minimum attribution set (operator, team, use-case) at the gateway layer? If not, implement it before the first request. Do not add it retroactively.

Audit log schema: Does the operator operate in a regulated industry? If yes, map their compliance requirements to a specific log schema before deployment. Generic trace spans do not satisfy financial services, healthcare, or legal audit requirements.

Failover configuration: Is there a secondary provider configured for automatic failover? If not, is there a documented manual procedure and a stated SLA for the outage window?

Cost ceiling configuration: Is there a per-tenant monthly budget ceiling with an alert at 70% of budget? Not a per-account ceiling. Per tenant. An over-active team should not consume another team's budget.

These five checks take about two hours to complete and review. Three incidents in month one took about three weeks to fully resolve, plus the relationship cost.

What I'd page on

If I were setting up the alert set for a new enterprise agent deployment, these are the four alerts I'd configure first:

Alert one: per-tenant request rate above 80% of the tenant's configured rate limit. Not 100%. Leave 20% headroom to investigate before hitting the ceiling.

Alert two: per-request cost moving average above threshold for any single tenant (set the threshold based on the expected per-request cost, alert at 3x). Catches batch jobs and runaway loops before month-end.

Alert three: agent response with no trace ID in the audit log. Means the audit trail has a gap. You want to know about this in real time, not when a compliance officer asks.

Alert four: first-request p99 latency above 2x the steady-state p99. Catches cold-start regressions before the operator's peak usage hits them.

None of these alerts require custom infrastructure. They require that your gateway and your agent emit the right metadata on every request.

Get the metadata right before handoff. Fix the alerts before go-live. The 3am page is not a production incident. It's a pre-deployment checklist item you deferred.