A product manager asked what one feature had cost us in April. I pulled the number from our trace backend, divided by the sample rate, and gave her a figure just over eighty dollars. She came back a week later with the provider invoice broken out by API key, which we had wired per feature the previous year. That feature had cost $238.
Mine was the wrong one, by nearly 3x. The invoice is ground truth and my estimate was extrapolated from 10 percent of requests, uniformly sampled, decided at the head of each trace. For that feature, in that month, 10 percent was not enough.
I had set that sample rate myself, eighteen months earlier, for a good reason. Full-fidelity traces on 4.2 million LLM requests a month were not affordable. Ten percent was a round number that made the bill work. What I had never done is ask what 10 percent uniform sampling does to the specific questions we point observability at, which turn out to be almost entirely questions about tails.
Why the estimate was that bad
LLM request cost is heavy-tailed and my sampled estimator did not care.
For a high-volume feature the estimate is fine. Six hundred thousand requests a month, 10 percent kept, the mean of the sample converges and the extrapolation is within a couple of percent. The feature in question did about 9,000 requests in April, so the sample held roughly 900 of them. Its cost distribution had two populations: 36 requests that dragged a full contract into context, around 400,000 input tokens on a premium tier, at about $6.00 each, and 8,964 that cost around a quarter of a cent. Two hundred and sixteen dollars of document calls, twenty-two dollars of everything else.
The sample contained one of the 36. Scaled up by ten, that produced my $82. The expected count was 3.6, and the chance of drawing exactly one is 9.0 percent, so this was not a freak month. Drawing a 6 sits at 8.3 percent, near enough the same, and it would have handed the same PM $382, sixty percent above the invoice, with exactly the same confidence. The estimator overshoots as readily as it undershoots. April happened to undershoot.
The arithmetic is worth doing once. Per-request cost across those two populations has a mean near 2.6 cents and a standard deviation near 38 cents, so the coefficient of variation is about 14. Relative standard error on a scaled-up total is roughly that CV divided by the square root of the sampled count, which at n equals 900 is around 48 percent. In dollars, the standard error on the monthly total was about $114 against a true total of $238. My miss was $156, which is under one and a half standard errors. That is the part that should worry anyone reading this: a 3x error was not an anomaly. It was an ordinary draw from a number I had been reporting as a fact.
Nothing about this is specific to LLMs. It is the ordinary behavior of a ratio estimator on a skewed population. What is specific to LLMs is that the skew is enormous and the thing we most want to count sits in the tail.
Head sampling decides before it knows anything
Our OpenTelemetry collector made the keep-or-drop decision on the first span, hashing the trace ID. That is what head-based sampling is, and it is why it is cheap.
It also means the decision cannot depend on cost, latency, token count, or whether the request errored, because none of those exist yet. The six-dollar request and the quarter-cent request are equally likely to be kept. That is fair in the sense a coin is fair, and it is exactly backwards for an investigation. I am not looking for a representative request. I am looking for the one that cost six dollars or took forty seconds, and head sampling keeps those at exactly the same rate as everything else.
The third problem followed from the second and was the one that actually made people stop trusting the system. Support escalates a bad answer with a request ID. Nine times in ten the trace was gone. We measured it at 91 percent rather than the 90 the sample rate implies, and I never chased the extra point. We had built an observability platform with a one-in-ten chance of answering the only question anyone ever asked it, and the team's rational response was to stop asking.
What it costs, honestly
The reason nobody had revisited the sample rate is that everyone assumed the alternative was 100 percent traces at 30-day retention, which is expensive and obviously so. That framing is the mistake. Three different questions were being served by one pipeline, and they have different shapes.
At 4.2M requests a month, our mean full trace with prompt and completion bodies is 34 KB. That is 143 GB a month at full fidelity. A metrics-only record, no payloads, holding request ID, feature, model, input and output token counts, latency, status and a cost field, is 210 bytes. At 100 percent that is under a gigabyte.
10% uniform, head-based (what we had): traces per month 420,000, ingested per month 14.3 GB, retention 30 days, error traces kept 10%, requests over $0.05 kept 10%
Metrics census, no payloads: traces per month n/a, ingested per month 0.9 GB, retention 13 months, error traces kept n/a, requests over $0.05 kept n/a
Full fidelity, short retention: traces per month 4,200,000, ingested per month 143 GB (9.5 GB standing), retention 48 hours, error traces kept 100%, requests over $0.05 kept 100%
Tail-based keep-rules: traces per month 248,000, ingested per month 16.0 GB, retention 30 days, error traces kept 100%, requests over $0.05 kept 100%
The census answers every cost, volume, latency and error-rate question by counting rather than estimating. The April question now has an exact answer and it reconciles against the invoice to within rounding. This is the path that removed the problem this post opens with, and it costs less than a gigabyte a month.
The 48-hour tier is the one I would argue hardest for, because it is the one I nearly missed. Escalations arrive fast. Ours had a median age of about four hours from the user hitting the problem to a request ID landing in our queue, and the ninetieth percentile was inside a day. Holding every trace for 48 hours in cheap storage is 143 GB of ingest a month but only about 9.5 GB standing, because everything expires. Trace-lookup on an escalation went from 9 percent to essentially 100 percent inside the window, and the residual is escalations older than two days, which fall back to the rules below.
The obvious question, and the one I would ask of anyone else's diagram, is why both trace tiers exist. If you already hold 100 percent of completed traces for 48 hours, you can evaluate the keep-rules as a scheduled query over data you already have and extend the TTL on whatever matches, which deletes the in-flight sampler and everything downstream of it in this post. The reason we did not is that the two tiers are different storage. The 48-hour tier is flat unindexed objects, which is why it is cheap; the 30-day tier lives in the queryable backend. Promoting a trace between them after the fact means paying the indexing ingest a second time, per trace, forever. Deciding in flight costs a buffer. Deciding later costs a second write into the expensive store, and the second one is the larger bill at our volume. If your long-term store is cheap to write, invert this without hesitation.
The keep-rules decide what earns 30-day retention, after the trace completes: all errors, 0.7 percent of requests; everything over 8 seconds, 1.1 percent; everything over 5 cents, 2.3 percent; plus a 3 percent uniform baseline so "normal" stays visible. Those overlap, and the union is 5.9 percent.
The row I got wrong in my first version of this analysis is the last one, and I want to show the error rather than the corrected table alone, because it is the same error as the one in the opening. I originally costed tail-based at 8.4 GB by multiplying 248,000 traces by the 34 KB corpus mean. But the rules deliberately select long multi-step agent runs and large-document requests, so a rule-kept trace is not an average trace. Measured, they mean 96 KB. Costing the baseline at 34 KB and the rule-selected 2.9 percent at 96 KB gives 16.0 GB, not 8.4.
So tail-based is not cheaper than the 10 percent uniform sampling it replaced. It is about 12 percent more expensive on the 30-day tier, and it takes error and expensive-request coverage from 10 percent to 100. I had briefly believed the better story, which was that we would pay less and see more, and I believed it because I had used a mean trace size on a population selected for being large. Same mistake, one layer down.
And 12 percent is the number for one row, so I should give the aggregate before anyone else does. Across all four paths this pipeline ingests 0.9 plus 143 plus 16.0, about 160 GB a month against the 14.3 it replaced. That is 11x the bytes through the collector and the export path. Standing bytes are the gentler number, roughly 37 GB against 14.3, because the big tier expires in two days. That is the actual shape of the trade: storage at rest stays close to rounding error, and what genuinely went up 11x is the write path, so what you are really buying is collector capacity and egress. Size that before you turn it on. We did not, and our collector pool was the first thing to fall over.
The part that bit us
Tail-based sampling has to buffer. The collector holds the spans of an in-flight trace until the trace ends, then decides, and that buffer has two limits: a time limit and a count limit. I assumed both were the problem. Only one was.
Start with the limit that was not the problem. I assumed we were overflowing the trace-count ceiling and that adding capacity would fix it. The arithmetic says otherwise: 4.2M requests a month is 1.6 traces a second, and that is a ceiling rather than a count, because most of our traffic is one call per trace while an agent session bundles several into one. So at a 30-second decision wait we held at most about 49 traces in flight against a num_traces default of 50,000. You cannot fill a fifty-thousand-trace ceiling with forty-nine traces. Nothing was overflowing, and at this volume nothing on this pipeline has ever been dropped for space.
The real defect is in what a policy sees when the timer fires. The latency policy measures a trace's duration as the earliest span start to the latest span end among the spans it is holding at that moment. For a trace that has finished, that is the trace's duration, which is what you want. For a trace still running, it is however long the collector has been holding it, which is the decision wait.
So at decision_wait: 30s, every session still open at thirty seconds reported a duration of about thirty seconds, cleared our 8-second threshold, and was kept. Not because it was slow. Because it was still going. The keep-rule was measuring my own timeout. And that kept nothing extra. A session still open at thirty seconds runs past thirty seconds, so it is over the 8-second threshold whenever you evaluate it, and the slow rule keeps it either way. All the early decision changed was when.
What it changed instead is which rule did the keeping, and that is the real defect. Every policy is evaluated against the spans the processor is holding at decision time. status_code scans those spans for an error status; numeric_attribute scans them for the cost attribute. Decide at thirty seconds and neither can see an error that fires at minute four, or the cost of the eleven model calls that had not happened yet. Those sessions were retained regardless, because the latency rule had already caught them for being long.
So on every session over thirty seconds, two of my four keep-rules were dead, and the pipeline looked completely healthy because a third rule was quietly covering for them. Nothing in the retention numbers moves, which is exactly why it survives review. It only surfaces the day somebody tightens the latency threshold, or drops the rule because the traces it keeps look redundant, and error and cost coverage on long sessions falls to the 3 percent baseline without a single alert firing.
The fix is to stop deciding on unfinished traces. The processor can trigger the decision off the root span closing instead of off a fixed timer, which is what tail sampling was supposed to be doing in the first place.
processors:
tail_sampling:
decision_wait: 600s # a ceiling, longer than any session we run
decision_wait_after_root_received: 5s # decide 5s AFTER the session's root span closes, so the
# latency rule sees a finished trace instead of my timeout
num_traces: 20000 # ~1.6 traces/sec x ~305s = ~490 in flight. headroom, not a
# real constraint at our volume. size from YOUR arrival rate.
policies:
- name: errors
type: status_code
status_code: {status_codes: [ERROR]}
- name: slow
type: latency
latency: {threshold_ms: 8000}
- name: expensive
type: numeric_attribute
numeric_attribute: {key: llm.cost_micros, min_value: 50000} # 50,000 micros = $0.05.
# llm.cost_micros is ours: the GenAI semantic conventions
# define token-usage attributes and no cost attribute at all.
- name: baseline
type: probabilistic
probabilistic: {sampling_percentage: 3}
The decision now lands a few seconds after the session actually ends, on a complete trace, so every rule is evaluated against the whole thing and the error and cost policies can see what they were written to see. Buffer occupancy goes to roughly 490 traces, which is still nothing against 20,000, but work that out for your own arrival rate before copying either number: at a hundred times our volume the same config is the difference between a comfortable pool and an out-of-memory loop.
The other thing this pipeline needs, and the first thing I look for in anyone else's setup, is that every span of a trace reaches the same collector instance. Tail sampling cannot decide on a trace it only partly holds, and the processor's own documentation is explicit that all spans for a given trace must arrive at the same instance. Behind a normal load balancer they do not. That means a two-layer deployment: a front tier running the load-balancing exporter, routing by trace ID, and a back tier running the tail sampler. We ran one tier for most of a sprint and could not work out why the error policy kept firing on fragments.
We also stopped relying on the trace being whole. Whatever the sampler does, the census record for that request still exists on path one, keyed by request ID, so cost and latency questions never depend on a trace surviving intact. That separation is what makes the rest of this survivable.
What I'd page on
Sampled-total drift: alert when a feature's trace-derived monthly cost and its provider-invoice cost differ by more than 15 percent. It is the check that would have caught this eighteen months earlier, and it costs one scheduled query.
Late-span age, as a histogram, not the dropped-too-early counter. This is the correction I most want to pass on. sampling_trace_dropped_too_early is a trace-count overflow metric, so at our volume it reads zero permanently and I spent a week treating that zero as reassurance. The late-span age histogram records how long after a decision the rest of a trace turns up, which tells you how far past decision_wait your sessions actually run, which is the number you needed in order to set it. Alert on the upper percentiles moving, and read the histogram when you tune.
Split-trace rate: spans arriving at a collector instance for a trace whose root landed elsewhere. Non-zero means your load-balancing tier is misrouting and every keep-rule below it is deciding on fragments.
And a warn rather than a page: trace-lookup miss rate on support escalations, bucketed by escalation age. Inside 48 hours it should be near zero now. Outside 48 hours it will not be, and knowing where that line sits is more useful than a single number. Before any of this, ours was 91 percent and nobody had measured it.
Top comments (0)