We had spent a quarter rolling out distributed tracing. Context propagation through eleven services, a collector, a backend with a bill attached to it. Then a customer reported checkout timeouts, I opened the trace explorer to find one, filtered to errors on that endpoint for the previous day, and got nothing. Not a slow trace. Not a broken one. Zero results, from a system carrying about eleven million requests a day.
The sampler was head-based at one percent. That decision is made at the first service, on the root span, before anything has happened, using nothing but a random number. Our timeout rate on that endpoint was roughly four in ten thousand. One percent of four in ten thousand is a number small enough that over a day we would expect a couple of matching traces, and we happened to have none. Every trace we had stored was a successful request completing in ninety milliseconds, which is the one thing we never needed to look at.
I had understood sampling as a cost control and never thought about what it selects for. Uniform random sampling gives you a representative sample of your traffic, and your traffic is overwhelmingly boring. The events worth storing are by definition the rare ones, so the sampling strategy that saves the most money is precisely the one that discards the entire value of the system.
We moved the decision to the end. Tail sampling in the collector buffers the spans of a trace, waits for it to complete, then decides: keep every trace containing an error, keep every trace over the endpoint's p99 latency, keep one percent of the rest. That required running the collector as a stateful layer with consistent routing by trace ID so all spans of a trace reach the same instance, which was the real work and took two weeks.
Two smaller things mattered as much. The root service now honours an explicit sampling decision, so a support engineer can force a trace for a specific customer session. And our latency histograms carry exemplars, so clicking a spike on the dashboard jumps to a trace that actually produced it.
Storage went up eighteen percent. We can now open an incident and find the request.
– Sergey Shinder
Top comments (0)