During a forty minute outage I went looking for the request that started it and found a hole. Logs from 14:02 to 14:31 were present but thin, and for two of our busiest services there was nothing at all between 14:06 and 14:24. Not errors. Not empty results. An absence, in the exact window that mattered, from the exact services that were failing.
The log agent had done what it was configured to do. Each node ran a shipper with a per-source rate limit and a memory buffer of a few tens of megabytes. Normally we produce a modest, boring volume. Once the failures started, every service began logging stack traces at a rate maybe thirty times higher, and the client retries multiplied that again. The buffer filled in seconds. The shipper's overflow policy was drop_newest, and the backend was applying its own ingest quota at the same time, rejecting batches with a 429 that the agent counted and then discarded.
The part that stung is that the agent had been telling us. It exported a dropped-events counter that had been non-zero for months during smaller spikes, and nothing was watching it. We were monitoring whether the logging pipeline was up, never whether it was complete.
What we changed: the shipper now writes to a disk buffer sized in gigabytes instead of a memory buffer sized in megabytes, so a backend slowdown becomes lag rather than loss. Dropped events and buffer utilisation are alerting metrics, and lag over two minutes pages. Sampling is now deliberate rather than accidental: repeated identical stack traces collapse to one line per second with a count, which cuts volume during exactly the storms that used to cause drops. And anything carrying a request id at error level is exempt from sampling, because those are the lines you actually reconstruct an incident from.
We also stopped assuming that gaps mean quiet. Absence of logs is now its own alert, per service, based on a floor of expected volume.
A telemetry pipeline that degrades under load degrades when you need it. Test yours the way you would test a service: push it thirty times harder than normal and find out what it throws away.
– Sergey Shinder
Top comments (0)