The orders API had a p95 of nine hundred milliseconds. It is supposed to be around two hundred and fifty, and the number in my head from the last time I had looked properly was two hundred and forty. The escalation asked one question, and it took most of a day to answer: when did this start, and what did we do?
Nothing had failed. Nothing had alerted, because the latency alert is set at two seconds and the error budget was intact. The service was slower than it used to be, and that is a category of problem our monitoring had no opinion about.
Dating it was archaeology. The history was there, so the step was findable once somebody plotted thirty days rather than the six hours every dashboard opens on: a clean jump on a Thursday afternoon, five weeks earlier. Then we had to establish what happened on that Thursday, and that meant four different systems. Deploy history in the CI tool. Flag changes in a product tool with a thirty day audit log, exportable as a CSV. Infrastructure applies in a pipeline's run history. And database changes in a ticket queue, which is where it eventually turned up: a DBA had enabled an audit logging parameter during a compliance push, which added a synchronous write to every transaction.
No part of that was anybody's fault. The change was legitimate, reviewed, and announced in a channel. It simply was not recorded anywhere a graph could see it.
What we built afterwards is unglamorous. Deploys, flag flips, infrastructure applies and database changes all publish an event to one stream, and every dashboard draws that stream as annotations, so a step and its cause appear in the same picture. A weekly job posts each service's p50, p95, error rate and throughput beside the previous week's with the percentage difference, into the team channel. In its first month that report found two more regressions, one of them a cache that had stopped being used.
And we keep thirteen months of a small set of series through recording rules, because a comparison against last quarter is worth nothing if the data stops at fifteen days.
Our monitoring was built to answer whether the system is bad right now. Most of what it has cost us is being worse than it was, and nothing anywhere was asking that.
– Sergey Shinder
Top comments (0)