One slow count query took every metric our dispatch service had off the dashboard

During a database slowdown in September our dispatch service failed about one request in six for twenty minutes, and our error rate alert never fired. When we opened the dashboard afterwards, it had a gap over exactly those twenty minutes. No data at all, on every panel the service had.

The service exposes its metrics for Prometheus to scrape every fifteen seconds, with the default scrape timeout of ten. Two years earlier we had added a gauge for the number of dispatches waiting to go out, implemented as a custom collector that ran a count query against the database each time the endpoint was read. Every one of eight pods ran it, and both of our Prometheus replicas scraped every pod, so the database answered that same question about once a second. On a normal day it took a third of a second.

When the database slowed down that query took around fourteen seconds. Prometheus gave up at ten, and a scrape that times out is a failed scrape, all of it. The target's up series went to zero, and Prometheus marked every series it had collected from that target as stale, the request counters and latency histograms as well as the one gauge that was slow. Our error rate alert divided one rate by another, and with no samples on either side there was nothing to compare with the threshold. The alert that did fire was TargetDown, five minutes in, routed as a ticket because a missing target is usually a deploy.

Meanwhile sixteen count queries every fifteen seconds were adding load to a database that was already struggling.

The pending count now comes from one place. A small background task in a single instance runs the query every thirty seconds with its own timeout and publishes the result, and the service's metrics endpoint serves only values it already holds in memory. We alert when scrape_duration_seconds for any target passes half its timeout. And TargetDown for a service carrying live traffic now pages, because we learned it can be the only thing still speaking.

A metrics endpoint that depends on the system it measures will go quiet exactly when that system is in trouble, and it can take everything else it reports down with it.

– Sergey Shinder