Our request graph showed thousands of calls a second because gunicorn had four counters

Our address validation API is a Flask app served by gunicorn with four synchronous workers. We added Prometheus metrics with the official Python client: a counter of requests by status and a histogram of latency, and a /metrics route returning generate_latest. The load balancer said the service handled about forty requests a second. The new dashboard showed a jagged line that spiked to three or four thousand.

For two weeks we believed the spikes were bursts the load balancer was smoothing, and a capacity request went in on the strength of them. Then someone asked why the error ratio occasionally went above one hundred percent.

Each gunicorn worker is a separate process with its own copy of every counter. A scrape of /metrics is just another request, and it is answered by whichever worker accepts it. So each scrape returned one worker's counts, and successive scrapes reached different workers. The counter Prometheus stored went up, then down, then up again.

A counter is only supposed to go down when its process restarts, and rate treats any decrease as exactly that. It assumes the counter started again from zero and adds the whole new value as fresh increase. Our workers had each counted about a million requests, so every time a scrape landed on a worker slightly behind the previous one, a million requests appeared inside a five minute window. Errors and requests were inflated by different workers at different moments, which is how the ratio passed one hundred.

The client has a multiprocess mode for exactly this. We set PROMETHEUS_MULTIPROC_DIR to an empty directory that the container clears at start, and each worker now writes its values to files there. The /metrics route builds a fresh registry with MultiProcessCollector, which reads and sums every worker's files, and gunicorn's child_exit hook calls mark_process_dead so a replaced worker's live gauges are dropped. We added an alert on resets of the request counter over an hour, which for a service that has not restarted should be zero.

The capacity request was withdrawn. Every series on our dashboard had been reporting faithfully on one quarter of the service, a different quarter each time.

– Sergey Shinder