For four months every deploy of our search API scaled it from fourteen pods to its maximum of sixty within a minute. It stayed there for five minutes, scaled back down, and the deploy itself usually finished late.
The autoscaler scaled on requests per second per pod, read from Prometheus through the metrics adapter. Its query used a recording rule somebody had written to make dashboards faster: the sum of the request counter across all pods, stored as a series of its own. The autoscaler took the rate of that sum.
rate is built for counters, and counters only go up, so it treats any decrease as a reset: the process restarted and the counter began again from zero, and the value after the drop is counted as new increase. For one pod's counter that is exactly right. For a sum across pods it is badly wrong. When a deploy replaced a pod that had served three million requests, the sum fell by three million, rate saw a reset, and counted the remaining forty million as requests that had arrived inside its two minute window. To the autoscaler that was traffic forty times our peak.
The rollout paid for it. The namespace has a resource quota sized for normal peak plus the surge of a rolling update. With sixty pods running, the quota refused the new version's surge pods, the rollout waited, and twice the deploy job reached its timeout and rolled back a perfectly good release.
The fix was to swap two words: rate first, then sum. Each pod's counter resets on its own and rate handles that, and a sum of per pod rates has no resets to misread. Across our recording rules we found nine more taking a rate or an increase of a sum, including the request panel in our capacity report. A check in the rules repository now rejects any rate, irate or increase whose argument is the output of another recording rule. The autoscaler may at most double the replica count per minute, so a wrong number can only be so wrong. And every deploy now annotates the request graphs, so a spike that lines up with a deploy is visible as exactly that.
Counter arithmetic has an order. Sum a set of counters first and every restart inside the set looks, to rate, like the whole fleet starting again from zero.
– Sergey Shinder
Top comments (0)