A marketing email went out at ten on a Thursday and traffic roughly tripled over four minutes. Latency rose, the autoscaler did what it exists to do, and the API went from six pods to thirty-four. Error rate went up rather than down, and the errors all said the same thing: the database was out of connection slots.
The arithmetic is embarrassing when you write it out. Each pod holds a pool with a maximum of twenty connections, and it opens them under load. Thirty-four pods is six hundred and eighty. The instance allows five hundred, and it was already sharing those with a worker fleet, two internal tools and an analytics job. Nothing in our configuration knew this. The pool size was set per pod, the replica ceiling was set per deployment, and no file anywhere multiplied the two together.
Worse, the loop closed. Requests were failing and slow, so the autoscaler added pods, so more connections were demanded, so more requests failed. The scaling response made the shortage it was reacting to. For about twenty minutes we made the situation worse automatically, faster than anyone could read a dashboard.
The immediate fix at the time was to cap the replica ceiling by hand, which restored service in about three minutes and is still the first thing I would do.
Afterwards, three things. A connection pooler in transaction mode now sits in front of the database, so hundreds of client connections multiplex onto a few dozen server ones and a pod holding an idle connection costs almost nothing. Pool size is derived from a budget rather than chosen: a service is allocated a share of the database's connections, and the per pod maximum is that share divided by the replica ceiling, computed in the chart so the two numbers can never drift apart again. And connection usage is graphed against the limit with an alert at seventy percent, which is now on the same dashboard as the autoscaler's own metrics.
We also load test at the replica ceiling rather than at expected load, because what we needed to know was not whether thirty-four pods could serve the traffic. It was whether everything behind them could survive thirty-four pods existing.
Scaling out only helps when the scarce thing is the thing you are duplicating. Ours moved the shortage into a shared resource no autoscaler was watching.
– Sergey Shinder
Top comments (0)