We shipped a routine update to a busy service. Standard rolling deploy, nothing exotic. Within ninety seconds the database it talked to fell over, connections maxed out, and a service that had nothing to do with our change started throwing errors. We hadn't touched the database. We hadn't touched the other service. And we took them both down anyway.
Here's the mechanism. During a rolling update, Kubernetes briefly runs old and new pods at the same time — that's the whole point, no downtime. But our new pods opened a fresh connection pool on startup before the old ones drained theirs. For a window of maybe thirty seconds, we had nearly double the usual pod count, each holding a full pool of database connections. The database's connection limit didn't care about our good intentions. It hit the ceiling, started refusing connections, and everything sharing that database went down with us.
The lesson that stuck: a deploy isn't an isolated event. It's a transient spike in resource demand, and the resources it spikes are often shared with things you didn't think you were touching. Connection pools, rate limits on a downstream API, cache capacity, disk on a shared node — a rollout momentarily doubles your footprint on all of them.
The fixes were unglamorous and effective. We set maxSurge low so the rollout adds pods a few at a time instead of doubling. We added a preStop hook and honest connection draining so old pods release their pools before new ones grab theirs. And we sized the database connection limit against peak pod count during a deploy, not steady state — because steady state isn't when you fall over, the rollout is.
The bigger shift was learning to see shared, finite resources as the real constraint on how we deploy. Your rollout strategy isn't just about your service's availability. It's a promise about how much of every shared thing you'll consume while you transition, and something downstream is quietly counting on you to keep it.
Before you tune a rollout, ask what it briefly doubles, and who else is standing under that ceiling with you.
– Sergey Shinder
Top comments (0)