We ran a three-node message broker as a StatefulSet, and for two years every operational habit we applied to it came from Deployments. Scale it down to save money overnight, scale it back in the morning. Delete a stuck pod and let the controller sort it out. Change the storage class and roll it. Each of those is a normal, safe thing to do to a stateless service. Applied to a StatefulSet, they range from slow to destructive.
The incident was the overnight scale-down. Someone had automated it months earlier, and it had worked fine, because scaling from three to one and back to three usually recovers. Then one night the scale-up raced with a node replacement, pod-1 came back with an empty volume because its PVC had been reclaimed by a policy nobody had reviewed, and the broker cluster now had two members that disagreed about who held which partitions. We spent the morning reconciling data by hand.
The mental model that fixed it for us is that a StatefulSet is not a way to run replicas. It's a way to give specific identities durable homes. Pod-1 is not interchangeable with pod-2. It has its own volume, its own place in the cluster's own membership view, and often its own role. When you delete pod-1 you are not removing capacity, you are removing a named member from a quorum, and the application inside has opinions about that which Kubernetes knows nothing about.
Practically: we removed the scale-down automation entirely, because the saving was about forty dollars a month and the risk was the data. Reclaim policy went to Retain, so a deleted claim leaves the disk behind. We stopped deleting pods to fix things and started using the broker's own admin commands to drain and remove a member deliberately before touching the pod. Rolling updates got podManagementPolicy: OrderedReady and a real readiness probe that reports cluster membership, not just process liveness, so the rollout waits for the member to rejoin rather than for the port to open.
The general rule I'd offer: for stateful workloads, orchestration should follow the application's own lifecycle, not replace it. Kubernetes will happily reschedule your data layer. It has no idea what your data means.
– Sergey Shinder
Top comments (0)