Kubernetes Cookbook Part 9: StatefulSets for Apps That Remember

So far in this series, every Pod you have run has been interchangeable. A Deployment does not care if it replaces Pod A with Pod B, because the app inside holds no state of its own. That assumption breaks the moment you run something that remembers: a database, a message queue, a distributed cache. Those apps need a stable name, stable network identity, and storage that belongs to them and nobody else. In this part you will cook with StatefulSets, the workload controller built exactly for apps that remember.

What you need

• A local cluster (kind or minikube) from Part 1.
• Working knowledge of Deployments and Services from Part 4 and Part 5.
• Storage basics from Part 8, especially PersistentVolumeClaims (PVCs). StatefulSets build directly on them.
• kubectl configured to talk to your cluster.
• A helper Pod for DNS lookups; we use nicolaka/netshoot below.

Why not just use a Deployment for a database? Two reasons. First, a Deployment gives Pods random names like web-7c9d4f6b8-x1v2t, and the name changes every time the Pod restarts. Database clusters need peers to find each other at a fixed address. Second, a Deployment mounts one shared PVC (or none at all). A database replica must own its own disk, and that disk must survive a Pod being deleted. StatefulSets solve both problems.

Recipe 1: A StatefulSet with a headless Service

Problem: You want three replicas that each get a predictable name and a stable DNS (Domain Name System) entry, so other Pods can reach a specific replica directly.

A StatefulSet always works together with a headless Service — a Service with clusterIP: None. The Service does not load-balance; it only creates DNS records, one per Pod. Create nginx-statefulset.yaml:

apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  clusterIP: None
  selector:
    app: nginx
  ports:
    - port: 80
      name: web
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  serviceName: nginx
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.27
          ports:
            - containerPort: 80
              name: web
          command:
            - /bin/sh
            - -c
            - |
              echo "Hello from $(hostname)" > /usr/share/nginx/html/index.html
              nginx -g "daemon off;"

Apply it and watch the Pods come up:

kubectl apply -f nginx-statefulset.yaml
kubectl get pods -w -l app=nginx

NAME    READY   STATUS    RESTARTS   AGE
web-0   1/1     Running   0          12s
web-1   1/1     Running   0          8s
web-2   1/1     Running   0          4s

Notice the names: web-0, web-1, web-2. The StatefulSet name plus an ordinal, starting at zero. No random suffix. Now verify the per-Pod DNS from a debug Pod:

kubectl run dns-test --rm -it --image=nicolaka/netshoot -- bash
nslookup web-0.nginx.default.svc.cluster.local
curl web-1.nginx.default.svc.cluster.local

Server:    10.96.0.10
Address:   10.96.0.10#53

Name:   web-0.nginx.default.svc.cluster.local
Address: 10.244.0.7

How it works: Every Pod in a StatefulSet gets a DNS record of the form <pod-name>.<service-name>.<namespace>.svc.cluster.local. The headless Service tells Kubernetes not to allocate a virtual IP for the Service itself. Instead, DNS resolves each Pod name straight to that Pod's IP address. The serviceName field on the StatefulSet is mandatory — it ties the StatefulSet to the headless Service that publishes these records. Each replica prints its own hostname into its index page, so curling each Pod's DNS name returns a different answer. That is the whole point: the replicas are individuals, not clones.

Recipe 2: Per-Pod storage with volumeClaimTemplates

Problem: Each replica needs its own disk, created automatically, and the data must survive the Pod being deleted.

You could not do this with a plain Deployment: one PVC in a Deployment is shared by every replica (or impossible to scale with ReadWriteOnce volumes). StatefulSets add volumeClaimTemplates, a cookie cutter that stamps out one PVC per Pod. Extend your StatefulSet with a template and a mount:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  serviceName: nginx
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.27
          ports:
            - containerPort: 80
              name: web
          volumeMounts:
            - name: data
              mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 100Mi

Apply it, then write distinct data into each Pod:

kubectl apply -f nginx-statefulset.yaml
kubectl exec web-0 -- sh -c 'echo "data belonging to web-0" > /usr/share/nginx/html/index.html'
kubectl exec web-1 -- sh -c 'echo "data belonging to web-1" > /usr/share/nginx/html/index.html'
kubectl exec web-2 -- sh -c 'echo "data belonging to web-2" > /usr/share/nginx/html/index.html'

Now the interesting part. Delete web-1 and watch what happens:

kubectl delete pod web-1
kubectl get pods -l app=nginx -w
kubectl exec web-1 -- cat /usr/share/nginx/html/index.html

data belonging to web-1

The Pod came back with the same name and its old data. List the PVCs the StatefulSet created for you:

kubectl get pvc

NAME           STATUS   VOLUME     CAPACITY   STORAGECLASS   AGE
data-web-0     Bound    pvc-a1b2   100Mi      standard       5m
data-web-1     Bound    pvc-c3d4   100Mi      standard       5m
data-web-2     Bound    pvc-e5f6   100Mi      standard       5m

How it works: For each Pod, the StatefulSet creates a PVC named <template-name>-<pod-name> — here data-web-0 and friends. Deleting a Pod does not delete its PVC. When the StatefulSet recreates web-1, it reattaches the existing data-web-1 volume, so the file you wrote is still there. The Pod is disposable; the identity and the disk are not. That pairing of stable name plus stable storage is the core of what a StatefulSet buys you.

Recipe 3: Orderly scaling

Problem: You want to add replicas, but a database cluster must join new members one at a time, not all at once.

StatefulSets scale in strict order. Scale up to five and watch:

kubectl scale statefulset web --replicas=5
kubectl get pods -l app=nginx -w

web-0   1/1     Running   0          20m
web-1   1/1     Running   0          20m
web-2   1/1     Running   0          20m
web-3   1/1     Running   0          6s
web-4   0/1     Pending   0          2s

web-4 does not even start until web-3 is Running and Ready. Now scale back down and watch the reverse:

kubectl scale statefulset web --replicas=3
kubectl get pods -l app=nginx -w

web-4 terminates first, fully, before web-3 is touched. Lowest ordinals are the last to die.

How it works: The default behavior is called OrderedReady. A Pod with ordinal N is only created after N-1 is ready, and Pods are terminated in reverse ordinal order. This matters enormously for clustered software: a new database replica must finish catching up before the next one joins, and shutdown order often matters for graceful leader election.

Variations:

• If your replicas truly do not depend on each other, set podManagementPolicy: Parallel in the StatefulSet spec. All Pods start and stop at once, trading order for speed.
• Scaling down does not delete the PVCs of the removed Pods. Scale back up later and the data is still waiting.

Recipe 4: Careful updates with partitions

Problem: You want to roll out a new image slowly, canarying it on one replica before touching the rest.

StatefulSet rolling updates go from the highest ordinal down: web-2 updates first, then web-1, then web-0. The partition field lets you stop partway. Any Pod with an ordinal greater than or equal to the partition gets updated; the rest stay on the old version.

kubectl patch statefulset web --type='json' -p='[
  {"op": "replace", "path": "/spec/updateStrategy/rollingUpdate/partition", "value": 2},
  {"op": "replace", "path": "/spec/template/spec/containers/0/image", "value": "nginx:1.29"}
]'

Delete web-2 to trigger its update (the controller only recreates a Pod above the partition when it is deleted or restarted):

kubectl delete pod web-2
kubectl get pod web-2 -o jsonpath='{.spec.containers[0].image}'

nginx:1.29

Only web-2 moved. If the new version looks healthy, lower the partition to let the rest update:

kubectl patch statefulset web --type='json' \
  -p='[{"op": "replace", "path": "/spec/updateStrategy/rollingUpdate/partition", "value": 0}]'

How it works: With partition 2, the controller treats ordinals 0 and 1 as fenced off. It updates web-2, waits for it to be Ready, and stops. Setting the partition back to 0 resumes the normal ordered rollout. This gives you a built-in canary: the least important replica (by convention, the highest ordinal) takes the risk first. Two other strategies exist for updateStrategy.type: OnDelete, which never touches a Pod until you delete it yourself (full manual control), and the default RollingUpdate you just used.

Recipe 5: The cleanup gotcha

Problem: You are done with the StatefulSet, so you delete it — and your data silently survives.

kubectl delete statefulset web
kubectl delete service nginx
kubectl get pvc

NAME           STATUS   CAPACITY   AGE
data-web-0     Bound    100Mi      1h
data-web-1     Bound    100Mi      1h
data-web-2     Bound    100Mi      1h

The PVCs are still there. This is deliberate: Kubernetes refuses to guess whether your data is safe to destroy. If you are sure you want it gone, delete the PVCs explicitly:

kubectl delete pvc -l app=nginx

(The StatefulSet labels each template-created PVC with the Pod's labels, so -l app=nginx selects them.)

How it works: A StatefulSet owns its Pods but treats PVCs as precious cargo. Deleting the controller orphans the claims rather than cascading the deletion. On a real cluster with real data, that one design choice has saved many careers.

Variations and warnings:

• Running PostgreSQL or Redis by hand with raw StatefulSets is a great way to learn, and a rough way to run production. Real databases on Kubernetes are usually managed by Operators — higher-level controllers that handle failover, backups, and upgrades for you. Look at CloudNativePG for PostgreSQL or the Bitnami Helm charts for a batteries-included start.
• Never point two StatefulSets at the same PVC naming scheme; you will get confusing mount conflicts.

Where to go next

You now know how to run apps that remember: stable names, per-Pod storage, ordered scaling, and careful updates. In the final part of the series, you will lock all of it down — Part 10: Security and RBAC covers Role-Based Access Control (RBAC), ServiceAccounts, and the principle of least privilege. For the full StatefulSet field reference, the official docs are excellent: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/. The whole series lives at /series/kubernetes-cookbook.