DEV Community

Cover image for Kubernetes on AKS: From Zero to Production Workloads - StatefulSets and Persistent Storage (Part 6/9)
Rahimah Sulayman
Rahimah Sulayman

Posted on

Kubernetes on AKS: From Zero to Production Workloads - StatefulSets and Persistent Storage (Part 6/9)

#statefulset #kubernetes #aks #azure

Introduction

Applications that maintain data need more than a reliable way to run containers. They also need predictable identities, stable network addresses, and storage that remains available when Pods are replaced.

In this mission, I deployed a StatefulSet on Azure Kubernetes Service (AKS), paired it with a headless Service, and provisioned persistent Azure Disk volumes for its replicas. I then wrote data from inside a Pod, deleted that Pod, and verified that the replacement could still access the data. Finally, I tested the StatefulSet's DNS identity from another Pod in the cluster.

The objective was to demonstrate how Kubernetes supports stateful workloads whose identity and data must survive individual Pod replacement.

What This Part Covers

  • Inspecting the available Kubernetes StorageClasses.
  • Creating a headless Service.
  • Defining a StatefulSet with two replicas.
  • Provisioning persistent storage through managed-csi.
  • Verifying Pod identities and PersistentVolumeClaims.
  • Writing data to a Pod's persistent volume.
  • Deleting and waiting for a StatefulSet Pod to return.
  • Verifying that data survives Pod replacement.
  • Testing stable DNS access through the headless Service.

1. Inspect the Available StorageClasses

Before defining the StatefulSet, I inspected the StorageClasses available in the cluster:

kubectl get storageclass
Enter fullscreen mode Exit fullscreen mode

storageclass

The output showed several StorageClasses for Azure Files and Azure Disk. The managed-csi StorageClass was available for persistent disk provisioning.

Its relevant properties included:

Property Value
StorageClass managed-csi
Provisioner disk.csi.azure.com
Reclaim policy Delete
Volume binding mode WaitForFirstConsumer
Volume expansion Enabled

The managed-csi StorageClass uses the Azure Disk Container Storage Interface (CSI) driver. It allows Kubernetes to provision Azure-managed disks for workloads that request persistent storage.

The WaitForFirstConsumer binding mode delays volume binding and provisioning until a Pod using the claim is scheduled. This allows storage provisioning to account for the Pod's scheduling requirements.

The reclaim policy is also important. Delete means that a dynamically provisioned volume can be deleted when its PersistentVolume is released, depending on the lifecycle of the associated storage resources. Deleting an individual StatefulSet Pod does not, by itself, release its PersistentVolumeClaim.

2. Define a Headless Service and StatefulSet

I created manifests/06-statefulset.yaml to define both the headless Service and the StatefulSet.

cat << 'EOF' > manifests/06-statefulset.yaml
apiVersion: v1
kind: Service
metadata:
  name: store
spec:
  clusterIP: None
  selector:
    app: store
  ports:
    - port: 80
      targetPort: 80
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: store
spec:
  serviceName: store
  replicas: 2
  selector:
    matchLabels:
      app: store
  template:
    metadata:
      labels:
        app: store
    spec:
      containers:
        - name: nginx
          image: nginx:1.27
          ports:
            - containerPort: 80
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
            limits:
              cpu: 200m
          volumeMounts:
            - name: data
              mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: managed-csi
        resources:
          requests:
            storage: 1Gi
EOF
Enter fullscreen mode Exit fullscreen mode

statefulSet

Why use a headless Service?

The Service is configured with:

clusterIP: None
Enter fullscreen mode Exit fullscreen mode

This makes it a headless Service. Instead of providing a conventional cluster IP, it enables DNS records to resolve directly to the endpoints of the selected Pods.

The Service selects Pods labelled app: store and exposes port 80. It also provides the network identity associated with the StatefulSet.

Why use a StatefulSet?

The StatefulSet is named store and declares two replicas:

serviceName: store
replicas: 2
Enter fullscreen mode Exit fullscreen mode

Unlike ordinary Deployment-managed Pods, StatefulSet Pods have stable ordinal identities. The two replicas use these names:

  • store-0
  • store-1

The StatefulSet also uses the store headless Service as its governing Service, allowing the Pods to have predictable DNS identities.

Persistent storage

The manifest defines a volumeClaimTemplates entry named data. Each replica receives its own PersistentVolumeClaim requesting 1Gi of storage from the managed-csi StorageClass.

The important settings are:

accessModes:
  - ReadWriteOnce
storageClassName: managed-csi
resources:
  requests:
    storage: 1Gi
Enter fullscreen mode Exit fullscreen mode

The container mounts the volume at:

/usr/share/nginx/html
Enter fullscreen mode Exit fullscreen mode

This is the directory NGINX serves as its default document root. Writing a file there allows the test to demonstrate whether the contents remain available after a Pod is replaced.

Because each StatefulSet replica receives its own claim, store-0 and store-1 do not share the same per-replica volume.

3. Deploy the StatefulSet

I applied the manifest:

kubectl apply -f manifests/06-statefulset.yaml
Enter fullscreen mode Exit fullscreen mode

Kubernetes confirmed that both resources were created:

service/store created
statefulset.apps/store created
Enter fullscreen mode Exit fullscreen mode

I then monitored the StatefulSet rollout:

kubectl rollout status statefulset/store
Enter fullscreen mode Exit fullscreen mode

The command reported:

Waiting for 1 pods to be ready...
partitioned roll out complete: 2 new pods have been updated...
Enter fullscreen mode Exit fullscreen mode

After the rollout, I inspected the StatefulSet, Pods, and PersistentVolumeClaims:

kubectl get statefulset,pods,pvc -l app=store
Enter fullscreen mode Exit fullscreen mode

The two Pods were running:

NAME        READY   STATUS    RESTARTS
pod/store-0 1/1     Running   0
pod/store-1 1/1     Running   0
Enter fullscreen mode Exit fullscreen mode

The PersistentVolumeClaims were also bound:

NAME                    STATUS   CAPACITY   ACCESS MODES   STORAGECLASS
persistentvolumeclaim/data-store-0   Bound   1Gi   RWO   managed-csi
persistentvolumeclaim/data-store-1   Bound   1Gi   RWO   managed-csi
Enter fullscreen mode Exit fullscreen mode

verify pvc

This verified that the StatefulSet had created both replicas and Kubernetes had provisioned a separate 1Gi persistent volume claim for each one.

The stable Pod names and individual claims are central to the StatefulSet model: a replacement for store-0 is associated with the identity and storage claim belonging to store-0, rather than being treated as an entirely unrelated replica.

4. Write Data to the Persistent Volume

To test persistence, I wrote a message to the NGINX document root inside store-0:

kubectl exec store-0 -- sh -c 'echo "Hello from $(hostname), written at $(date)" > /usr/share/nginx/html/index.html'
Enter fullscreen mode Exit fullscreen mode

kubectl exec

The command writes a message containing the Pod's hostname and the current date into index.html.

Because the document root is backed by the persistent volume mounted at /usr/share/nginx/html, the test data is stored on the volume rather than only in the container's writable layer.

This creates a clear verification point: after the Pod is replaced, the file should still contain the original message if the same persistent claim is attached successfully.

5. Delete the Pod and Wait for Replacement

I deleted store-0 to test the StatefulSet's recovery behavior:

kubectl delete pod store-0
Enter fullscreen mode Exit fullscreen mode

delete pod

Kubernetes confirmed that the Pod had been deleted from the k8s-lab namespace.

The StatefulSet controller then worked to restore the desired replica count. Because store-0 is part of a StatefulSet, Kubernetes recreates the Pod with the same ordinal identity and reuses its associated PersistentVolumeClaim.

I waited for the replacement Pod to become ready:

kubectl wait --for=condition=Ready pod/store-0 --timeout=180s
Enter fullscreen mode Exit fullscreen mode

The command returned:

pod/store-0 condition met
Enter fullscreen mode Exit fullscreen mode

kubectl wait

This confirmed that store-0 had returned to the Ready state.

6. Verify That the Data Survived

With store-0 ready again, I inspected the file:

kubectl exec store-0 -- cat /usr/share/nginx/html/index.html
Enter fullscreen mode Exit fullscreen mode

The output was:

Hello from store-0, written at Mon Oct 26 09:56:05 UTC 2026
Enter fullscreen mode Exit fullscreen mode

kubectl exec

The message remained available after the original Pod had been deleted and replaced.

This confirmed the key behavior being tested: the replacement Pod retained access to the data stored on the PersistentVolume associated with store-0.

The Pod itself is replaceable, but its identity and associated persistent storage allow the application to recover without losing the file created during the test.

This does not mean that every storage configuration automatically survives every failure scenario. Data persistence depends on the volume lifecycle, reclaim policy, storage backend, and how the claims and volumes are managed. In this test, the StatefulSet's per-replica claim remained available through Pod replacement.

7. Verify Stable DNS from Another Pod

The headless Service also provides predictable network identities for the StatefulSet Pods.

To verify DNS access, I launched a temporary BusyBox Pod and requested the content through the hostname store-0.store:

kubectl run dns-test --rm -i --restart=Never \
  --image=busybox:1.36 -- wget -qO- http://store-0.store
Enter fullscreen mode Exit fullscreen mode

The response contained:

Hello from store-0, written at Mon Oct 26 09:56:05 UTC 2026
Enter fullscreen mode Exit fullscreen mode

dns

The temporary Pod returned the same content written before store-0 was deleted.

This verified two aspects of the implementation:

  1. The name store-0.store resolved and allowed access to the NGINX endpoint.
  2. The endpoint served the data that persisted across the Pod replacement.

The store-0.store hostname is derived from the StatefulSet Pod identity and its governing headless Service. This predictable naming is particularly useful for applications that need to communicate with specific replicas.

The temporary dns-test Pod was configured with --rm, so Kubernetes removed it when the command completed.

8. StatefulSet vs. Deployment

StatefulSets and Deployments both manage replicated workloads, but they address different operational requirements.

Capability Deployment StatefulSet
Pod identity Typically interchangeable Pod names Stable ordinal names such as store-0
Stable per-replica storage Not provided automatically through a Deployment's usual model Supported through volumeClaimTemplates
Common use cases Stateless web applications and APIs Stateful applications requiring stable identity or storage
Scaling and replacement Reconciles desired replica count Reconciles replicas while preserving ordinal identity
Networking Commonly accessed through a Service Can use a headless Service for per-Pod DNS identities

A StatefulSet is not automatically the right choice for every application that writes files. The appropriate workload resource depends on whether the application needs stable identity, ordered operations, or persistent storage tied to individual replicas.

In this mission, those features made the StatefulSet a suitable way to demonstrate data persistence across Pod replacement.

9. Operational Takeaways

This mission demonstrated how StatefulSets, headless Services, and persistent storage work together on AKS.

The main takeaways were:

  • Stable workload identity: StatefulSet replicas have predictable names that remain associated with their ordinals when Pods are replaced.
  • Persistent storage: Each replica receives its own PersistentVolumeClaim through volumeClaimTemplates.
  • Dynamic provisioning: The managed-csi StorageClass allows Kubernetes to provision Azure Disk storage for the claims.
  • Pod recovery: Deleting store-0 caused the StatefulSet controller to recreate the same logical replica.
  • Data continuity: The file written before deletion remained available after the replacement Pod became ready.
  • Stable DNS: A headless Service enables per-Pod DNS identities, allowing another Pod to access store-0 through store-0.store.
  • Storage lifecycle awareness: Persistent volume behavior depends on the claim and volume lifecycle, access modes, storage class, and reclaim policy—not only on the Pod resource.

Conclusion

The store StatefulSet was deployed with two replicas, a headless Service, and separate 1Gi persistent volume claims backed by the managed-csi StorageClass.

I wrote a file to store-0, deleted the Pod, waited for the replacement to become ready, and verified that the original file content remained available. A request to http://store-0.store from a temporary Pod returned the same content, confirming that the stable DNS identity and persistent storage were working together as intended.

This is the primary distinction between simply recreating a container and managing a stateful workload: the replacement can recover with its expected identity and retain access to its data.

In Part 7, the series will continue with the next mission, building on the workload and storage capabilities established here.

This article is Part 6 of the nine-part series, “Kubernetes on AKS: From Zero to Production Workloads.”

Top comments (1)

Collapse
 
rahimah_dev profile image
Rahimah Sulayman •

Stateful workloads require more than restarting containers, they need predictable identity and reliable data persistence. In this mission, the StatefulSet, headless Service, and Azure Disk-backed PersistentVolumeClaims worked together to preserve store-0's identity and data across Pod replacement. Verifying the stored content and accessing it through stable DNS confirmed how these Kubernetes primitives support stateful applications on AKS.