PART 1 — StatefulSet + Headless Service + Persistent Storage

Goal: Deploy a stateful MySQL cluster correctly in Kubernetes and understand every moving part.

---

LAB PREREQUISITES (DO NOT SKIP)

You need one working Kubernetes cluster:

• Minikube

Verify:

kubectl get nodes

You must see Ready.

---

STEP 0 — WHAT WE ARE BUILDING (VISUAL FIRST)

Final architecture

• 1 Headless Service (identity)
• 1 StatefulSet (mysql pods)
• 3 Pods: mysql-0, mysql-1, mysql-2
• 1 PVC per pod
• Stable DNS names
• Data survives restarts

---

STEP 1 — CREATE A NAMESPACE (CLEAN ISOLATION)

kubectl create namespace mysql-lab
kubectl config set-context --current --namespace=mysql-lab

Verify:

kubectl get ns

---

STEP 2 — HEADLESS SERVICE (MOST IMPORTANT FILE)

File: mysql-headless.yaml

apiVersion: v1
kind: Service
metadata:
  name: mysql-headless
spec:
  clusterIP: None
  selector:
    app: mysql
  ports:
    - port: 3306
      name: mysql

Apply:

kubectl apply -f mysql-headless.yaml

Verify:

kubectl get svc

You must see:

mysql-headless   ClusterIP   None

---

🔍 WHAT THIS DOES (VISUAL)

• ❌ No virtual IP
• ✅ DNS returns individual pod IPs
• ❌ No kube-proxy load balancing
• ✅ Enables pod-level DNS

⚠️ Alone this is NOT ENOUGH — identity still missing

---

STEP 3 — STATEFULSET (THE CORE)

File: mysql-statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  serviceName: mysql-headless
  replicas: 3
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
        - name: mysql
          image: mysql:8.0
          env:
            - name: MYSQL_ROOT_PASSWORD
              value: rootpass
          ports:
            - containerPort: 3306
          volumeMounts:
            - name: data
              mountPath: /var/lib/mysql
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi

Apply:

kubectl apply -f mysql-statefulset.yaml

---

STEP 4 — WATCH POD CREATION (ORDER MATTERS)

kubectl get pods -w

You will see:

mysql-0  → Running
mysql-1  → Running
mysql-2  → Running

🔍 IMPORTANT

• Pods start one by one
• NOT parallel
• This is guaranteed by StatefulSet

---

STEP 5 — VERIFY STABLE POD NAMES

kubectl get pods

Expected:

mysql-0
mysql-1
mysql-2

Restart one pod:

kubectl delete pod mysql-1

Watch:

kubectl get pods -w

Result:

• mysql-1 comes back
• Same name
• Same identity

---

VISUAL: DEPLOYMENT VS STATEFULSET

This is why Deployments are unsafe for databases.

---

STEP 6 — VERIFY PERSISTENT STORAGE

kubectl get pvc

Expected:

data-mysql-0
data-mysql-1
data-mysql-2

Each pod has:

• Its own disk
• Independent lifecycle
• Data survives pod deletion

---

VISUAL: VOLUMECLAIMTEMPLATES

---

STEP 7 — VERIFY DNS (CRITICAL STEP)

Run a temporary client pod:

kubectl run dns-test --image=busybox:1.36 -it --rm --restart=Never -- sh

Inside the pod:

nslookup mysql-0.mysql-headless
nslookup mysql-1.mysql-headless
nslookup mysql-2.mysql-headless

Each resolves to a different IP.

Exit:

exit

---

VISUAL: POD DNS NAMES

Format:

<pod-name>.<headless-service>.<namespace>.svc.cluster.local

---

STEP 8 — CONNECT TO MYSQL (REAL TEST)

kubectl exec -it mysql-0 -- mysql -uroot -prootpass

Inside MySQL:

CREATE DATABASE labdb;
USE labdb;
CREATE TABLE test (id INT);
INSERT INTO test VALUES (1);
SELECT * FROM test;

Exit:

exit

---

STEP 9 — DELETE POD, DATA MUST SURVIVE

kubectl delete pod mysql-0

Wait until it comes back:

kubectl get pods -w

Reconnect:

kubectl exec -it mysql-0 -- mysql -uroot -prootpass

Check:

USE labdb;
SELECT * FROM test;

✅ Data is still there.

---

VISUAL: DATA SURVIVES POD DELETION

---

STEP 10 — WHAT THIS LAB PROVES (LOCK THIS IN)

Kubernetes guarantees:

• Stable pod identity
• Ordered startup & shutdown
• Persistent storage
• Predictable DNS

Kubernetes does NOT:

• Replicate MySQL data
• Promote replicas
• Manage DB internals

---

SENIOR DEVOPS MENTAL MODEL (INTERVIEW READY)

“StatefulSets are required for databases because they provide stable pod identities and ordered lifecycle management. Headless services expose pod-level DNS so clients can target specific replicas. VolumeClaimTemplates ensure each pod has persistent storage that survives restarts.”