DEV Community

Aisalkyn Aidarova
Aisalkyn Aidarova

Posted on

MINI PROJECT: Headless Service with StatefulSet (MySQL-style behavior)

#devops #kubernetes #networking #tutorial

🧪

🎯 Goal

By the end, you will clearly see:

  • Why ClusterIP hides pod identity
  • Why Headless Service exposes pod identity
  • How StatefulSet + Headless Service gives stable DNS per pod
  • Why databases need this

🧠 Mental model (keep this in mind)

Setup DNS result
Deployment + ClusterIP One virtual IP
Deployment + Headless Multiple pod IPs
StatefulSet + Headless Stable pod DNS names (this is the magic)

🧩 Project structure 

headless-demo/
├── mysql-headless.yaml
├── mysql-statefulset.yaml
└── dns-test.yaml
Enter fullscreen mode Exit fullscreen mode

1️⃣ Headless Service (NO Cluster IP)

📄 mysql-headless.yaml

apiVersion: v1
kind: Service
metadata:
  name: mysql-headless
spec:
  clusterIP: None            # 👈 THIS MAKES IT HEADLESS
  selector:
    app: mysql
  ports:
    - port: 3306
Enter fullscreen mode Exit fullscreen mode

📌 Important:

  • No virtual IP
  • DNS will return pod IPs

2️⃣ StatefulSet (stable pod identity)

📄 mysql-statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  serviceName: mysql-headless   # 👈 REQUIRED
  replicas: 3
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
        - name: mysql
          image: mysql:8.0
          env:
            - name: MYSQL_ROOT_PASSWORD
              value: root
          ports:
            - containerPort: 3306
Enter fullscreen mode Exit fullscreen mode

📌 What StatefulSet guarantees:

  • Pods named: 
  mysql-0
  mysql-1
  mysql-2
Enter fullscreen mode Exit fullscreen mode
  • Names NEVER change
  • Identity is stable

3️⃣ DNS test pod (to observe behavior)

📄 dns-test.yaml

apiVersion: v1
kind: Pod
metadata:
  name: dns-test
spec:
  containers:
    - name: dns
      image: busybox:1.28
      command: ["sleep", "3600"]
Enter fullscreen mode Exit fullscreen mode

4️⃣ Apply everything (ORDER MATTERS) 

kubectl apply -f mysql-headless.yaml
kubectl apply -f mysql-statefulset.yaml
kubectl apply -f dns-test.yaml
Enter fullscreen mode Exit fullscreen mode

Wait:

kubectl get pods
Enter fullscreen mode Exit fullscreen mode

You should see:

mysql-0   Running
mysql-1   Running
mysql-2   Running
dns-test  Running
Enter fullscreen mode Exit fullscreen mode

5️⃣ 🔥 THE MOST IMPORTANT PART — DNS BEHAVIOR

Enter the test pod 

kubectl exec -it dns-test -- sh
Enter fullscreen mode Exit fullscreen mode

🔍 DNS lookup of the HEADLESS service 

nslookup mysql-headless
Enter fullscreen mode Exit fullscreen mode

✅ You will see MULTIPLE IPs (one per pod):

Address: 10.244.0.12
Address: 10.244.0.13
Address: 10.244.0.14
Enter fullscreen mode Exit fullscreen mode

👉 This is DNS round-robin.

🔥 DNS lookup of INDIVIDUAL PODS (THIS IS THE KEY) 

nslookup mysql-0.mysql-headless
nslookup mysql-1.mysql-headless
nslookup mysql-2.mysql-headless
Enter fullscreen mode Exit fullscreen mode

✅ Each one resolves to a specific pod IP.

This is what ClusterIP can NEVER do.

6️⃣ Why databases NEED this

Imagine:

Role DNS
Primary DB mysql-0.mysql-headless
Replica 1 mysql-1.mysql-headless
Replica 2 mysql-2.mysql-headless

Now:

  • Writes → mysql-0.mysql-headless
  • Reads → replicas
  • Replication → stable target

💡 This is impossible with Deployment + ClusterIP.

7️⃣ Visual intuition (what’s happening)

Image

Image

8️⃣ One-line interview answer (remember this)

“A headless service removes the virtual IP and exposes pod identities via DNS. Combined with StatefulSets, it enables stable per-pod DNS, which is required for databases and leader-follower architectures.”

🔥 SAME APP, TWO SERVICES

🟦 CASE 1 — ClusterIP Service (DEFAULT BEHAVIOR)

YAML (normal service) 

apiVersion: v1
kind: Service
metadata:
  name: mysql-clusterip
spec:
  selector:
    app: mysql
  ports:
    - port: 3306
Enter fullscreen mode Exit fullscreen mode

What Kubernetes does

  • Creates ONE virtual IP
  • Hides all pod IPs
  • kube-proxy load balances traffic

DNS behavior

Inside the cluster:

nslookup mysql-clusterip
Enter fullscreen mode Exit fullscreen mode

Result:

Name: mysql-clusterip
Address: 10.96.120.15   <-- ONE IP
Enter fullscreen mode Exit fullscreen mode

That IP is NOT a pod.
It’s a virtual service IP.

🔁 What happens to traffic 

todo-app
   |
   v
mysql-clusterip (10.96.120.15)
   |
   +--> mysql-0
   +--> mysql-1
   +--> mysql-2
Enter fullscreen mode Exit fullscreen mode

Kubernetes decides where each request goes.

❌ Why this breaks databases

Let’s say:

  1. Signup request → mysql-2 (write happens there)
  2. Login request → mysql-0 (no data)
  3. ❌ login fails

Because:

  • write and read hit different pods
  • pod identity is hidden
  • you cannot say “always go to mysql-0”

ClusterIP is stateless-friendly, stateful-hostile.

🟨 CASE 2 — Headless Service (NO CLUSTER IP)

Now we change only one line.

YAML (headless service) 

apiVersion: v1
kind: Service
metadata:
  name: mysql-headless
spec:
  clusterIP: None   # 👈 THIS IS EVERYTHING
  selector:
    app: mysql
  ports:
    - port: 3306
Enter fullscreen mode Exit fullscreen mode

🔍 DNS behavior (this is the key difference)

Inside a pod:

nslookup mysql-headless
Enter fullscreen mode Exit fullscreen mode

Result:

Address: 10.244.0.10   (mysql-0)
Address: 10.244.0.11   (mysql-1)
Address: 10.244.0.12   (mysql-2)
Enter fullscreen mode Exit fullscreen mode

⚠️ No virtual IP
⚠️ DNS returns pod IPs directly

This is DNS round-robin, not kube-proxy load balancing.

🧠 Still not enough alone (important)

If you stop here and use:

mysql-headless
Enter fullscreen mode Exit fullscreen mode

Your app may still hit different pods, because DNS answers rotate.

So headless alone is not the full solution.

🟩 ENTER STATEFULSET (THIS IS THE MISSING PIECE)

StatefulSet guarantees stable pod names:

mysql-0
mysql-1
mysql-2
Enter fullscreen mode Exit fullscreen mode

And Kubernetes automatically creates DNS records like:

mysql-0.mysql-headless
mysql-1.mysql-headless
mysql-2.mysql-headless
Enter fullscreen mode Exit fullscreen mode

🔥 THIS IS THE MOMENT IT CLICKS

Now you can do this:

Write traffic 

mysql-0.mysql-headless
Enter fullscreen mode Exit fullscreen mode

Read traffic 

mysql-1.mysql-headless
mysql-2.mysql-headless
Enter fullscreen mode Exit fullscreen mode

No guessing. No randomness.

📊 SIDE-BY-SIDE SUMMARY (MEMORIZE THIS)

Feature ClusterIP Headless
Has virtual IP ✅ ❌
Hides pod identity ✅ ❌
DNS returns 1 IP Multiple pod IPs
Pod-specific DNS ❌ ✅
Good for web apps ✅ ❌
Good for databases ❌ ✅ (with StatefulSet)

🧪 Why we used dns-test pod

You cannot “see DNS” from your laptop.

So we create a pod just to ask:

nslookup <service-name>
Enter fullscreen mode Exit fullscreen mode

That’s how SREs debug real prod issues.

🎯 ONE-LINE INTERVIEW ANSWER (IMPORTANT)

“ClusterIP services hide pod identity and load balance traffic, which is unsuitable for databases. Headless services expose pod IPs via DNS, and when combined with StatefulSets, provide stable per-pod DNS required for stateful workloads.”

Image

Image

Top comments (0)