DEV Community

Aisalkyn Aidarova
Aisalkyn Aidarova

Posted on

statefulset prod lab

#kubernetes #tutorial #devops #architecture

JumpToTech DevOps Lab

Production-Style StatefulSet + Persistent Storage + Argo CD

Estimated time: 2–3 hours

Level: Intermediate

Technologies: Kubernetes, StatefulSet, PV, PVC, StorageClass, Nginx, Git, GitHub, Argo CD

1. Lab Objective

In this lab you will deploy a stateful workload using Kubernetes and manage it through Argo CD.

By the end of the lab, you should understand this architecture:

GitHub Repository
        |
        v
     Argo CD
        |
        v
   Kubernetes
        |
        v
   StatefulSet
        |
   +----+----+
   |    |    |
 web-0 web-1 web-2
   |    |    |
  PVC  PVC  PVC
   |    |    |
   PV   PV   PV
   |    |    |
   +----+----+
        |
        v
Persistent Storage
Enter fullscreen mode Exit fullscreen mode

You will prove that:

Pod != Persistent Data
Enter fullscreen mode Exit fullscreen mode

A Pod can be destroyed and recreated while its persistent data survives.

2. Important Concepts

Before starting, understand these components.

StatefulSet

A StatefulSet manages Pods that require stable identities.

Instead of names such as:

web-5d8478b7c4-xk29p
web-5d8478b7c4-8h2mm
Enter fullscreen mode Exit fullscreen mode

a StatefulSet creates:

web-0
web-1
web-2
Enter fullscreen mode Exit fullscreen mode

These identities are stable.

If:

web-1
Enter fullscreen mode Exit fullscreen mode

is deleted, StatefulSet recreates:

web-1
Enter fullscreen mode Exit fullscreen mode

rather than creating a randomly named replacement.

3. PV vs PVC vs Storage

Understand this chain:

Pod
 |
 v
PVC
 |
 v
PV
 |
 v
Actual Storage
Enter fullscreen mode Exit fullscreen mode

PVC — PersistentVolumeClaim

PVC is a request for storage.

Example:

I need 2Gi of persistent storage.
Enter fullscreen mode Exit fullscreen mode

PV — PersistentVolume

PV is the Kubernetes resource representing persistent storage.

Actual Storage

The real storage backend could be:

AWS EBS
AWS EFS
Azure Disk
Google Persistent Disk
NFS
Ceph
etc.
Enter fullscreen mode Exit fullscreen mode

For example:

Pod
 ↓
PVC
 ↓
PV
 ↓
AWS EBS
Enter fullscreen mode Exit fullscreen mode

4. StorageClass

StorageClass defines how a class of storage can be dynamically provisioned.

Check available StorageClasses:

kubectl get storageclass
Enter fullscreen mode Exit fullscreen mode

Short version:

kubectl get sc
Enter fullscreen mode Exit fullscreen mode

Look for the default StorageClass:

kubectl get sc
Enter fullscreen mode Exit fullscreen mode

Example:

NAME                 PROVISIONER
standard (default)   ...
Enter fullscreen mode Exit fullscreen mode

The exact output depends on your cluster.

Conceptually:

PVC
 |
 | requests storage
 v
StorageClass
 |
 v
CSI Provisioner
 |
 v
Actual Volume
 |
 v
PV
 |
 v
PVC becomes Bound
Enter fullscreen mode Exit fullscreen mode

This is called:

Dynamic Provisioning

5. Check the Cluster

Run:

kubectl cluster-info
Enter fullscreen mode Exit fullscreen mode

Then:

kubectl get nodes
Enter fullscreen mode Exit fullscreen mode

Expected:

NAME       STATUS   ROLES
node-1     Ready    ...
Enter fullscreen mode Exit fullscreen mode

Do not continue if your nodes are not Ready.

6. Create the Project

Create a directory:

mkdir statefulset-production-lab
cd statefulset-production-lab
Enter fullscreen mode Exit fullscreen mode

Create directories:

mkdir -p k8s
Enter fullscreen mode Exit fullscreen mode

Check:

tree
Enter fullscreen mode Exit fullscreen mode

Expected:

statefulset-production-lab/
└── k8s/
Enter fullscreen mode Exit fullscreen mode

If tree is unavailable:

find .
Enter fullscreen mode Exit fullscreen mode

7. Create Namespace

Create:

vim k8s/namespace.yaml
Enter fullscreen mode Exit fullscreen mode

Add:

apiVersion: v1
kind: Namespace
metadata:
  name: stateful-app
  labels:
    app.kubernetes.io/part-of: stateful-app
Enter fullscreen mode Exit fullscreen mode

Apply:

kubectl apply -f k8s/namespace.yaml
Enter fullscreen mode Exit fullscreen mode

Verify:

kubectl get ns
Enter fullscreen mode Exit fullscreen mode

8. Create ConfigMap

Production applications should separate configuration from the container image.

Create:

vim k8s/configmap.yaml
Enter fullscreen mode Exit fullscreen mode

Add:

apiVersion: v1
kind: ConfigMap
metadata:
  name: web-config
  namespace: stateful-app
data:
  APP_ENV: "production"
  LOG_LEVEL: "info"
Enter fullscreen mode Exit fullscreen mode

Apply:

kubectl apply -f k8s/configmap.yaml
Enter fullscreen mode Exit fullscreen mode

Check:

kubectl get configmap -n stateful-app
Enter fullscreen mode Exit fullscreen mode

9. Create Secret

Create:

vim k8s/secret.yaml
Enter fullscreen mode Exit fullscreen mode

For this training lab:

apiVersion: v1
kind: Secret
metadata:
  name: web-secret
  namespace: stateful-app
type: Opaque
stringData:
  APP_PASSWORD: "jumptotech-lab-password"
Enter fullscreen mode Exit fullscreen mode

Apply:

kubectl apply -f k8s/secret.yaml
Enter fullscreen mode Exit fullscreen mode

Check:

kubectl get secret -n stateful-app
Enter fullscreen mode Exit fullscreen mode

Production note: committing plaintext secrets to Git is not a production secret-management strategy. Real GitOps environments commonly integrate an external secrets solution, cloud secret manager, or encrypted/sealed secret workflow. This file is included only to demonstrate Kubernetes Secret consumption.

10. Create Headless Service

StatefulSets commonly use a Headless Service to provide stable network identities.

Create:

vim k8s/headless-service.yaml
Enter fullscreen mode Exit fullscreen mode

Add:

apiVersion: v1
kind: Service
metadata:
  name: web-headless
  namespace: stateful-app
spec:
  clusterIP: None

  selector:
    app: web

  ports:
    - name: http
      port: 80
      targetPort: 80
Enter fullscreen mode Exit fullscreen mode

Notice:

clusterIP: None
Enter fullscreen mode Exit fullscreen mode

This makes it a:

Headless Service

Architecture:

Headless Service
       |
       +---- web-0
       |
       +---- web-1
       |
       +---- web-2
Enter fullscreen mode Exit fullscreen mode

11. Create Production-Style StatefulSet

Create:

vim k8s/statefulset.yaml
Enter fullscreen mode Exit fullscreen mode

Add:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
  namespace: stateful-app

spec:
  serviceName: web-headless

  replicas: 3

  podManagementPolicy: OrderedReady

  updateStrategy:
    type: RollingUpdate

  selector:
    matchLabels:
      app: web

  template:
    metadata:
      labels:
        app: web

    spec:
      terminationGracePeriodSeconds: 30

      containers:
        - name: nginx
          image: nginx:1.27

          ports:
            - name: http
              containerPort: 80

          envFrom:
            - configMapRef:
                name: web-config

            - secretRef:
                name: web-secret

          resources:
            requests:
              cpu: "100m"
              memory: "128Mi"

            limits:
              cpu: "500m"
              memory: "512Mi"

          readinessProbe:
            httpGet:
              path: /
              port: 80

            initialDelaySeconds: 5
            periodSeconds: 10

          livenessProbe:
            httpGet:
              path: /
              port: 80

            initialDelaySeconds: 15
            periodSeconds: 20

          volumeMounts:
            - name: web-data
              mountPath: /usr/share/nginx/html

  volumeClaimTemplates:

    - metadata:
        name: web-data

      spec:
        accessModes:
          - ReadWriteOnce

        resources:
          requests:
            storage: 1Gi
Enter fullscreen mode Exit fullscreen mode

12. Understand volumeClaimTemplates

This is one of the most important parts:

volumeClaimTemplates:

  - metadata:
      name: web-data
Enter fullscreen mode Exit fullscreen mode

StatefulSet uses this template to create a PVC for each replica.

With:

replicas: 3
Enter fullscreen mode Exit fullscreen mode

you should receive approximately:

web-data-web-0
web-data-web-1
web-data-web-2
Enter fullscreen mode Exit fullscreen mode

Architecture:

web-0
 |
 +---- web-data-web-0
          |
          v
          PV

web-1
 |
 +---- web-data-web-1
          |
          v
          PV

web-2
 |
 +---- web-data-web-2
          |
          v
          PV
Enter fullscreen mode Exit fullscreen mode

Each replica therefore maintains its own storage claim.

13. Create PodDisruptionBudget

Create:

vim k8s/pdb.yaml
Enter fullscreen mode Exit fullscreen mode

Add:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
  namespace: stateful-app

spec:
  minAvailable: 2

  selector:
    matchLabels:
      app: web
Enter fullscreen mode Exit fullscreen mode

The PDB protects application availability during voluntary disruptions, such as certain node-maintenance operations.

It does not prevent every possible Pod failure or deletion.

14. Check All Files

Run:

tree
Enter fullscreen mode Exit fullscreen mode

Expected:

.
└── k8s
    ├── configmap.yaml
    ├── headless-service.yaml
    ├── namespace.yaml
    ├── pdb.yaml
    ├── secret.yaml
    └── statefulset.yaml
Enter fullscreen mode Exit fullscreen mode

15. Validate Before Deployment

Use client-side parsing/validation first:

kubectl apply --dry-run=client -f k8s/
Enter fullscreen mode Exit fullscreen mode

Then inspect:

kubectl diff -f k8s/
Enter fullscreen mode Exit fullscreen mode

kubectl diff may return a non-zero exit code when differences exist; that does not necessarily mean the manifests are invalid.

16. Deploy

Apply:

kubectl apply -f k8s/
Enter fullscreen mode Exit fullscreen mode

Watch:

kubectl get pods -n stateful-app -w
Enter fullscreen mode Exit fullscreen mode

Expected:

web-0
web-1
web-2
Enter fullscreen mode Exit fullscreen mode

Because OrderedReady is configured, StatefulSet normally progresses through ordinal replicas in order as they become Ready.

Press:

CTRL+C
Enter fullscreen mode Exit fullscreen mode

when complete.

17. Inspect StatefulSet

Run:

kubectl get statefulset -n stateful-app
Enter fullscreen mode Exit fullscreen mode

Short version:

kubectl get sts -n stateful-app
Enter fullscreen mode Exit fullscreen mode

Then:

kubectl describe sts web -n stateful-app
Enter fullscreen mode Exit fullscreen mode

Students should identify:

Replicas
Pod Management Policy
Update Strategy
Volume Claims
Selector
Containers
Enter fullscreen mode Exit fullscreen mode

18. Inspect Pods

Run:

kubectl get pods -n stateful-app -o wide
Enter fullscreen mode Exit fullscreen mode

Expected identities:

web-0
web-1
web-2
Enter fullscreen mode Exit fullscreen mode

Compare this with a Deployment:

Deployment:

web-7d8bc9c87c-d7sk2
web-7d8bc9c87c-j82lm


StatefulSet:

web-0
web-1
web-2
Enter fullscreen mode Exit fullscreen mode

19. Inspect PVCs

Run:

kubectl get pvc -n stateful-app
Enter fullscreen mode Exit fullscreen mode

Expected:

NAME             STATUS
web-data-web-0   Bound
web-data-web-1   Bound
web-data-web-2   Bound
Enter fullscreen mode Exit fullscreen mode

The critical status is:

Bound
Enter fullscreen mode Exit fullscreen mode

Now inspect one:

kubectl describe pvc web-data-web-0 -n stateful-app
Enter fullscreen mode Exit fullscreen mode

Look for:

Status
Volume
Capacity
Access Modes
StorageClass
Enter fullscreen mode Exit fullscreen mode

20. Inspect PVs

Run:

kubectl get pv
Enter fullscreen mode Exit fullscreen mode

Notice that PV is cluster-scoped, while PVC is namespaced.

Find the PV bound to web-data-web-0:

kubectl get pvc web-data-web-0 \
  -n stateful-app \
  -o jsonpath='{.spec.volumeName}{"\n"}'
Enter fullscreen mode Exit fullscreen mode

Save it:

PV_NAME=$(kubectl get pvc web-data-web-0 \
  -n stateful-app \
  -o jsonpath='{.spec.volumeName}')
Enter fullscreen mode Exit fullscreen mode

Inspect:

kubectl describe pv "$PV_NAME"
Enter fullscreen mode Exit fullscreen mode

Architecture:

web-0
 |
 v
web-data-web-0
 |
 v
PV
 |
 v
Storage Backend
Enter fullscreen mode Exit fullscreen mode

21. Write Persistent Data

Enter:

kubectl exec -it web-0 \
  -n stateful-app \
  -- sh
Enter fullscreen mode Exit fullscreen mode

Inside:

echo "<h1>JumpToTech Stateful Application</h1>" \
> /usr/share/nginx/html/index.html
Enter fullscreen mode Exit fullscreen mode

Add another file:

echo "This data must survive Pod deletion." \
> /usr/share/nginx/html/data.txt
Enter fullscreen mode Exit fullscreen mode

Check:

cat /usr/share/nginx/html/index.html
Enter fullscreen mode Exit fullscreen mode

Check:

cat /usr/share/nginx/html/data.txt
Enter fullscreen mode Exit fullscreen mode

Exit:

exit
Enter fullscreen mode Exit fullscreen mode

22. Verify Application

Run:

kubectl port-forward \
  pod/web-0 \
  8080:80 \
  -n stateful-app
Enter fullscreen mode Exit fullscreen mode

Open another terminal:

curl localhost:8080
Enter fullscreen mode Exit fullscreen mode

Expected:

<h1>JumpToTech Stateful Application</h1>
Enter fullscreen mode Exit fullscreen mode

Stop port-forward with:

CTRL+C
Enter fullscreen mode Exit fullscreen mode

23. Failure Test

Now deliberately destroy the Pod.

Before:

kubectl get pods -n stateful-app
Enter fullscreen mode Exit fullscreen mode

Delete:

kubectl delete pod web-0 -n stateful-app
Enter fullscreen mode Exit fullscreen mode

Watch:

kubectl get pods -n stateful-app -w
Enter fullscreen mode Exit fullscreen mode

StatefulSet should restore:

web-0
Enter fullscreen mode Exit fullscreen mode

Wait until:

web-0   1/1   Running
Enter fullscreen mode Exit fullscreen mode

24. Verify Data Survived

Run:

kubectl exec web-0 \
  -n stateful-app \
  -- cat /usr/share/nginx/html/data.txt
Enter fullscreen mode Exit fullscreen mode

Expected:

This data must survive Pod deletion.
Enter fullscreen mode Exit fullscreen mode

This proves:

Pod lifecycle
      !=
Persistent storage lifecycle
Enter fullscreen mode Exit fullscreen mode

The Pod was destroyed.

The persistent claim/storage remained available.

The recreated web-0 mounted its storage again.

25. Compare Data Between Replicas

Check web-1:

kubectl exec web-1 \
  -n stateful-app \
  -- ls -la /usr/share/nginx/html
Enter fullscreen mode Exit fullscreen mode

Do not assume it contains the file written to web-0.

Why?

Because:

web-0 → PVC-0 → Storage-0

web-1 → PVC-1 → Storage-1

web-2 → PVC-2 → Storage-2
Enter fullscreen mode Exit fullscreen mode

The replicas have independent persistent claims.

26. Scale Up

Scale:

kubectl scale statefulset web \
  --replicas=5 \
  -n stateful-app
Enter fullscreen mode Exit fullscreen mode

Watch:

kubectl get pods -n stateful-app -w
Enter fullscreen mode Exit fullscreen mode

Expected:

web-0
web-1
web-2
web-3
web-4
Enter fullscreen mode Exit fullscreen mode

Check claims:

kubectl get pvc -n stateful-app
Enter fullscreen mode Exit fullscreen mode

You should now see additional claims for the new replicas.

27. Scale Down

Scale back:

kubectl scale statefulset web \
  --replicas=3 \
  -n stateful-app
Enter fullscreen mode Exit fullscreen mode

Check:

kubectl get pods -n stateful-app
Enter fullscreen mode Exit fullscreen mode

Then:

kubectl get pvc -n stateful-app
Enter fullscreen mode Exit fullscreen mode

Notice the important behavior:

Do not assume scaling down deletes the corresponding PVCs.

StatefulSet storage retention is intentionally conservative because automatically deleting persistent data could be dangerous. StatefulSet also supports PVC retention-policy configuration when a workload explicitly requires different behavior.

28. Examine Stable Network Identity

Run:

kubectl get svc -n stateful-app
Enter fullscreen mode Exit fullscreen mode

You should see:

web-headless
Enter fullscreen mode Exit fullscreen mode

with:

CLUSTER-IP: None
Enter fullscreen mode Exit fullscreen mode

Check DNS from inside the cluster:

kubectl run dns-test \
  --image=busybox:1.36 \
  --restart=Never \
  -n stateful-app \
  -- sleep 3600
Enter fullscreen mode Exit fullscreen mode

Wait:

kubectl wait \
  --for=condition=Ready \
  pod/dns-test \
  -n stateful-app \
  --timeout=60s
Enter fullscreen mode Exit fullscreen mode

Resolve:

kubectl exec dns-test \
  -n stateful-app \
  -- nslookup web-0.web-headless.stateful-app.svc.cluster.local
Enter fullscreen mode Exit fullscreen mode

Try:

kubectl exec dns-test \
  -n stateful-app \
  -- nslookup web-1.web-headless.stateful-app.svc.cluster.local
Enter fullscreen mode Exit fullscreen mode

This demonstrates stable network identity.

Clean up the test Pod:

kubectl delete pod dns-test -n stateful-app
Enter fullscreen mode Exit fullscreen mode

29. Check ConfigMap Environment Variables

Run:

kubectl exec web-0 \
  -n stateful-app \
  -- printenv APP_ENV
Enter fullscreen mode Exit fullscreen mode

Expected:

production
Enter fullscreen mode Exit fullscreen mode

Check:

kubectl exec web-0 \
  -n stateful-app \
  -- printenv LOG_LEVEL
Enter fullscreen mode Exit fullscreen mode

Expected:

info
Enter fullscreen mode Exit fullscreen mode

30. Check Resource Requests and Limits

Run:

kubectl describe pod web-0 -n stateful-app
Enter fullscreen mode Exit fullscreen mode

Find:

Requests:
  cpu
  memory

Limits:
  cpu
  memory
Enter fullscreen mode Exit fullscreen mode

These help Kubernetes make scheduling decisions and enforce container resource limits.

31. Check Probes

Run:

kubectl describe pod web-0 -n stateful-app
Enter fullscreen mode Exit fullscreen mode

Find:

Liveness
Readiness
Enter fullscreen mode Exit fullscreen mode

Readiness Probe

Answers:

Should this Pod currently receive traffic?
Enter fullscreen mode Exit fullscreen mode

Liveness Probe

Answers:

Does Kubernetes consider this container healthy enough to keep running,
or should it restart it?
Enter fullscreen mode Exit fullscreen mode

32. Check PDB

Run:

kubectl get pdb -n stateful-app
Enter fullscreen mode Exit fullscreen mode

Then:

kubectl describe pdb web-pdb -n stateful-app
Enter fullscreen mode Exit fullscreen mode

Remember:

PDB != replica controller
PDB != protection against every failure
PDB != backup
Enter fullscreen mode Exit fullscreen mode

It primarily constrains voluntary disruptions.

PART II — GITOPS WITH ARGO CD

33. Why Argo CD?

So far deployment was performed manually:

Engineer
   |
kubectl apply
   |
   v
Kubernetes
Enter fullscreen mode Exit fullscreen mode

Production GitOps introduces a different operating model:

Engineer
   |
   v
Git
   |
   v
GitHub
   |
   v
Argo CD
   |
   v
Kubernetes
Enter fullscreen mode Exit fullscreen mode

Git becomes the desired-state source.

34. Important GitOps Rule

Once Argo CD owns this application, avoid treating manual kubectl apply as the normal deployment process.

Desired workflow:

Change YAML
   ↓
git add
   ↓
git commit
   ↓
git push
   ↓
Argo CD detects desired-state change
   ↓
Argo CD syncs
   ↓
Kubernetes changes
Enter fullscreen mode Exit fullscreen mode

35. Create .gitignore

Create:

vim .gitignore
Enter fullscreen mode Exit fullscreen mode

Add:

.DS_Store
*.log
.env
Enter fullscreen mode Exit fullscreen mode

Do not put credentials, kubeconfigs, tokens, or private keys in the repository.

36. Git Repository

Initialize:

git init
Enter fullscreen mode Exit fullscreen mode

Create branch:

git checkout -b main
Enter fullscreen mode Exit fullscreen mode

Check:

git status
Enter fullscreen mode Exit fullscreen mode

Add files:

git add .
Enter fullscreen mode Exit fullscreen mode

Commit:

git commit -m "feat: add production-style stateful workload"
Enter fullscreen mode Exit fullscreen mode

37. Push to GitHub

Create an empty repository in your GitHub account.

Example repository name:

statefulset-production-lab
Enter fullscreen mode Exit fullscreen mode

Add your actual remote:

git remote add origin <YOUR-GITHUB-REPOSITORY-URL>
Enter fullscreen mode Exit fullscreen mode

Verify:

git remote -v
Enter fullscreen mode Exit fullscreen mode

Push:

git push -u origin main
Enter fullscreen mode Exit fullscreen mode

The repository should contain:

statefulset-production-lab/
│
├── .gitignore
│
└── k8s/
    ├── namespace.yaml
    ├── configmap.yaml
    ├── secret.yaml
    ├── headless-service.yaml
    ├── statefulset.yaml
    └── pdb.yaml
Enter fullscreen mode Exit fullscreen mode

Again: the demo secret.yaml is for training. Do not use plaintext Git secrets for real production credentials.

38. Install Argo CD

Check first:

kubectl get ns argocd
Enter fullscreen mode Exit fullscreen mode

If Argo CD is already installed, do not reinstall it.

If it is not installed, install Argo CD using the installation method provided for your training cluster.

After installation verify:

kubectl get pods -n argocd
Enter fullscreen mode Exit fullscreen mode

Wait until the required Argo CD components are running.

39. Access Argo CD

For a local/training environment:

kubectl port-forward \
  svc/argocd-server \
  -n argocd \
  8081:443
Enter fullscreen mode Exit fullscreen mode

Then access:

https://localhost:8081
Enter fullscreen mode Exit fullscreen mode

Retrieve the initial admin password when using the standard initial-secret setup:

kubectl -n argocd \
  get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" \
  | base64 -d
Enter fullscreen mode Exit fullscreen mode

Then:

echo
Enter fullscreen mode Exit fullscreen mode

Username:

admin
Enter fullscreen mode Exit fullscreen mode

40. Create Argo CD Application Manifest

Create a new directory:

mkdir argocd
Enter fullscreen mode Exit fullscreen mode

Create:

vim argocd/application.yaml
Enter fullscreen mode Exit fullscreen mode

Add:

apiVersion: argoproj.io/v1alpha1
kind: Application

metadata:
  name: stateful-app
  namespace: argocd

spec:
  project: default

  source:
    repoURL: YOUR_GITHUB_REPOSITORY_URL
    targetRevision: main
    path: k8s

  destination:
    server: https://kubernetes.default.svc
    namespace: stateful-app

  syncPolicy:

    automated:
      prune: true
      selfHeal: true

    syncOptions:
      - CreateNamespace=true
Enter fullscreen mode Exit fullscreen mode

Replace:

YOUR_GITHUB_REPOSITORY_URL
Enter fullscreen mode Exit fullscreen mode

with your repository URL.

41. Important: Avoid the Argo CD Bootstrap Loop

The Argo CD Application manifest itself is stored under:

argocd/application.yaml
Enter fullscreen mode Exit fullscreen mode

but the Application watches only:

path: k8s
Enter fullscreen mode Exit fullscreen mode

Therefore Argo CD manages the Kubernetes workload manifests under k8s/, while the Application definition can be bootstrapped separately.

Commit it:

git add argocd/application.yaml
git commit -m "feat: add Argo CD application"
git push
Enter fullscreen mode Exit fullscreen mode

Bootstrap the Argo CD Application:

kubectl apply -f argocd/application.yaml
Enter fullscreen mode Exit fullscreen mode

42. Check Argo CD

Run:

kubectl get applications -n argocd
Enter fullscreen mode Exit fullscreen mode

Expected eventually:

NAME           SYNC STATUS   HEALTH STATUS
stateful-app   Synced        Healthy
Enter fullscreen mode Exit fullscreen mode

You can also inspect:

kubectl describe application stateful-app -n argocd
Enter fullscreen mode Exit fullscreen mode

43. Understand Automated Sync

We configured:

automated:
  prune: true
  selfHeal: true
Enter fullscreen mode Exit fullscreen mode

Automated sync

Argo CD can automatically synchronize Git desired state into Kubernetes.

prune: true

If a resource managed by the Application is removed from Git, Argo CD can remove that managed resource from the cluster during sync.

selfHeal: true

If a managed Kubernetes resource drifts from Git, Argo CD can restore the Git-defined desired state.

This creates:

              GIT
               |
               |
         desired state
               |
               v
            Argo CD
               |
        compare/reconcile
               |
               v
           Kubernetes
Enter fullscreen mode Exit fullscreen mode

44. GitOps Scaling Test

Do not scale manually for this test.

Edit:

vim k8s/statefulset.yaml
Enter fullscreen mode Exit fullscreen mode

Change:

replicas: 3
Enter fullscreen mode Exit fullscreen mode

to:

replicas: 4
Enter fullscreen mode Exit fullscreen mode

Commit:

git add k8s/statefulset.yaml
git commit -m "scale: increase StatefulSet to four replicas"
git push
Enter fullscreen mode Exit fullscreen mode

Watch:

kubectl get pods -n stateful-app -w
Enter fullscreen mode Exit fullscreen mode

Eventually:

web-0
web-1
web-2
web-3
Enter fullscreen mode Exit fullscreen mode

Check PVC:

kubectl get pvc -n stateful-app
Enter fullscreen mode Exit fullscreen mode

The new replica should receive its corresponding claim.

45. GitOps Self-Healing Test

Git says:

replicas: 4
Enter fullscreen mode Exit fullscreen mode

Now deliberately introduce drift:

kubectl scale statefulset web \
  --replicas=2 \
  -n stateful-app
Enter fullscreen mode Exit fullscreen mode

Check:

kubectl get sts web -n stateful-app
Enter fullscreen mode Exit fullscreen mode

Argo CD should detect that live state differs from Git.

With automated self-healing enabled, it should reconcile toward the desired state stored in Git.

Check:

kubectl get pods -n stateful-app -w
Enter fullscreen mode Exit fullscreen mode

The desired replica count should return to the Git-defined value.

This demonstrates:

Git = Desired State

Cluster = Live State

Argo CD = Reconciliation
Enter fullscreen mode Exit fullscreen mode

46. GitOps Update Test

Change image:

image: nginx:1.27
Enter fullscreen mode Exit fullscreen mode

to a newer approved image tag available for your environment.

Commit:

git add k8s/statefulset.yaml
git commit -m "chore: update nginx image"
git push
Enter fullscreen mode Exit fullscreen mode

Watch:

kubectl rollout status \
  statefulset/web \
  -n stateful-app
Enter fullscreen mode Exit fullscreen mode

Then:

kubectl get pods -n stateful-app
Enter fullscreen mode Exit fullscreen mode

Inspect:

kubectl describe sts web -n stateful-app
Enter fullscreen mode Exit fullscreen mode

47. Check Persistent Data After Update

Check:

kubectl exec web-0 \
  -n stateful-app \
  -- cat /usr/share/nginx/html/data.txt
Enter fullscreen mode Exit fullscreen mode

The original persistent data should still exist if the same PVC/storage remains attached.

This demonstrates the separation between:

Application lifecycle
Container lifecycle
Pod lifecycle
Storage lifecycle
Enter fullscreen mode Exit fullscreen mode

48. Production Architecture

The completed system conceptually looks like:

                         DEVELOPER
                             |
                             |
                         git push
                             |
                             v
                         GITHUB
                             |
                             |
                             v
                         ARGO CD
                             |
                    desired-state sync
                             |
                             v
                     KUBERNETES CLUSTER
                             |
                             v
                        STATEFULSET
                             |
                +------------+------------+
                |            |            |
                v            v            v
              web-0        web-1        web-2
                |            |            |
                v            v            v
              PVC-0        PVC-1        PVC-2
                |            |            |
                v            v            v
               PV-0         PV-1         PV-2
                |            |            |
                +------------+------------+
                             |
                             v
                     STORAGE BACKEND
Enter fullscreen mode Exit fullscreen mode

In an AWS EKS environment using EBS, the lower portion may conceptually be:

StatefulSet
    |
    v
   Pod
    |
    v
   PVC
    |
    v
StorageClass
    |
    v
EBS CSI Driver
    |
    v
AWS EBS Volume
Enter fullscreen mode Exit fullscreen mode

49. Troubleshooting

Pod Pending

Run:

kubectl describe pod web-0 -n stateful-app
Enter fullscreen mode Exit fullscreen mode

Check events:

kubectl get events \
  -n stateful-app \
  --sort-by=.lastTimestamp
Enter fullscreen mode Exit fullscreen mode

PVC Pending

Run:

kubectl get pvc -n stateful-app
Enter fullscreen mode Exit fullscreen mode

Then:

kubectl describe pvc web-data-web-0 -n stateful-app
Enter fullscreen mode Exit fullscreen mode

Check:

kubectl get sc
Enter fullscreen mode Exit fullscreen mode

Potential causes include:

No suitable/default StorageClass
CSI driver unavailable
Provisioning failure
Topology/zone constraints
Storage quota
Access-mode mismatch
Enter fullscreen mode Exit fullscreen mode

StatefulSet Not Ready

Run:

kubectl describe sts web -n stateful-app
Enter fullscreen mode Exit fullscreen mode

Then:

kubectl describe pod web-0 -n stateful-app
Enter fullscreen mode Exit fullscreen mode

Logs:

kubectl logs web-0 -n stateful-app
Enter fullscreen mode Exit fullscreen mode

Argo CD OutOfSync

Check:

kubectl describe application stateful-app -n argocd
Enter fullscreen mode Exit fullscreen mode

Confirm:

repository URL
branch
path
permissions
manifest validity
destination
Enter fullscreen mode Exit fullscreen mode

50. Useful Commands Cheat Sheet 

kubectl get sts -A

kubectl get pods -A

kubectl get pvc -A

kubectl get pv

kubectl get sc

kubectl get svc -A

kubectl get pdb -A

kubectl describe sts web -n stateful-app

kubectl describe pod web-0 -n stateful-app

kubectl describe pvc web-data-web-0 -n stateful-app

kubectl logs web-0 -n stateful-app

kubectl get events -n stateful-app --sort-by=.lastTimestamp

kubectl get applications -n argocd
Enter fullscreen mode Exit fullscreen mode

51. Interview Questions

Students must be able to answer these after completing the lab.

What is a StatefulSet?

A StatefulSet is a Kubernetes workload controller designed for applications whose replicas require stable identities, stable network identities, ordered lifecycle semantics, or stable per-replica storage.

Deployment vs StatefulSet?

Deployment replicas are generally interchangeable.

StatefulSet replicas have stable ordinal identities such as:

web-0
web-1
web-2
Enter fullscreen mode Exit fullscreen mode

and commonly use stable per-replica storage.

What is PVC?

A PersistentVolumeClaim is a request for persistent storage.

What is PV?

A PersistentVolume is a Kubernetes resource representing persistent storage available to workloads.

What is the difference between PV and actual storage?

PV is a Kubernetes resource.

The actual storage backend may be something such as:

AWS EBS
EFS
NFS
Azure Disk
Google Persistent Disk
Enter fullscreen mode Exit fullscreen mode

What is StorageClass?

StorageClass describes a class of storage and how storage may be dynamically provisioned.

What is dynamic provisioning?

Dynamic provisioning allows storage to be provisioned automatically when a PVC requests it rather than requiring an administrator to manually pre-create every PV.

What does volumeClaimTemplates do?

It allows a StatefulSet to create a persistent storage claim for each replica.

Why does StatefulSet use stable identity?

Because individual replicas may need a consistent identity, network name, and association with their own persistent storage.

Does deleting a StatefulSet Pod delete its data?

Not necessarily. Pod lifecycle and persistent storage lifecycle are separate. PVC/PV retention and reclaim behavior determine what happens to persistent storage.

Does StatefulSet automatically configure database replication?

No.

Running:

mysql-0
mysql-1
mysql-2
Enter fullscreen mode Exit fullscreen mode

does not automatically create a properly replicated MySQL cluster.

Database replication, leader election, failover, consistency, backups, and recovery require database-specific configuration or an appropriate Kubernetes Operator/application architecture.

Can Deployment use PVC?

Yes.

Persistent storage is not exclusive to StatefulSet.

StatefulSet is selected when stable per-replica identity and lifecycle/storage relationships are important.

Why use a Headless Service?

It supports direct service discovery and stable DNS identities for StatefulSet replicas rather than providing only one load-balanced Service virtual IP.

What does Argo CD do?

Argo CD continuously compares Kubernetes live state with the desired state stored in Git and can synchronize the cluster to that desired state.

52. Student Assignment

Complete all tasks without copying the finished StatefulSet from another student.

Your final environment must contain:

Namespace
ConfigMap
Secret
Headless Service
StatefulSet
3+ replicas
PVC per replica
PV-backed persistent storage
Resource requests
Resource limits
Readiness probe
Liveness probe
PodDisruptionBudget
Git repository
Argo CD Application
Automated synchronization
Self-healing
Persistent-data test
Enter fullscreen mode Exit fullscreen mode

Perform and document these tests:

1. Delete web-0.
2. Prove web-0 is recreated.
3. Prove its persistent data survives.
4. Show all PVCs.
5. Identify the PV bound to web-0.
6. Scale through Git.
7. Show Argo CD synchronizing the change.
8. Introduce replica-count drift manually.
9. Show Argo CD self-healing.
10. Explain Pod → PVC → PV → actual storage.
Enter fullscreen mode Exit fullscreen mode

53. Required Submission

Submit:

GitHub repository link
Enter fullscreen mode Exit fullscreen mode

and screenshots/output showing:

kubectl get sts -n stateful-app

kubectl get pods -n stateful-app

kubectl get pvc -n stateful-app

kubectl get pv

kubectl get sc

kubectl get svc -n stateful-app

kubectl get pdb -n stateful-app

kubectl get applications -n argocd
Enter fullscreen mode Exit fullscreen mode

Also submit:

kubectl exec web-0 \
  -n stateful-app \
  -- cat /usr/share/nginx/html/data.txt
Enter fullscreen mode Exit fullscreen mode

after deleting and recreating web-0.

Final Knowledge Map

Every student should be able to reproduce this from memory:

                    GIT
                     |
                     v
                  ARGO CD
                     |
                     v
                 KUBERNETES
                     |
                     v
                STATEFULSET
                     |
          +----------+----------+
          |          |          |
        web-0      web-1      web-2
          |          |          |
          v          v          v
        PVC-0      PVC-1      PVC-2
          |          |          |
          v          v          v
         PV-0       PV-1       PV-2
          |          |          |
          +----------+----------+
                     |
                     v
              ACTUAL STORAGE
Enter fullscreen mode Exit fullscreen mode

Memorize:

StatefulSet  = stable workload identity

PVC          = request for storage

PV           = Kubernetes persistent-storage resource

StorageClass = how/class through which storage is provisioned

CSI Driver   = Kubernetes-to-storage-system integration

EBS/EFS/etc. = actual storage backend

Argo CD      = GitOps reconciliation

Git          = desired state
Enter fullscreen mode Exit fullscreen mode

Production principle:

Pods are replaceable.

Persistent data must not depend on
the lifetime of a particular container.

Git defines desired Kubernetes state.

Argo CD continuously reconciles
the cluster toward that state.
Enter fullscreen mode Exit fullscreen mode

Top comments (0)