Pods Deployments ReplicaSets StatefulSets Storage Troubleshooting

🚀 JumpToTech Kubernetes Core Hands-On Lab

Objective

By the end of this lab, you should understand:

• Control Plane
• Worker Nodes
• Pods
• Containers
• Deployment
• ReplicaSet
• Self-healing
• Scaling
• Labels and selectors
• StatefulSet
• Stable Pod identity
• PersistentVolume (PV)
• PersistentVolumeClaim (PVC)
• StorageClass
• Basic Kubernetes troubleshooting

---

PART 1 — Understand Your Kubernetes Cluster

Before creating anything, check the cluster.

Run:

kubectl cluster-info

Then:

kubectl get nodes

Now:

kubectl get nodes -o wide

Question

What are these nodes?

Answer

Nodes are machines participating in the Kubernetes cluster.

Application workloads normally run on Worker Nodes.

A simplified architecture looks like this:

                    KUBERNETES CLUSTER

                       CONTROL PLANE
                 ┌─────────────────────┐
                 │ API Server          │
                 │ Scheduler           │
                 │ Controller Manager  │
                 │ etcd                │
                 └──────────┬──────────┘
                            │
                 ───────────┼──────────
                            │
                ┌───────────┴───────────┐
                ↓                       ↓
          WORKER NODE 1           WORKER NODE 2
          ┌────────────┐          ┌────────────┐
          │ kubelet    │          │ kubelet    │
          │ Pod        │          │ Pod        │
          │ Container  │          │ Container  │
          └────────────┘          └────────────┘

Important components

API Server

Receives Kubernetes API requests.

For example:

kubectl get pods

and:

kubectl apply -f deployment.yaml

communicate with the Kubernetes API.

Scheduler

Decides which Worker Node should run a newly created unscheduled Pod.

Controller Manager

Runs controllers that continuously work to make the actual state match the desired state.

etcd

Stores Kubernetes cluster state.

kubelet

Runs on each Worker Node and makes sure assigned Pods are running.

---

PART 2 — Create a Standalone Pod

Create a new directory:

mkdir kubernetes-core-lab
cd kubernetes-core-lab

Create:

nano pod.yaml

Add:

apiVersion: v1
kind: Pod

metadata:
  name: nginx-pod
  labels:
    app: nginx

spec:
  containers:
    - name: nginx
      image: nginx:1.27
      ports:
        - containerPort: 80

Save the file.

Apply it:

kubectl apply -f pod.yaml

Check:

kubectl get pods

You should see:

NAME        READY   STATUS    RESTARTS   AGE
nginx-pod   1/1     Running   0          ...

Now:

kubectl get pods -o wide

Question

What additional information do you see?

You should see information including:

IP
NODE

The NODE column tells you which Worker Node is running the Pod.

---

PART 3 — Inspect the Pod

Run:

kubectl describe pod nginx-pod

Find:

Node:
Labels:
Status:
IP:
Containers:
Events:

Important

kubectl describe is one of the most important troubleshooting commands.

When something is wrong, always look at:

Events

near the bottom.

Now check logs:

kubectl logs nginx-pod

---

PART 4 — Delete the Standalone Pod

Run:

kubectl delete pod nginx-pod

Check:

kubectl get pods

Question

Did the Pod come back?

Answer

No.

Why?

Because we created a standalone Pod.

There is no higher-level workload controller responsible for maintaining that Pod.

This demonstrates an important production concept:

Standalone Pod

DELETE POD
    ↓
Pod disappears
    ↓
Nothing recreates it

This is one reason production applications are normally managed through higher-level workload resources such as Deployments rather than manually creating individual Pods.

---

PART 5 — Create a Deployment

Now create:

nano deployment.yaml

Add:

apiVersion: apps/v1
kind: Deployment

metadata:
  name: nginx-deployment

spec:
  replicas: 3

  selector:
    matchLabels:
      app: nginx

  template:
    metadata:
      labels:
        app: nginx

    spec:
      containers:
        - name: nginx
          image: nginx:1.27
          ports:
            - containerPort: 80

Apply:

kubectl apply -f deployment.yaml

Check:

kubectl get deployments

Then:

kubectl get rs

Then:

kubectl get pods

---

PART 6 — Understand What Kubernetes Created

We created only:

Deployment

But Kubernetes created:

Deployment
     ↓
ReplicaSet
     ↓
Pods

Run:

kubectl get deployment

Then:

kubectl get rs

Then:

kubectl get pods

Question

Did we manually create the ReplicaSet?

Answer

No.

The Deployment created and manages the ReplicaSet.

Question

Did we manually create the individual Pods?

Answer

No.

The ReplicaSet maintains the required Pod replicas.

Conceptually:

Deployment
     │
     │ manages
     ↓
ReplicaSet
     │
     │ maintains
     ↓
┌─────────┐
│ Pod #1  │
├─────────┤
│ Pod #2  │
├─────────┤
│ Pod #3  │
└─────────┘

---

PART 7 — Desired State

Look inside deployment.yaml:

replicas: 3

This tells Kubernetes:

DESIRED STATE = 3 Pods

Check:

kubectl get deployment nginx-deployment

You may see something similar to:

NAME               READY   UP-TO-DATE   AVAILABLE
nginx-deployment   3/3     3            3

Kubernetes continuously tries to make:

ACTUAL STATE
     =
DESIRED STATE

This process is called reconciliation.

---

PART 8 — Test Kubernetes Self-Healing

Get Pods:

kubectl get pods

Copy one Deployment Pod name.

Example:

nginx-deployment-7c79c4bf97-abcd1

Delete it:

kubectl delete pod nginx-deployment-7c79c4bf97-abcd1

Immediately run:

kubectl get pods

Better:

kubectl get pods -w

Watch carefully.

You should see a replacement Pod.

Press:

CTRL+C

to stop watching.

What happened?

Before:

Desired = 3
Actual  = 3

We deleted one:

Desired = 3
Actual  = 2

Kubernetes detects the difference.

The ReplicaSet works to restore:

Desired = 3
Actual  = 3

Interview Question

Why did the Pod come back?

Answer

Because the Deployment's ReplicaSet maintains the desired number of replicas. When one Pod disappeared, the ReplicaSet created a replacement.

---

PART 9 — Scale the Deployment

Currently:

replicas: 3

Scale to 6:

kubectl scale deployment nginx-deployment --replicas=6

Check:

kubectl get deployment

Then:

kubectl get pods

Question

How many Pods?

Answer:

6

Now:

kubectl get rs

The ReplicaSet should show the desired/current replica counts.

Now scale down:

kubectl scale deployment nginx-deployment --replicas=2

Watch:

kubectl get pods -w

Kubernetes terminates extra Pods.

Stop with:

CTRL+C

---

PART 10 — Labels and Selectors

Run:

kubectl get pods --show-labels

You should see labels including:

app=nginx

Now:

kubectl get pods -l app=nginx

Why are labels important?

Labels allow Kubernetes resources to identify/select other resources.

Our Deployment contains:

selector:
  matchLabels:
    app: nginx

and the Pod template contains:

labels:
  app: nginx

These must match for this Deployment.

Think:

Deployment selector
        ↓
    app=nginx
        ↓
   Matching Pods

---

PART 11 — Deployment Update

Check the current image:

kubectl get deployment nginx-deployment \
  -o jsonpath='{.spec.template.spec.containers[0].image}'

Now update Nginx:

kubectl set image deployment/nginx-deployment \
  nginx=nginx:1.28

Watch:

kubectl get pods -w

Stop with:

CTRL+C

Check rollout:

kubectl rollout status deployment/nginx-deployment

Check history:

kubectl rollout history deployment/nginx-deployment

What happened?

Kubernetes performed a rollout.

A Deployment allows us to update an application while Kubernetes manages replacement of Pods according to its rollout strategy.

---

PART 12 — Rollback

Suppose our new application version has a problem.

Rollback:

kubectl rollout undo deployment/nginx-deployment

Check:

kubectl rollout status deployment/nginx-deployment

Then:

kubectl get pods

Interview Question

How do you rollback a Deployment?

Answer:

kubectl rollout undo deployment/<deployment-name>

---

PART 13 — Deployment vs StatefulSet

Now we move to StatefulSet.

Deployment Pods might look like:

nginx-deployment-76f8b987d8-x8j2p
nginx-deployment-76f8b987d8-82ksa
nginx-deployment-76f8b987d8-p91ka

The exact generated suffixes can change.

StatefulSet Pods look like:

web-0
web-1
web-2

StatefulSet gives Pods stable ordinal identities.

Concept:

DEPLOYMENT

Pod A
Pod B
Pod C

Pods are generally interchangeable.

Compared with:

STATEFULSET

web-0
web-1
web-2

Each Pod has a stable identity.

---

PART 14 — Create a StorageClass

First check existing StorageClasses:

kubectl get storageclass

If your cluster already has a default StorageClass, record its name:

kubectl get storageclass

Example on an AWS EKS cluster might be:

gp2

or:

gp3

depending on the cluster configuration.

Do not blindly create a new StorageClass if your cluster already has the correct one.

Set the StorageClass name you actually have:

export STORAGE_CLASS=gp2

Replace gp2 if your cluster uses a different StorageClass.

Verify:

kubectl get storageclass "$STORAGE_CLASS"

---

PART 15 — Create a StatefulSet

Create:

nano statefulset.yaml

Add the following.

Replace gp2 with your cluster's StorageClass if necessary.

apiVersion: v1
kind: Service
metadata:
  name: nginx-headless
spec:
  clusterIP: None
  selector:
    app: nginx-stateful
  ports:
    - port: 80
      targetPort: 80
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nginx-stateful
spec:
  serviceName: nginx-headless
  replicas: 3

  selector:
    matchLabels:
      app: nginx-stateful

  template:
    metadata:
      labels:
        app: nginx-stateful

    spec:
      containers:
        - name: nginx
          image: nginx:1.27

          ports:
            - containerPort: 80

          volumeMounts:
            - name: nginx-data
              mountPath: /usr/share/nginx/html

  volumeClaimTemplates:
    - metadata:
        name: nginx-data

      spec:
        accessModes:
          - ReadWriteOnce

        storageClassName: gp2

        resources:
          requests:
            storage: 1Gi

Apply:

kubectl apply -f statefulset.yaml

Watch:

kubectl get pods -w

You should eventually see:

nginx-stateful-0
nginx-stateful-1
nginx-stateful-2

---

PART 16 — Observe StatefulSet Identity

Run:

kubectl get statefulset

Then:

kubectl get pods

Notice:

nginx-stateful-0
nginx-stateful-1
nginx-stateful-2

Compare:

DEPLOYMENT

nginx-deployment-xxxxx-abc12
nginx-deployment-xxxxx-def34

with:

STATEFULSET

nginx-stateful-0
nginx-stateful-1
nginx-stateful-2

This stable identity is one of the most important StatefulSet concepts.

---

PART 17 — PVC and PV

Run:

kubectl get pvc

You may see:

nginx-data-nginx-stateful-0
nginx-data-nginx-stateful-1
nginx-data-nginx-stateful-2

Check:

kubectl get pv

Now:

kubectl get storageclass

Understand the relationship:

StatefulSet
     ↓
nginx-stateful-0
     ↓
PVC
     ↓
PV
     ↓
Storage

PVC

PersistentVolumeClaim.

The application requests storage.

PV

PersistentVolume.

Represents persistent storage available/bound for Kubernetes use.

StorageClass

Describes a class of storage and, with dynamic provisioning, tells Kubernetes which provisioner and parameters to use to create storage.

---

PART 18 — Write Data

Let's demonstrate why persistent storage matters.

Run:

kubectl exec nginx-stateful-0 -- \
  sh -c 'echo "Hello from JumpToTech StatefulSet" > /usr/share/nginx/html/index.html'

Check:

kubectl exec nginx-stateful-0 -- \
  cat /usr/share/nginx/html/index.html

Expected:

Hello from JumpToTech StatefulSet

---

PART 19 — Delete the StatefulSet Pod

Delete:

kubectl delete pod nginx-stateful-0

Watch:

kubectl get pods -w

You should see:

nginx-stateful-0

created again.

Notice the name.

It is still:

nginx-stateful-0

not a random new identity.

Once it is Running, check the file:

kubectl exec nginx-stateful-0 -- \
  cat /usr/share/nginx/html/index.html

If storage was correctly provisioned and reattached, you should still see:

Hello from JumpToTech StatefulSet

This demonstrates two different concepts

Stable identity
+
Persistent storage

Do not confuse them.

StatefulSet gives the Pod stable identity.

PVC/PV provide persistent storage.

---

PART 20 — Compare Deployment and StatefulSet

Feature	Deployment	StatefulSet
Common use	Stateless apps	Stateful apps
Pod identity	Interchangeable/generated	Stable ordinal identity
Example	Web/API	Databases/clustered stateful systems
Typical names	app-xxxxx-xxxxx	app-0, app-1
Scaling	Designed for interchangeable replicas	Can use ordered behavior
Storage	Can use persistent storage	Commonly paired with per-Pod PVCs
Rollout	Deployment strategy	StatefulSet strategy

Important:

A StatefulSet does not automatically make a database production-ready.

The database itself must still be configured correctly for replication, backups, failover, security, and recovery.

---

PART 21 — BREAK THE APPLICATION

Now we intentionally create problems.

This is the most important section.

A DevOps engineer does not only deploy applications.

A DevOps engineer must be able to troubleshoot them.

---

TROUBLESHOOTING LAB 1 — Wrong Image

Run:

kubectl set image deployment/nginx-deployment \
  nginx=nginx:this-image-does-not-exist

Wait:

kubectl get pods

You may see:

ErrImagePull

followed by:

ImagePullBackOff

What do you do?

First:

kubectl describe pod <problem-pod-name>

Go to:

Events

You should find information about the image pull failure.

Question

Is Kubernetes broken?

Answer

No.

The requested container image/tag cannot be successfully pulled.

Fix:

kubectl set image deployment/nginx-deployment \
  nginx=nginx:1.27

Check:

kubectl rollout status deployment/nginx-deployment

---

TROUBLESHOOTING LAB 2 — CrashLoopBackOff

Create:

nano broken-pod.yaml

Add:

apiVersion: v1
kind: Pod

metadata:
  name: crash-demo

spec:
  containers:
    - name: crash
      image: busybox:1.36
      command:
        - sh
        - -c
        - echo "Application starting"; sleep 3; exit 1

Apply:

kubectl apply -f broken-pod.yaml

Watch:

kubectl get pods -w

Eventually the container will repeatedly restart and may show:

CrashLoopBackOff

Stop watching:

CTRL+C

Now troubleshoot:

kubectl describe pod crash-demo

Then:

kubectl logs crash-demo

And:

kubectl logs crash-demo --previous

Important troubleshooting pattern

Something is wrong
        ↓
kubectl get pods
        ↓
kubectl describe pod
        ↓
Check Events
        ↓
kubectl logs
        ↓
kubectl logs --previous

---

TROUBLESHOOTING LAB 3 — Pending PVC

Do this as a demonstration rather than leaving broken resources in the cluster.

Create:

nano bad-pvc.yaml

Add:

apiVersion: v1
kind: PersistentVolumeClaim

metadata:
  name: broken-pvc

spec:
  accessModes:
    - ReadWriteOnce

  storageClassName: storage-class-does-not-exist

  resources:
    requests:
      storage: 1Gi

Apply:

kubectl apply -f bad-pvc.yaml

Check:

kubectl get pvc

You should see it remain:

Pending

Investigate:

kubectl describe pvc broken-pvc

Then:

kubectl get storageclass

Lesson

If you see:

Pod Pending

and that Pod needs a PVC, check:

kubectl get pvc
kubectl describe pvc <pvc-name>
kubectl get storageclass

Delete the intentionally broken claim:

kubectl delete pvc broken-pvc

---

PART 22 — The Kubernetes Troubleshooting Mindset

Do not randomly delete things.

Follow a process.

Step 1

Check:

kubectl get pods

Step 2

Get more information:

kubectl get pods -o wide

Step 3

Describe the problematic resource:

kubectl describe pod <pod-name>

Step 4

Read:

Events

Step 5

Check application logs:

kubectl logs <pod-name>

For a previously crashed container:

kubectl logs <pod-name> --previous

Step 6

If storage is involved:

kubectl get pvc
kubectl get pv
kubectl get storageclass
kubectl describe pvc <pvc-name>

Step 7

If Deployment is involved:

kubectl get deployment
kubectl get rs
kubectl rollout status deployment/<name>
kubectl rollout history deployment/<name>

---

PART 23 — Final Architecture

You should now understand this:

USER
 │
 │ kubectl apply
 ↓
API SERVER
 │
 ↓
Desired state stored/processed
 │
 ↓
CONTROLLERS
 │
 ↓
Deployment
 │
 ↓
ReplicaSet
 │
 ↓
Pods
 │
 ↓
SCHEDULER
 │
 │ chooses Node
 ↓
WORKER NODE
 │
 ↓
kubelet
 │
 ↓
Container Runtime
 │
 ↓
Container Running

For a StatefulSet:

StatefulSet
     │
     ├── nginx-stateful-0
     │         │
     │         └── PVC → PV → Storage
     │
     ├── nginx-stateful-1
     │         │
     │         └── PVC → PV → Storage
     │
     └── nginx-stateful-2
               │
               └── PVC → PV → Storage

---

PART 24 — Student Challenge

Do this section without instructor help.

Challenge 1

Create a Deployment named:

student-web

Requirements:

Image: nginx:1.27
Replicas: 4
Label: app=student-web
Container port: 80

Verify:

kubectl get deployment
kubectl get rs
kubectl get pods

---

Challenge 2

Delete one Pod.

Prove Kubernetes recreates it.

---

Challenge 3

Scale:

4 → 7

Then:

7 → 2

---

Challenge 4

Update Nginx to:

nginx:1.28

Watch the rollout.

---

Challenge 5

Break the image intentionally.

Use:

nginx:does-not-exist

Troubleshoot it.

Then repair it.

---

Challenge 6

Explain:

Deployment
    ↓
ReplicaSet
    ↓
Pod
    ↓
Container

---

Challenge 7

Explain the difference between:

Deployment

and:

StatefulSet

without looking at your notes.

---

FINAL INTERVIEW QUESTIONS

1. What is a Pod?

A Pod is Kubernetes' smallest deployable unit and contains one or more containers.

2. What is a Deployment?

A Deployment manages stateless application workloads and declaratively manages ReplicaSets and application updates.

3. What is a ReplicaSet?

A ReplicaSet maintains the desired number of matching Pod replicas.

4. What happens when a Deployment Pod dies?

Its ReplicaSet works to create a replacement so the desired replica count is restored.

5. What is StatefulSet?

A workload controller designed for applications that need stable Pod identities and often stable per-Pod storage.

6. Deployment vs StatefulSet?

Deployment replicas are generally interchangeable.

StatefulSet Pods have stable ordinal identities such as:

mysql-0
mysql-1
mysql-2

7. What does Scheduler do?

It chooses an appropriate Node for an unscheduled Pod.

8. What does kubelet do?

It runs on each Worker Node and works with the container runtime to ensure assigned Pods are running.

9. What does API Server do?

It exposes the Kubernetes API and is the central entry point for API operations.

10. What is etcd?

It is the distributed key-value store used by Kubernetes to store cluster state.

11. What is PV?

PersistentVolume — a Kubernetes resource representing persistent storage.

12. What is PVC?

PersistentVolumeClaim — a request for persistent storage.

13. What does PVC Bound mean?

The claim has been successfully bound to a PersistentVolume.

14. What does Pending mean?

The resource cannot yet proceed to its required running/bound state. You need to investigate why.

15. What is ImagePullBackOff?

Kubernetes cannot successfully pull the requested container image and is backing off between retry attempts.

16. What is CrashLoopBackOff?

A container repeatedly starts, fails, and Kubernetes delays subsequent restart attempts.

17. First command when a Pod has a problem?

Start with:

kubectl get pods

Then investigate:

kubectl describe pod <pod-name>

and:

kubectl logs <pod-name>

18. Where do you look in kubectl describe?

Especially:

Events

19. How do you see which Node runs a Pod?

kubectl get pods -o wide

20. Explain what happens after kubectl apply -f deployment.yaml.

A strong answer:

kubectl sends an API request to the Kubernetes API Server.

The requested desired state is persisted in the cluster.

The Deployment controller observes the Deployment and manages the appropriate ReplicaSet.

The ReplicaSet works to create the required Pods.

The Scheduler assigns unscheduled Pods to suitable Nodes.

The kubelet on each selected Worker Node observes the assigned Pod and works with the container runtime to start the required containers.

Kubernetes controllers continuously reconcile actual state with desired state.

---

CLEANUP

Delete the lab resources:

kubectl delete -f deployment.yaml
kubectl delete -f statefulset.yaml
kubectl delete -f broken-pod.yaml
kubectl delete pod nginx-pod --ignore-not-found
kubectl delete pvc broken-pvc --ignore-not-found
kubectl delete deployment student-web --ignore-not-found

Check:

kubectl get pods
kubectl get deployment
kubectl get statefulset
kubectl get pvc

---

What You Must Remember

CONTROL PLANE
     ↓
Manages the cluster

WORKER NODE
     ↓
Runs workloads

POD
     ↓
Runs container(s)

DEPLOYMENT
     ↓
Manages ReplicaSets and stateless application rollout

REPLICASET
     ↓
Maintains Pod replicas

STATEFULSET
     ↓
Stable Pod identity

PVC
     ↓
Requests storage

PV
     ↓
Represents persistent storage

And the most important Kubernetes idea:

        DESIRED STATE
              ↓
       Kubernetes checks
              ↓
         ACTUAL STATE
              ↓
      Something is wrong?
              ↓
        RECONCILIATION
              ↓
   Move actual state toward
       the desired state