JumpToTech — Weekend Kubernetes Production Lab

Restaurant Company: Deploy, Configure, Scale & Troubleshoot

Goal

You are the DevOps Engineer responsible for deploying the Restaurant Company application into Kubernetes.

By the end of this lab, you should be able to explain and demonstrate:

EKS Cluster
    ↓
Worker Node
    ↓
Deployment
    ↓
ReplicaSet
    ↓
Pods
    ↓
Service
    ↓
Application

ConfigMap + Secret
        ↓
       Pods

Resources
CPU + Memory
        ↓
       Pods

Metrics Server
        ↓
       HPA
        ↓
Deployment replicas

Do not simply copy commands. After every section, understand what Kubernetes created and why.

---

Part 1 — Verify Your Kubernetes Environment

Before creating anything:

kubectl get nodes

Your Worker Node should show:

STATUS
Ready

Run:

kubectl get nodes -o wide

Then:

kubectl describe node <NODE-NAME>

Find:

Capacity:
  cpu:
  memory:
  pods:

Allocatable:
  cpu:
  memory:
  pods:

Questions

Students must answer:

1. What is a Worker Node?
2. What is Capacity?
3. What is Allocatable?
4. Why can Allocatable be smaller than Capacity?
5. How many Pods can your Node run?
6. Is Pod capacity the same thing as CPU capacity?
7. What does 1930m CPU mean if you see it?

Remember:

1000m = 1 CPU core
500m  = 0.5 CPU
100m  = 0.1 CPU

---

Part 2 — Create Your Working Directory

mkdir -p ~/weekend-k8s-lab
cd ~/weekend-k8s-lab

Verify:

pwd

You will create:

weekend-k8s-lab/
├── configmap.yaml
├── secret.yaml
├── deployment.yaml
├── service.yaml
└── hpa.yaml

---

Part 3 — ConfigMap

Create:

nano configmap.yaml

Add:

apiVersion: v1
kind: ConfigMap
metadata:
  name: restaurant-config
data:
  APP_ENV: "production"
  APP_NAME: "restaurant-company"

Apply:

kubectl apply -f configmap.yaml

Verify:

kubectl get configmap

kubectl describe configmap restaurant-config

Understand

A ConfigMap stores non-sensitive configuration separately from the application image.

Examples:

APP_ENV
APP_NAME
LOG_LEVEL
API_URL

Interview Question

Why shouldn't configuration always be hard-coded into the Docker image?

---

Part 4 — Secret

Create:

nano secret.yaml

For this classroom lab only:

apiVersion: v1
kind: Secret
metadata:
  name: restaurant-secret
type: Opaque
stringData:
  DB_PASSWORD: "JumpToTech123"

Apply:

kubectl apply -f secret.yaml

Check:

kubectl get secret

kubectl describe secret restaurant-secret

You may see:

Data
====
DB_PASSWORD:  13 bytes

Important

Do not commit real production passwords to GitHub.

Also remember:

Kubernetes Secret is not automatically "safe because it is a Secret." Base64 encoding is not encryption.

---

Part 5 — Deployment

Create:

nano deployment.yaml

Use:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment

spec:
  replicas: 2

  selector:
    matchLabels:
      app: nginx

  template:
    metadata:
      labels:
        app: nginx

    spec:
      containers:
        - name: nginx
          image: nginx:1.27

          ports:
            - containerPort: 80

          env:
            - name: APP_ENV
              valueFrom:
                configMapKeyRef:
                  name: restaurant-config
                  key: APP_ENV

            - name: APP_NAME
              valueFrom:
                configMapKeyRef:
                  name: restaurant-config
                  key: APP_NAME

            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: restaurant-secret
                  key: DB_PASSWORD

          resources:
            requests:
              cpu: "100m"
              memory: "128Mi"

            limits:
              cpu: "500m"
              memory: "256Mi"

          readinessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 5
            periodSeconds: 10

          livenessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 10
            periodSeconds: 20

Apply:

kubectl apply -f deployment.yaml

Check:

kubectl get deployment

kubectl get pods

kubectl get rs

Students should understand:

Deployment
     ↓
ReplicaSet
     ↓
Pods

---

Part 6 — Prove Self-Healing

Get Pods:

kubectl get pods

Delete ONE Deployment-managed Pod:

kubectl delete pod <POD-NAME>

Immediately:

kubectl get pods -w

Observe

You deleted one Pod.

But the desired state says:

replicas: 2

Therefore:

Desired = 2
Actual  = 1

ReplicaSet
    ↓
creates replacement
    ↓
Actual = 2

Interview Question

Why did Kubernetes recreate the Pod?

Do not answer:

"Because Kubernetes always recreates Pods."

Correct idea:

The Pod is managed by a ReplicaSet created by the Deployment, and the controller reconciles actual state with desired state.

---

Part 7 — Verify ConfigMap and Secret Injection

Find a Pod:

kubectl get pods

Check ConfigMap values:

kubectl exec -it <POD-NAME> -- printenv APP_ENV

Expected:

production

Then:

kubectl exec -it <POD-NAME> -- printenv APP_NAME

Expected:

restaurant-company

For this classroom lab only, verify that the Secret variable exists:

kubectl exec -it <POD-NAME> -- printenv DB_PASSWORD

Do not expose production secrets this way.

---

Part 8 — Understand Resources

Your Deployment contains:

resources:
  requests:
    cpu: "100m"
    memory: "128Mi"

  limits:
    cpu: "500m"
    memory: "256Mi"

Requests

Requests are especially important to scheduling.

New Pod
   ↓
CPU request = 100m
Memory request = 128Mi
   ↓
Scheduler
   ↓
Find suitable Node

Limits

Limits constrain resource usage.

CPU limit exceeded
→ CPU may be throttled

Memory limit exceeded
→ Container may be terminated / OOMKilled

Verify:

kubectl describe pod <POD-NAME>

Find:

Requests:
  cpu:     100m
  memory:  128Mi

Limits:
  cpu:     500m
  memory:  256Mi

Calculation

If there are 6 Pods:

6 × 100m CPU = 600m requested CPU

6 × 128Mi = 768Mi requested Memory

---

Part 9 — Readiness Probe

Your Deployment contains:

readinessProbe:
  httpGet:
    path: /
    port: 80

Readiness asks:

Should Kubernetes send normal Service traffic to this Pod?

Now intentionally break it.

Change:

path: /

to:

path: /wrong-page

Apply:

kubectl apply -f deployment.yaml

Watch:

kubectl get pods -w

You may see:

READY
0/1

But:

STATUS
Running

Important Interview Question

Can a Pod be Running but not Ready?

YES.

Investigate:

kubectl describe pod <POD-NAME>

Look at:

Events:

Now fix:

path: /

Apply again:

kubectl apply -f deployment.yaml

Watch:

kubectl get pods -w

Expected eventually:

1/1 Running

---

Part 10 — Liveness Probe

Your Deployment contains:

livenessProbe:
  httpGet:
    path: /
    port: 80

Remember:

Readiness
→ Should this workload receive traffic?

Liveness
→ Is this workload healthy enough to keep running?

Repeated liveness failures can cause kubelet to restart the Container.

Interview Question

Explain the difference between:

Readiness Probe
vs
Liveness Probe

in your own words.

---

Part 11 — Create a Service

Create:

nano service.yaml

Add:

apiVersion: v1
kind: Service
metadata:
  name: nginx-service

spec:
  selector:
    app: nginx

  ports:
    - port: 80
      targetPort: 80

  type: ClusterIP

Apply:

kubectl apply -f service.yaml

Check:

kubectl get svc

Then:

kubectl get endpoints nginx-service

You should see Pod IP addresses.

Architecture:

Client
   ↓
nginx-service
   ↓
selector:
app: nginx
   ↓
matching Pods

Important

The Service does not randomly discover your application.

It uses:

selector:
  app: nginx

and finds Pods with:

labels:
  app: nginx

---

Part 12 — Break the Service

Now create a real incident.

Change:

selector:
  app: nginx

to:

selector:
  app: wrong-label

Apply:

kubectl apply -f service.yaml

Check:

kubectl get endpoints nginx-service

You may see no endpoints.

Now troubleshoot:

kubectl get pods --show-labels

Compare:

Service selector
vs
Pod labels

Fix:

selector:
  app: nginx

Apply:

kubectl apply -f service.yaml

Check again:

kubectl get endpoints nginx-service

Interview Scenario

Pods are Running, but the Service has no endpoints. What do you check?

Answer:

Pod labels and Service selector.

---

Part 13 — Metrics Server

Now check resource usage:

kubectl top nodes

Then:

kubectl top pods

Understand the difference:

REQUEST
100m CPU

≠

ACTUAL USAGE
for example 5m CPU

≠

LIMIT
500m CPU

Three different concepts.

Architecture:

Pod / Container
      ↓
    kubelet
      ↓
Metrics Server
      ↓
Kubernetes Metrics API
      ↓
HPA

---

Part 14 — Create HPA

Create:

nano hpa.yaml

Add:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: nginx-hpa

spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: nginx-deployment

  minReplicas: 2
  maxReplicas: 8

  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50

Apply:

kubectl apply -f hpa.yaml

Check:

kubectl get hpa

You may see:

TARGETS
cpu: 0%/50%

Interpret:

0% = current CPU utilization
50% = target

Also:

kubectl describe hpa nginx-hpa

---

Part 15 — Understand HPA Before Testing It

HPA does not simply mean:

CPU reaches exactly 50%
→ create exactly 1 Pod

Instead, HPA evaluates the metric and calculates a desired replica count.

Also:

HPA
 ↓
changes desired replicas
 ↓
Deployment / ReplicaSet
 ↓
Pods are created/removed

For this CPU-utilization HPA, your CPU request matters:

requests:
  cpu: "100m"

Very simplified example:

CPU request = 100m

actual usage 10m
≈ 10%

actual usage 50m
≈ 50%

actual usage 80m
≈ 80%

---

Part 16 — Generate Traffic

Make sure:

kubectl get svc nginx-service

Then create a load generator:

kubectl run load-generator \
  --image=busybox:1.36 \
  --restart=Never \
  -- /bin/sh -c \
  'for i in 1 2 3 4 5 6 7 8 9 10; do while true; do wget -q -O- http://nginx-service > /dev/null; done & done; wait'

Now use multiple terminals.

Terminal 1

kubectl get hpa -w

Terminal 2

kubectl get pods -w

Terminal 3

kubectl top pods

Observe:

Load Generator
      ↓
HTTP requests
      ↓
Service
      ↓
Nginx Pods
      ↓
CPU utilization ↑
      ↓
Metrics Server
      ↓
HPA
      ↓
Desired replicas ↑
      ↓
Deployment / ReplicaSet
      ↓
More Pods

Do not expect exact CPU percentages or exact scaling timing to be identical on every student's cluster.

---

Part 17 — Stop Traffic

Delete:

kubectl delete pod load-generator

Watch:

kubectl get hpa -w

CPU usage should decrease.

Eventually HPA can reduce replicas toward:

minReplicas = 2

Scale-down may not happen immediately.

---

Part 18 — Production Incident: Pending Pods

This is based on a real incident from class.

Suppose:

Node Pod Capacity = 29
Current Pods      = 29

HPA wants additional replicas.

What happens?

HPA
 ↓
Desired replicas ↑
 ↓
ReplicaSet creates new Pod objects
 ↓
Scheduler tries to place them
 ↓
Node = 29/29 Pods
 ↓
No available Pod slot
 ↓
Pending

Investigate:

kubectl get pods

Then:

kubectl describe pod <PENDING-POD>

Look at:

Events:

You might find:

FailedScheduling
Too many pods

Important

This is different from:

Insufficient cpu

or:

Insufficient memory

Do not guess the root cause.

Read the Events.

---

Part 19 — Kubernetes Troubleshooting Challenge

For each scenario, write:

1. Symptom
2. Command used
3. Root cause
4. Fix
5. Verification

Incident A

Pod = Pending

Start:

kubectl describe pod <POD>

Check Events.

Incident B

Pod = Running
READY = 0/1

Think:

Readiness Probe

Incident C

Pods Running
Service exists
Endpoints = none

Think:

Service selector
vs
Pod labels

Incident D

Container keeps restarting

Start:

kubectl describe pod <POD>
kubectl logs <POD>

Consider liveness failures, application errors, OOMKilled, etc.

Incident E

HPA shows unknown CPU metric

Check:

kubectl top pods
kubectl describe hpa nginx-hpa

Also verify the workload has appropriate CPU requests for utilization-based CPU HPA.

---

Part 20 — Commands You Must Know

By Monday, students should be comfortable with:

kubectl get nodes
kubectl describe node <NODE>

kubectl get pods
kubectl get pods -o wide
kubectl get pods -w
kubectl describe pod <POD>
kubectl logs <POD>

kubectl get deployment
kubectl describe deployment nginx-deployment
kubectl get rs

kubectl get svc
kubectl describe svc nginx-service
kubectl get endpoints nginx-service

kubectl get configmap
kubectl get secret

kubectl apply -f deployment.yaml
kubectl apply -f service.yaml

kubectl top nodes
kubectl top pods

kubectl get hpa
kubectl get hpa -w
kubectl describe hpa nginx-hpa

---

Final Architecture — Draw This Yourself

Every student must be able to draw and explain this without notes:

                    USERS
                      ↓
              Load Balancer
                      ↓
                   Service
                      ↓
          ┌───────────┼───────────┐
          ↓           ↓           ↓
        Pod 1       Pod 2       Pod 3
          │           │           │
          └──── Deployment ───────┘
                    ↓
               ReplicaSet


ConfigMap ─────────→ Pods
Secret ────────────→ Pods


Worker Node
├── CPU
├── Memory
├── Storage
├── Networking
└── Pod Capacity


Pods
 ↓
CPU / Memory usage
 ↓
kubelet
 ↓
Metrics Server
 ↓
HPA
 ↓
Deployment desired replicas
 ↓
ReplicaSet
 ↓
More / Fewer Pods

Weekend Interview Questions

Students should prepare answers to these without reading from notes:

1. What is Kubernetes?
2. What is Amazon EKS?
3. What is the Control Plane?
4. What is a Worker Node?
5. What is a Pod?
6. What is a Container?
7. What does kubectl do?
8. What does the Scheduler do?
9. What is a Deployment?
10. What is a ReplicaSet?
11. What does replicas: 3 mean?
12. What is desired state?
13. What is reconciliation?
14. What happens when you delete a Deployment-managed Pod?
15. Why can Pod IP addresses change?
16. What problem does a Service solve?
17. What is ClusterIP?
18. What is LoadBalancer?
19. How does a Service find Pods?
20. What are labels?
21. What is a selector?
22. What are endpoints?
23. What happens when Service selector does not match Pod labels?
24. What is a ConfigMap?
25. What is a Secret?
26. ConfigMap vs Secret?
27. What are resource requests?
28. What are resource limits?
29. Who uses requests during scheduling?
30. What is 100m CPU?
31. What is 1000m CPU?
32. What can happen when CPU reaches its limit?
33. What can happen when Memory exceeds its limit?
34. What does OOMKilled mean?
35. What is a Readiness Probe?
36. What is a Liveness Probe?
37. Can a Pod be Running but 0/1 Ready?
38. What is Metrics Server?
39. What does kubectl top pods show?
40. What is HPA?
41. What does Horizontal scaling mean?
42. What is minReplicas?
43. What is maxReplicas?
44. What does averageUtilization: 50 mean?
45. Why are CPU requests important for CPU-utilization HPA?
46. Does HPA directly create Pods?
47. Does HPA create Worker Nodes?
48. What happens if HPA wants more Pods but the Node is full?
49. What does Pending mean?
50. How do you find why a Pod is Pending?
51. What does FailedScheduling mean?
52. What does Too many pods mean?
53. What's the difference between CPU capacity and Pod capacity?
54. What do you check when Pods are Running but Service has no endpoints?
55. What commands would you use if an application is down?

Required Submission

Each student should submit screenshots showing:

1. kubectl get nodes
2. kubectl get pods
3. kubectl get deployment
4. kubectl get rs
5. kubectl get svc
6. kubectl get endpoints nginx-service
7. kubectl get configmap
8. kubectl get secret
9. kubectl describe pod showing Requests/Limits
10. kubectl top nodes
11. kubectl top pods
12. kubectl get hpa
13. HPA before load
14. Load generator Running
15. HPA during CPU load
16. Additional Nginx Pods created by scaling
17. HPA after load is removed
18. One troubleshooting example and its Events