DEV Community: Aisalkyn Aidarova

WEEKEND HOMEWORK — DEVOPS FOUNDATION + KUBERNETES

Aisalkyn Aidarova — Sat, 10 Oct 2026 00:21:07 +0000

This homework is not about memorizing definitions. Your task is to prove that you understand where everything lives, what each component does, and how everything connects together.

Your assignment

Create a 10–15 minute screen recording with your voice.

You may draw the architecture on paper, whiteboard, PowerPoint, Excalidraw, or on your computer.

Do not read definitions from Google, ChatGPT, or your notes during the recording. Explain everything in your own words as if you are answering an interview question.

TASK 1 — Build a Computer

Draw a computer/server:

COMPUTER / SERVER
┌──────────────────────────────┐
│ CPU                          │
│ RAM                          │
│ Storage / SSD                │
│ Network                      │
│ Operating System             │
└──────────────────────────────┘

Explain:

CPU

What is CPU?
What does CPU do?
What is a CPU core?
What does 100m, 500m, 1000m mean in Kubernetes?
Does CPU store files?

RAM

What is RAM?
Why does an application need RAM?
RAM vs Storage?
What can happen when an application runs out of memory?
What does OOMKilled mean?

Storage

What is SSD/storage?
What normally remains after a reboot?
RAM vs persistent storage?

TASK 2 — Explain Operating System

Explain:

What is an Operating System?
Why does a computer need an OS?
What is Linux?
What does the OS manage?
What is the Kernel?
Application vs Process vs Service.
Why do servers commonly run Linux?

You must be able to explain this relationship:

Hardware / Virtual Hardware
        ↓
CPU + RAM + Storage + Network
        ↓
Operating System
        ↓
Applications / Processes

TASK 3 — Computer vs Server vs EC2

Explain:

What is a computer?
What is a server?
Is a server also a computer?
What makes a computer act as a server?
What is AWS EC2?
Is EC2 an Operating System?
Where does Linux run when we create an Ubuntu EC2 instance?
Where are CPU and RAM in this architecture?

Draw:

AWS
 ↓
EC2 Instance
 ↓
CPU + RAM + Storage + Network
 ↓
Linux OS
 ↓
Applications

TASK 4 — Networking

Explain:

What is an IP address?
What is a Private IP?
What is a Public IP?
Private IP vs Public IP.
What is a Port?
What are ports 22, 80, and 443 commonly used for?
What is an AWS Security Group?
Why can an application be running but still not open in your browser?

Explain this:

Your Laptop
     ↓
Internet
     ↓
Public endpoint / Public IP
     ↓
Security Group
     ↓
Port
     ↓
Server
     ↓
Application

TASK 5 — Image vs Container

This section is very important.

Explain:

What is a Docker Image?
What is a Container?
Image vs Container.
Is an Image a running application?
Can we create multiple Containers from the same Image?
What is a Container Runtime?
What is Docker/containerd doing?
Where does an Image come from?

Draw:

Container Registry
ECR / Docker Hub
        ↓
      Image
        ↓
Container Runtime
        ↓
     Container
        ↓
   Application

Use Nginx as your example:

nginx:1.27 Image
       ↓
Container Runtime
       ↓
Nginx Container
       ↓
Nginx Process

Do not say that the Image itself is running.

TASK 6 — Connect This to Kubernetes

Now extend your architecture.

Explain:

What is Kubernetes?
What is EKS?
What is a Cluster?
What is a Control Plane?
What is a Worker Node?
What is a Pod?
Container vs Pod.
Worker Node vs Pod.
Where does the Container run?
Where does the Pod run?
Where are CPU and RAM?

Draw:

EKS CLUSTER

Control Plane
      ↓
Scheduler
      ↓

WORKER NODE / EC2
┌─────────────────────────────┐
│ CPU                         │
│ RAM                         │
│ Storage                     │
│ Network                     │
│ Linux OS                    │
│                             │
│   Pod                       │
│   ┌─────────────────────┐   │
│   │ Container           │   │
│   │                     │   │
│   │ Nginx Application   │   │
│   └─────────────────────┘   │
└─────────────────────────────┘

You must clearly explain:

Worker Node is the server. Pod runs on the Worker Node. Container runs inside the Pod. Application runs inside the Container.

TASK 7 — Explain How Kubernetes Uses an Image

Explain what happens when Kubernetes sees:

containers:
  - name: nginx
    image: nginx:1.27

Explain the complete process:

Deployment / Pod specification
          ↓
image: nginx:1.27
          ↓
Scheduler chooses Worker Node
          ↓
Container Runtime on Node
          ↓
Obtains Image if necessary
          ↓
Creates Container
          ↓
Nginx starts running

Explain the difference between:

Image → Container → Pod → Worker Node

TASK 8 — Deployment and ReplicaSet

Explain:

What is a Deployment?
What is a ReplicaSet?
What are replicas?
What does replicas: 3 mean?
What is desired state?
What happens if you delete one Deployment-managed Pod?
Who recreates it?

Draw:

Deployment
    ↓
ReplicaSet
    ↓
┌─────┬─────┬─────┐
Pod   Pod   Pod

Then explain:

Desired = 3
Actual = 2
     ↓
Controller reconciliation
     ↓
New Pod
     ↓
Actual = 3

TASK 9 — Service

Explain:

Why shouldn't users connect directly to changing Pod IP addresses?
What is a Kubernetes Service?
What is ClusterIP?
What is LoadBalancer?
What are Labels?
What is a Selector?
What are Endpoints?
How does a Service know which Pods belong to it?

Draw:

Users
  ↓
AWS Load Balancer
  ↓
Kubernetes Service
  ↓
selector: app=nginx
  ↓
┌─────┬─────┬─────┐
Pod   Pod   Pod

Explain what happens if:

Service selector = app: nginx

Pod label = app: restaurant

TASK 10 — ConfigMap and Secret

Explain:

What is ConfigMap?
What should we store in ConfigMap?
What is Secret?
ConfigMap vs Secret.
Why shouldn't passwords be hard-coded into application code or normal configuration?
How can a Pod receive a value from ConfigMap or Secret?

Draw:

ConfigMap ──────┐
                ↓
               Pod
                ↑
Secret ─────────┘

TASK 11 — Resources

Explain:

resources:
  requests:
    cpu: "100m"
    memory: "128Mi"

  limits:
    cpu: "500m"
    memory: "256Mi"

Explain:

What is requests?
What is limits?
Why does Scheduler care about requests?
What does 100m CPU mean?
What does 500m CPU mean?
What can happen when CPU reaches its limit?
What can happen when Memory exceeds its limit?

Calculate:

6 Pods
Each requests 100m CPU

Total CPU request = ?

Then:

6 Pods
Each requests 128Mi Memory

Total Memory request = ?

TASK 12 — Readiness and Liveness

Explain the difference:

Readiness
"Should Kubernetes send traffic to this Pod?"

Liveness
"Is this workload healthy enough to keep running?"

Answer:

Can a Pod be Running but 0/1 Ready?
What happens when Readiness fails?
What can happen after repeated Liveness failures?
Why do we need both?

TASK 13 — HPA

Explain:

What is HPA?
What does Horizontal scaling mean?
What is Metrics Server?
Where does HPA get CPU metrics?
What is minReplicas?
What is maxReplicas?
What does averageUtilization: 50 mean?
Why is requests.cpu important for CPU-utilization HPA?
Does HPA directly create Pods?
Does HPA create Worker Nodes?

Draw:

Users / Traffic ↑
       ↓
Service
       ↓
Pods
       ↓
CPU utilization ↑
       ↓
kubelet
       ↓
Metrics Server
       ↓
HPA
       ↓
Desired replicas ↑
       ↓
Deployment / ReplicaSet
       ↓
More Pods

TASK 14 — FULL ARCHITECTURE

This is the most important part of the homework.

Without notes, draw one complete architecture:

USER
 ↓
Browser
 ↓
DNS
 ↓
Public Endpoint / Load Balancer
 ↓
AWS Networking / Security
 ↓
Kubernetes Service
 ↓
Pod
 ↓
Container
 ↓
Nginx Application
 ↓
CPU + RAM


                 EKS CLUSTER

              Control Plane
                    ↓
                 Scheduler
                    ↓

              WORKER NODE
        ┌──────────────────────┐
        │ EC2 Server           │
        │                      │
        │ CPU                  │
        │ RAM                  │
        │ Storage              │
        │ Network              │
        │ Linux OS             │
        │                      │
        │ Pod                  │
        │  └─ Container        │
        │      └─ Nginx        │
        └──────────────────────┘

Then explain the entire journey of one user request.

Example scenario:

A customer opens the Restaurant Company website.

Start with the customer and continue until Nginx returns the response.

Do not skip CPU, RAM, OS, Server, Network, Service, Pod or Container.

TASK 15 — TROUBLESHOOTING INTERVIEW

Imagine the interviewer tells you:

“Our website is down. How would you troubleshoot it?”

Do NOT answer:

“I will restart the Pod.”

You don't know the root cause yet.

Explain your investigation:

Is Worker Node healthy?
        ↓
Are Pods Running?
        ↓
Are Pods Ready?
        ↓
What do Pod Events say?
        ↓
What do application logs say?
        ↓
Is Service correct?
        ↓
Does Service have Endpoints?
        ↓
Do Labels match Selectors?
        ↓
Are Probes passing?
        ↓
Are ConfigMap/Secret correct?
        ↓
Are Resources sufficient?
        ↓
Is Load Balancer working?
        ↓
Networking / Security Group / Ports?
        ↓
DNS?

Mention commands you would use:

kubectl get nodes
kubectl get pods
kubectl describe pod <POD>
kubectl logs <POD>
kubectl get svc
kubectl get endpoints
kubectl get pods --show-labels
kubectl top nodes
kubectl top pods
kubectl get hpa
kubectl describe hpa <HPA>

FINAL SUBMISSION

Submit one 10–15 minute video recording.

Your video must include:

Draw and explain CPU, RAM, Storage, OS and Server.
Explain Private IP vs Public IP.
Explain Image vs Container.
Explain Container vs Pod.
Explain Worker Node vs Pod.
Explain Deployment → ReplicaSet → Pods.
Explain Service → Labels → Selectors → Endpoints.
Explain ConfigMap and Secret.
Explain Requests and Limits.
Explain Readiness and Liveness.
Explain Metrics Server and HPA.
Draw the complete architecture from User → Application.
Explain one troubleshooting scenario.

Important rules

Do not read from notes. Do not just show commands. Do not memorize sentences without understanding them.

For every component, answer:

What is it?
Where does it live?
What does it do?
Why do we need it?
What happens if it fails?

The goal is that on Monday, if an interviewer asks:

“Explain how your application runs in Kubernetes.”

you can draw the architecture and explain the complete system confidently from CPU and Operating System all the way to Kubernetes and HPA.

FINAL KUBERNETES PRODUCTION CAPSTONE Dev Staging Production | Helm | Argo CD | Security | Storage | 10 Production Incidents

Aisalkyn Aidarova — Sat, 10 Oct 2026 00:15:08 +0000

“I built and operated a production-style Kubernetes platform with separate dev, staging, and production environments. I deployed applications with Helm, implemented GitOps with Argo CD, configured resources, probes, autoscaling, availability, RBAC, networking, persistent storage, and practiced production incident response.”

Your role

You have joined a financial company as a DevOps Engineer.

The company is building an online banking platform:

Customers
    ↓
Banking Application
    │
    ├── frontend
    ├── account-service
    ├── payment-service
    └── transaction-service
             ↓
          Database

The company wants Kubernetes environments built using production practices.

You are responsible for designing, deploying, securing, operating, troubleshooting, and explaining the platform.

This is not a copy/paste homework.

Every student must record a video showing the environment and explaining every major decision.

PART 1 — Architecture Design

Before creating anything, draw your architecture.

Your recording must explain:

                         INTERNET
                            │
                            ▼
                    Load Balancer / ALB
                            │
                            ▼
                         Ingress
                            │
                 ┌──────────┼──────────┐
                 ▼          ▼          ▼
             frontend    account    payment
                            │          │
                            └────┬─────┘
                                 ▼
                             Database

Then explain the Kubernetes architecture underneath:

                    KUBERNETES / EKS
                           │
              ┌────────────┴────────────┐
              │                         │
        CONTROL PLANE              WORKER NODES
              │                         │
        API Server                     kubelet
        Scheduler                      runtime
        Controllers                    CNI
        etcd                           kube-proxy
                                        │
                                       Pods

In your recording explain:

What happens from kubectl apply until the container is actually running?

You should be able to explain approximately:

kubectl
   ↓
API Server
   ↓
desired state stored
   ↓
Controllers
   ↓
Scheduler
   ↓
Worker Node selected
   ↓
kubelet
   ↓
CRI
   ↓
container runtime
   ↓
Container starts
   ↓
CNI networking
   ↓
Readiness
   ↓
Service can route traffic

Do not just read component definitions.

Explain the process.

PART 2 — Build Three Environments

The company requires:

DEV
STAGING
PRODUCTION

For this homework, you can implement them as namespaces:

kubectl create namespace dev
kubectl create namespace staging
kubectl create namespace production

Verify:

kubectl get namespaces

But explain an important production point

In your recording answer:

“Would a bank necessarily run dev, staging and production merely as three namespaces in one cluster?”

Expected explanation:

Not necessarily.

For stronger isolation, especially in regulated/security-sensitive organizations, production may use separate clusters and often separate AWS accounts/VPC boundaries.

For example:

AWS Organization
│
├── Development Account
│      └── Dev EKS
│
├── Staging / Pre-Prod Account
│      └── Staging EKS
│
└── Production Account
       └── Production EKS

Namespaces are acceptable for our training environment because creating several full EKS environments costs additional money and operational effort.

But students must understand the difference between:

Lab architecture and real production architecture.

PART 3 — Build the Application

Create a Deployment for the application.

Requirements:

Application: banking-app
Production replicas: 3
Container: nginx or your own application
Container port: 80

Your Deployment must include:

resources:
  requests:
    cpu:
    memory:
  limits:
    cpu:
    memory:

and:

readinessProbe:

and:

livenessProbe:

and a production rolling strategy:

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 0
    maxSurge: 1

Students must decide appropriate values and explain them.

PART 4 — Explain Requests and Limits

In the recording answer:

Why do we use requests?

Why do we use limits?

What happens when memory exceeds its limit?

What can happen when CPU reaches/exceeds its limit?

Why should we not blindly give every Pod huge requests?

Explain:

Requests
   ↓
Scheduling / reserved expectation

Limits
   ↓
Maximum resource boundary

Also explain why bad resource configuration can create:

Pending Pods
CPU throttling
OOMKilled
Wasted cluster capacity
Higher AWS cost

PART 5 — Probes

Implement:

Startup Probe (optional if appropriate)
Readiness Probe
Liveness Probe

Explain:

Readiness:
Should this Pod receive traffic?

Liveness:
Should Kubernetes restart the container?

Startup:
Has this slow-starting application successfully started?

Demonstrate what happens when readiness fails.

PART 6 — Service

Create:

ClusterIP Service

Explain:

Why shouldn't another application communicate directly with a Pod IP?

Explain:

Service
   ↓
Selector
   ↓
Pod labels
   ↓
EndpointSlices
   ↓
Ready Pods

Show:

kubectl get svc -n production
kubectl get endpointslice -n production
kubectl get pods -n production --show-labels

PART 7 — Ingress

Create an Ingress for the application.

Conceptually:

Customer
   ↓
DNS
   ↓
Load Balancer
   ↓
Ingress Controller
   ↓
Ingress Rules
   ↓
Service
   ↓
Pods

In your recording explain:

Ingress vs Ingress Controller

Ingress vs Service

Where HTTPS/TLS could terminate

How traffic reaches the application

For AWS/EKS, explain how an AWS load-balancing solution can integrate with Kubernetes rather than claiming the Ingress resource itself magically creates everything.

PART 8 — ConfigMap and Secret

Create a ConfigMap containing something such as:

APP_ENV=production
LOG_LEVEL=info

Create a Secret containing a dummy training credential.

Application must consume configuration through environment variables or mounted files.

Explain:

ConfigMap vs Secret

And answer:

“Is base64 encryption?”

Correct answer:

No.

For the banking scenario, discuss why real secrets should be managed carefully and why cloud secret-management/encryption solutions may be integrated rather than committing credentials to Git.

Never commit real secrets to the repository.

PART 9 — RBAC / Least Privilege

Create:

ServiceAccount
Role
RoleBinding

Create a developer ServiceAccount.

It should be allowed to:

get Pods
list Pods
watch Pods

It should not be allowed to:

delete Pods
delete Deployments
delete Secrets

Test:

kubectl auth can-i list pods \
--as=system:serviceaccount:production:developer \
-n production

Then:

kubectl auth can-i delete pods \
--as=system:serviceaccount:production:developer \
-n production

Explain:

Authentication
=
Who are you?

Authorization
=
What are you allowed to do?

Then explain why least privilege is especially important for a banking application.

PART 10 — NetworkPolicy

Create a production NetworkPolicy.

Architecture:

frontend
   │
   ▼
payment-service
   │
   ▼
database

Allowed:

frontend → payment ✓
payment → database ✓

Not allowed:

frontend → database ✗
random Pod → database ✗

Students must explain why RBAC cannot solve this problem.

Expected:

RBAC
→ Kubernetes API authorization

NetworkPolicy
→ workload network communication

Also explain that NetworkPolicy requires a network implementation that enforces it.

PART 11 — HPA

Configure HPA.

Example requirements:

minimum replicas: 3
maximum replicas: 10
CPU target: 60%

Students must show:

kubectl get hpa -n production
kubectl describe hpa -n production

Explain:

What happens when traffic increases?

And:

What happens if HPA reaches maxReplicas but traffic continues increasing?

Also answer:

What does CPU-based HPA need?

Metrics and appropriate resource requests.

PART 12 — PDB

Create a PodDisruptionBudget.

For example:

3 replicas

minAvailable: 2

Explain why this matters during voluntary disruption such as node maintenance.

Also explain:

Does PDB protect against every failure?

No.

PART 13 — Scheduling

Demonstrate or explain:

nodeSelector
node affinity
pod anti-affinity
taints
tolerations

For production, explain why we might avoid placing every replica on the same node.

Bad:

Node 1
├── payment-1
├── payment-2
└── payment-3

Node 1 fails
→ payment completely unavailable

Better:

Node 1 → payment-1
Node 2 → payment-2
Node 3 → payment-3

Discuss topology/AZ resilience where appropriate.

PART 14 — StatefulSet + Persistent Storage

Create a StatefulSet.

It must demonstrate:

Stable Pod identity
PVC
StorageClass
PV
Persistent data

Show:

kubectl get statefulset
kubectl get pvc
kubectl get pv
kubectl get storageclass

Write data.

Delete the Pod.

Allow StatefulSet to recreate it.

Prove the data still exists.

Then explain:

Stable identity and persistent storage are related but different concepts.

PART 15 — AWS EBS CSI

Explain:

StatefulSet
    ↓
PVC
    ↓
StorageClass
    ↓
EBS CSI Driver
    ↓
AWS EBS
    ↓
PV
    ↓
PVC Bound
    ↓
Pod Running

Students must explain the storage incident we previously encountered:

PVC Pending
↓
StorageClass existed
↓
ebs.csi.aws.com expected
↓
EBS CSI add-on/components missing
↓
No dynamic EBS volume
↓
Pod Pending

And how to investigate it.

PART 16 — Helm

Convert the application into a Helm chart.

Suggested structure:

banking-app/
│
├── Chart.yaml
├── values.yaml
├── values-dev.yaml
├── values-staging.yaml
├── values-prod.yaml
│
└── templates/
    ├── deployment.yaml
    ├── service.yaml
    ├── ingress.yaml
    ├── configmap.yaml
    ├── serviceaccount.yaml
    ├── hpa.yaml
    ├── pdb.yaml
    └── networkpolicy.yaml

Different environment values should demonstrate meaningful differences.

For example:

DEV
replicas: 1
smaller resources

STAGING
replicas: 2
production-like configuration

PRODUCTION
replicas: 3+
HPA
PDB
stronger availability settings

Demonstrate:

helm lint
helm template
helm upgrade --install
helm list
helm history

Explain why Helm is useful in a company with many applications and environments.

PART 17 — Argo CD / GitOps

Put Kubernetes/Helm configuration in Git.

Architecture:

Developer
   ↓
Pull Request
   ↓
Review
   ↓
Merge
   ↓
Git
   ↓
Argo CD
   ↓
EKS

Students must create/configure an Argo CD Application and demonstrate:

Synced
Healthy

Then manually modify something in Kubernetes.

Observe:

OutOfSync

Explain drift.

Then restore/reconcile through GitOps.

Critical production question:

“If Argo CD manages production, should engineers routinely modify production using kubectl edit?”

Expected:

Generally no. Desired changes should normally go through the controlled Git/change process, with emergency procedures handled according to company policy.

PART 18 — Banking Production Requirements

This section is mandatory in the recording.

Imagine this is a banking system handling financial transactions and sensitive data.

Before saying “production ready,” discuss:

High availability
Least privilege
Encryption
Secret management
Network segmentation
Auditability
Backups
Disaster recovery
Monitoring
Alerting
Logging
Change control
Image scanning
Vulnerability management
Patch management
Capacity planning
Rollback strategy
Business continuity
Compliance requirements

Do not simply say:

“We have three replicas, therefore the bank is production-ready.”

Production readiness is much larger than Kubernetes manifests.

PART 19 — What Costs Money in AWS?

Students must be able to answer this because engineers should understand cost.

Discuss potential charges for things such as:

EKS cluster/control-plane service
EC2 worker nodes
EBS volumes/snapshots
Load balancers
NAT Gateway
Data transfer
CloudWatch logs/metrics
ECR storage/data transfer
Public IPv4 where applicable
Route 53
Backup/storage services
Other managed AWS services

Do not memorize exact prices because pricing changes.

Instead explain:

“Which architectural decisions create cost?”

For example:

More nodes → more compute cost

More/larger EBS → more storage cost

Multiple ALBs → load-balancer cost

Large log ingestion → observability cost

NAT traffic → networking cost

PART 20 — Production Monitoring

Your application must have an observability plan.

Explain what you would monitor:

CPU
Memory
Pod restarts
Ready replicas
Node health
Request rate
Error rate
Latency
HTTP 5xx
HPA behavior
PVC/storage
Database health
Dependency health

And explain:

Metrics
Logs
Traces

A strong student should mention service-level signals such as:

Traffic
Errors
Latency
Saturation

not only Kubernetes CPU.

PART 21 — TEN REQUIRED PRODUCTION INCIDENTS

This is the most important part of the homework.

Each student must break the environment intentionally, investigate it without jumping directly to the answer, fix/mitigate it, and verify recovery on video.

Incident 1 — Bad Deployment / ImagePullBackOff

Break:

Deploy an invalid/nonexistent image tag.

Student must discover:

New rollout
↓
New Pods unhealthy
↓
ImagePullBackOff
↓
describe Pod
↓
Events
↓
Image cannot be pulled

Mitigate:

kubectl rollout history deployment/<name> -n production
kubectl rollout undo deployment/<name> -n production
kubectl rollout status deployment/<name> -n production

Interview explanation:

“A new release introduced an invalid image reference. I correlated the incident with the rollout, confirmed the image pull failure from Pod events, rolled back to the known-good revision, and verified service recovery.”

Incident 2 — Production Slow / HPA at Maximum

Generate load.

Students investigate:

Application slow
↓
Pods Running
↓
CPU/resource pressure
↓
HPA target exceeded
↓
maxReplicas reached
↓
capacity insufficient for current load

They must not say “traffic” without evidence.

Incident 3 — Service Selector Mismatch

Break Service selector:

Service:
app=wrong-app

Pods:
app=banking-app

Symptoms:

Pods 1/1 Running
Service exists
Application unreachable
No expected Service backend

Student must compare:

kubectl describe svc
kubectl get pods --show-labels
kubectl get endpointslice

Incident 4 — Readiness Probe Failure

Break:

readinessProbe:
  httpGet:
    path: /does-not-exist

Expected:

Running 0/1

Student must explain:

Running ≠ Ready.

Use describe to find probe failure.

Fix/rollback and verify readiness and Service routing.

Incident 5 — OOMKilled

Configure a workload with insufficient memory or a controlled memory-hungry training application.

Student must find:

RESTARTS ↑
↓
CrashLoopBackOff
↓
describe
↓
Last State
↓
OOMKilled

Then:

kubectl logs <pod> --previous

Explain why simply increasing memory may only be mitigation if the application has unbounded memory growth.

Incident 6 — Dependency Failure

Application logs:

Cannot connect to database/Kafka

Student is not allowed to say:

“Kafka is down.”

until proven.

Investigate:

DNS
Service
EndpointSlice
Port
NetworkPolicy
Authentication
Dependency health
Application configuration

Interview statement:

“The log proves the application cannot connect to the dependency; it does not by itself prove the dependency is down.”

Incident 7 — PVC Pending

Break storage provisioning.

Expected:

Pod Pending
↓
PVC Pending
↓
Events
↓
StorageClass
↓
CSI
↓
Cloud permissions/storage

Students must walk the entire storage dependency chain.

Incident 8 — RBAC Forbidden

Make a ServiceAccount attempt:

kubectl delete pod ...

Expected:

Forbidden

Student investigates:

kubectl auth can-i ...

Then Role/RoleBinding.

They must explain:

Authentication succeeded
but
Authorization denied

A Forbidden response is different from a network failure.

Incident 9 — NetworkPolicy Blocks Traffic

Create:

frontend → payment ✗

because NetworkPolicy blocks the connection.

Students must prove:

Pods Running
Service exists
Endpoints exist
DNS may work
BUT network connection blocked

Then inspect:

kubectl get networkpolicy
kubectl describe networkpolicy

They must explain why restarting the application doesn't fix the policy.

Incident 10 — Argo CD OutOfSync / Configuration Drift

Git says:

replicas: 3

Manually change live cluster:

kubectl scale deployment banking-app \
--replicas=1 \
-n production

Observe Argo CD.

Students explain:

Git desired state
      ≠
Live state

→ OutOfSync

Restore desired state through the GitOps workflow/reconciliation.

Explain why uncontrolled manual production changes are dangerous.

PART 22 — For Every Incident, Use the Same Professional Method

Students may not begin with random commands.

Every recording must follow:

1. ALERT / TICKET

2. CUSTOMER IMPACT
   What is broken?

3. SCOPE
   Which application/service/environment?

4. WHAT CHANGED?
   Deployment?
   Config?
   Traffic?
   Infrastructure?

5. OBSERVE
   Metrics
   Logs
   Events
   Traces

6. HYPOTHESIS
   What might explain the evidence?

7. PROVE / DISPROVE
   Collect evidence.

8. ROOT CAUSE
   What actually caused the problem?

9. MITIGATION / FIX
   Restore service safely.

10. VERIFY
    Prove service is healthy.

11. RCA
    Why did this happen?
    How do we prevent recurrence?

The sentence I want every student to learn is:

“I don't want to guess the root cause. I form a hypothesis and validate it with evidence.”

PART 23 — Production Verification

After every fix, students must prove recovery.

They cannot say:

“The command succeeded, so production is fixed.”

They must check the relevant layers:

Deployment healthy
       ↓
Pods Ready
       ↓
Restart count stable
       ↓
Service backend healthy
       ↓
Application responds
       ↓
Errors normal
       ↓
Latency normal
       ↓
Critical customer flow works

PART 24 — Git Repository Requirements

Repository should look professional.

For example:

banking-platform/
│
├── README.md
│
├── docs/
│   ├── architecture.md
│   ├── troubleshooting.md
│   └── incidents.md
│
├── helm/
│   └── banking-app/
│       ├── Chart.yaml
│       ├── values.yaml
│       ├── values-dev.yaml
│       ├── values-staging.yaml
│       ├── values-prod.yaml
│       └── templates/
│
├── argocd/
│   ├── dev.yaml
│   ├── staging.yaml
│   └── production.yaml
│
└── kubernetes/
    ├── rbac/
    ├── network-policy/
    ├── storage/
    └── troubleshooting/

README should explain architecture, environments, deployment process, security controls, availability, scaling, storage, GitOps and troubleshooting.

PART 25 — Required Video Recording

I would require approximately 45–60 minutes, but quality matters more than exact duration.

Tell students:

Do not read definitions from Google or from your notes. Share your screen and explain your own environment as though I am the interviewer.

The recording must show and explain the full lifecycle:

Architecture
    ↓
AWS / EKS
    ↓
Dev / Staging / Production
    ↓
Docker Image / ECR
    ↓
Deployment / ReplicaSet / Pods
    ↓
Resources / Probes
    ↓
Service / EndpointSlice
    ↓
Ingress
    ↓
ConfigMap / Secret
    ↓
RBAC
    ↓
NetworkPolicy
    ↓
HPA / PDB
    ↓
Scheduling
    ↓
StatefulSet / PV / PVC / StorageClass / CSI
    ↓
Helm
    ↓
Argo CD / GitOps
    ↓
Observability
    ↓
10 Production Incidents
    ↓
RCA / Verification

At random points in the recording they should be able to answer:

Why did you configure it this way?

That is more important than merely showing YAML.

JumpToTech — Weekend Kubernetes Production Lab

Aisalkyn Aidarova — Sat, 10 Oct 2026 00:02:17 +0000

Restaurant Company: Deploy, Configure, Scale & Troubleshoot

Goal

You are the DevOps Engineer responsible for deploying the Restaurant Company application into Kubernetes.

By the end of this lab, you should be able to explain and demonstrate:

EKS Cluster
    ↓
Worker Node
    ↓
Deployment
    ↓
ReplicaSet
    ↓
Pods
    ↓
Service
    ↓
Application

ConfigMap + Secret
        ↓
       Pods

Resources
CPU + Memory
        ↓
       Pods

Metrics Server
        ↓
       HPA
        ↓
Deployment replicas

Do not simply copy commands. After every section, understand what Kubernetes created and why.

Part 1 — Verify Your Kubernetes Environment

Before creating anything:

kubectl get nodes

Your Worker Node should show:

STATUS
Ready

Run:

kubectl get nodes -o wide

Then:

kubectl describe node <NODE-NAME>

Find:

Capacity:
  cpu:
  memory:
  pods:

Allocatable:
  cpu:
  memory:
  pods:

Questions

Students must answer:

What is a Worker Node?
What is Capacity?
What is Allocatable?
Why can Allocatable be smaller than Capacity?
How many Pods can your Node run?
Is Pod capacity the same thing as CPU capacity?
What does 1930m CPU mean if you see it?

Remember:

1000m = 1 CPU core
500m  = 0.5 CPU
100m  = 0.1 CPU

Part 2 — Create Your Working Directory

mkdir -p ~/weekend-k8s-lab
cd ~/weekend-k8s-lab

Verify:

pwd

You will create:

weekend-k8s-lab/
├── configmap.yaml
├── secret.yaml
├── deployment.yaml
├── service.yaml
└── hpa.yaml

Part 3 — ConfigMap

Create:

nano configmap.yaml

Add:

apiVersion: v1
kind: ConfigMap
metadata:
  name: restaurant-config
data:
  APP_ENV: "production"
  APP_NAME: "restaurant-company"

Apply:

kubectl apply -f configmap.yaml

Verify:

kubectl get configmap

kubectl describe configmap restaurant-config

Understand

A ConfigMap stores non-sensitive configuration separately from the application image.

Examples:

APP_ENV
APP_NAME
LOG_LEVEL
API_URL

Interview Question

Why shouldn't configuration always be hard-coded into the Docker image?

Part 4 — Secret

Create:

nano secret.yaml

For this classroom lab only:

apiVersion: v1
kind: Secret
metadata:
  name: restaurant-secret
type: Opaque
stringData:
  DB_PASSWORD: "JumpToTech123"

Apply:

kubectl apply -f secret.yaml

Check:

kubectl get secret

kubectl describe secret restaurant-secret

You may see:

Data
====
DB_PASSWORD:  13 bytes

Important

Do not commit real production passwords to GitHub.

Also remember:

Kubernetes Secret is not automatically "safe because it is a Secret." Base64 encoding is not encryption.

Part 5 — Deployment

Create:

nano deployment.yaml

Use:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment

spec:
  replicas: 2

  selector:
    matchLabels:
      app: nginx

  template:
    metadata:
      labels:
        app: nginx

    spec:
      containers:
        - name: nginx
          image: nginx:1.27

          ports:
            - containerPort: 80

          env:
            - name: APP_ENV
              valueFrom:
                configMapKeyRef:
                  name: restaurant-config
                  key: APP_ENV

            - name: APP_NAME
              valueFrom:
                configMapKeyRef:
                  name: restaurant-config
                  key: APP_NAME

            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: restaurant-secret
                  key: DB_PASSWORD

          resources:
            requests:
              cpu: "100m"
              memory: "128Mi"

            limits:
              cpu: "500m"
              memory: "256Mi"

          readinessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 5
            periodSeconds: 10

          livenessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 10
            periodSeconds: 20

Apply:

kubectl apply -f deployment.yaml

Check:

kubectl get deployment

kubectl get pods

kubectl get rs

Students should understand:

Deployment
     ↓
ReplicaSet
     ↓
Pods

Part 6 — Prove Self-Healing

Get Pods:

kubectl get pods

Delete ONE Deployment-managed Pod:

kubectl delete pod <POD-NAME>

Immediately:

kubectl get pods -w

Observe

You deleted one Pod.

But the desired state says:

replicas: 2

Therefore:

Desired = 2
Actual  = 1

ReplicaSet
    ↓
creates replacement
    ↓
Actual = 2

Interview Question

Why did Kubernetes recreate the Pod?

Do not answer:

"Because Kubernetes always recreates Pods."

Correct idea:

The Pod is managed by a ReplicaSet created by the Deployment, and the controller reconciles actual state with desired state.

Part 7 — Verify ConfigMap and Secret Injection

Find a Pod:

kubectl get pods

Check ConfigMap values:

kubectl exec -it <POD-NAME> -- printenv APP_ENV

Expected:

production

Then:

kubectl exec -it <POD-NAME> -- printenv APP_NAME

Expected:

restaurant-company

For this classroom lab only, verify that the Secret variable exists:

kubectl exec -it <POD-NAME> -- printenv DB_PASSWORD

Do not expose production secrets this way.

Part 8 — Understand Resources

Your Deployment contains:

resources:
  requests:
    cpu: "100m"
    memory: "128Mi"

  limits:
    cpu: "500m"
    memory: "256Mi"

Requests

Requests are especially important to scheduling.

New Pod
   ↓
CPU request = 100m
Memory request = 128Mi
   ↓
Scheduler
   ↓
Find suitable Node

Limits

Limits constrain resource usage.

CPU limit exceeded
→ CPU may be throttled

Memory limit exceeded
→ Container may be terminated / OOMKilled

Verify:

kubectl describe pod <POD-NAME>

Find:

Requests:
  cpu:     100m
  memory:  128Mi

Limits:
  cpu:     500m
  memory:  256Mi

Calculation

If there are 6 Pods:

6 × 100m CPU = 600m requested CPU

6 × 128Mi = 768Mi requested Memory

Part 9 — Readiness Probe

Your Deployment contains:

readinessProbe:
  httpGet:
    path: /
    port: 80

Readiness asks:

Should Kubernetes send normal Service traffic to this Pod?

Now intentionally break it.

Change:

path: /

to:

path: /wrong-page

Apply:

kubectl apply -f deployment.yaml

Watch:

kubectl get pods -w

You may see:

READY
0/1

But:

STATUS
Running

Important Interview Question

Can a Pod be Running but not Ready?

YES.

Investigate:

kubectl describe pod <POD-NAME>

Look at:

Events:

Now fix:

path: /

Apply again:

kubectl apply -f deployment.yaml

Watch:

kubectl get pods -w

Expected eventually:

1/1 Running

Part 10 — Liveness Probe

Your Deployment contains:

livenessProbe:
  httpGet:
    path: /
    port: 80

Remember:

Readiness
→ Should this workload receive traffic?

Liveness
→ Is this workload healthy enough to keep running?

Repeated liveness failures can cause kubelet to restart the Container.

Interview Question

Explain the difference between:

Readiness Probe
vs
Liveness Probe

in your own words.

Part 11 — Create a Service

Create:

nano service.yaml

Add:

apiVersion: v1
kind: Service
metadata:
  name: nginx-service

spec:
  selector:
    app: nginx

  ports:
    - port: 80
      targetPort: 80

  type: ClusterIP

Apply:

kubectl apply -f service.yaml

Check:

kubectl get svc

Then:

kubectl get endpoints nginx-service

You should see Pod IP addresses.

Architecture:

Client
   ↓
nginx-service
   ↓
selector:
app: nginx
   ↓
matching Pods

Important

The Service does not randomly discover your application.

It uses:

selector:
  app: nginx

and finds Pods with:

labels:
  app: nginx

Part 12 — Break the Service

Now create a real incident.

Change:

selector:
  app: nginx

to:

selector:
  app: wrong-label

Apply:

kubectl apply -f service.yaml

Check:

kubectl get endpoints nginx-service

You may see no endpoints.

Now troubleshoot:

kubectl get pods --show-labels

Compare:

Service selector
vs
Pod labels

Fix:

selector:
  app: nginx

Apply:

kubectl apply -f service.yaml

Check again:

kubectl get endpoints nginx-service

Interview Scenario

Pods are Running, but the Service has no endpoints. What do you check?

Answer:

Pod labels and Service selector.

Part 13 — Metrics Server

Now check resource usage:

kubectl top nodes

Then:

kubectl top pods

Understand the difference:

REQUEST
100m CPU

≠

ACTUAL USAGE
for example 5m CPU

≠

LIMIT
500m CPU

Three different concepts.

Architecture:

Pod / Container
      ↓
    kubelet
      ↓
Metrics Server
      ↓
Kubernetes Metrics API
      ↓
HPA

Part 14 — Create HPA

Create:

nano hpa.yaml

Add:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: nginx-hpa

spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: nginx-deployment

  minReplicas: 2
  maxReplicas: 8

  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50

Apply:

kubectl apply -f hpa.yaml

Check:

kubectl get hpa

You may see:

TARGETS
cpu: 0%/50%

Interpret:

0% = current CPU utilization
50% = target

Also:

kubectl describe hpa nginx-hpa

Part 15 — Understand HPA Before Testing It

HPA does not simply mean:

CPU reaches exactly 50%
→ create exactly 1 Pod

Instead, HPA evaluates the metric and calculates a desired replica count.

Also:

HPA
 ↓
changes desired replicas
 ↓
Deployment / ReplicaSet
 ↓
Pods are created/removed

For this CPU-utilization HPA, your CPU request matters:

requests:
  cpu: "100m"

Very simplified example:

CPU request = 100m

actual usage 10m
≈ 10%

actual usage 50m
≈ 50%

actual usage 80m
≈ 80%

Part 16 — Generate Traffic

Make sure:

kubectl get svc nginx-service

Then create a load generator:

kubectl run load-generator \
  --image=busybox:1.36 \
  --restart=Never \
  -- /bin/sh -c \
  'for i in 1 2 3 4 5 6 7 8 9 10; do while true; do wget -q -O- http://nginx-service > /dev/null; done & done; wait'

Now use multiple terminals.

Terminal 1

kubectl get hpa -w

Terminal 2

kubectl get pods -w

Terminal 3

kubectl top pods

Observe:

Load Generator
      ↓
HTTP requests
      ↓
Service
      ↓
Nginx Pods
      ↓
CPU utilization ↑
      ↓
Metrics Server
      ↓
HPA
      ↓
Desired replicas ↑
      ↓
Deployment / ReplicaSet
      ↓
More Pods

Do not expect exact CPU percentages or exact scaling timing to be identical on every student's cluster.

Part 17 — Stop Traffic

Delete:

kubectl delete pod load-generator

Watch:

kubectl get hpa -w

CPU usage should decrease.

Eventually HPA can reduce replicas toward:

minReplicas = 2

Scale-down may not happen immediately.

Part 18 — Production Incident: Pending Pods

This is based on a real incident from class.

Suppose:

Node Pod Capacity = 29
Current Pods      = 29

HPA wants additional replicas.

What happens?

HPA
 ↓
Desired replicas ↑
 ↓
ReplicaSet creates new Pod objects
 ↓
Scheduler tries to place them
 ↓
Node = 29/29 Pods
 ↓
No available Pod slot
 ↓
Pending

Investigate:

kubectl get pods

Then:

kubectl describe pod <PENDING-POD>

Look at:

Events:

You might find:

FailedScheduling
Too many pods

Important

This is different from:

Insufficient cpu

or:

Insufficient memory

Do not guess the root cause.

Read the Events.

Part 19 — Kubernetes Troubleshooting Challenge

For each scenario, write:

1. Symptom
2. Command used
3. Root cause
4. Fix
5. Verification

Incident A

Pod = Pending

Start:

kubectl describe pod <POD>

Check Events.

Incident B

Pod = Running
READY = 0/1

Think:

Readiness Probe

Incident C

Pods Running
Service exists
Endpoints = none

Think:

Service selector
vs
Pod labels

Incident D

Container keeps restarting

Start:

kubectl describe pod <POD>
kubectl logs <POD>

Consider liveness failures, application errors, OOMKilled, etc.

Incident E

HPA shows unknown CPU metric

Check:

kubectl top pods
kubectl describe hpa nginx-hpa

Also verify the workload has appropriate CPU requests for utilization-based CPU HPA.

Part 20 — Commands You Must Know

By Monday, students should be comfortable with:

kubectl get nodes
kubectl describe node <NODE>

kubectl get pods
kubectl get pods -o wide
kubectl get pods -w
kubectl describe pod <POD>
kubectl logs <POD>

kubectl get deployment
kubectl describe deployment nginx-deployment
kubectl get rs

kubectl get svc
kubectl describe svc nginx-service
kubectl get endpoints nginx-service

kubectl get configmap
kubectl get secret

kubectl apply -f deployment.yaml
kubectl apply -f service.yaml

kubectl top nodes
kubectl top pods

kubectl get hpa
kubectl get hpa -w
kubectl describe hpa nginx-hpa

Final Architecture — Draw This Yourself

Every student must be able to draw and explain this without notes:

                    USERS
                      ↓
              Load Balancer
                      ↓
                   Service
                      ↓
          ┌───────────┼───────────┐
          ↓           ↓           ↓
        Pod 1       Pod 2       Pod 3
          │           │           │
          └──── Deployment ───────┘
                    ↓
               ReplicaSet


ConfigMap ─────────→ Pods
Secret ────────────→ Pods


Worker Node
├── CPU
├── Memory
├── Storage
├── Networking
└── Pod Capacity


Pods
 ↓
CPU / Memory usage
 ↓
kubelet
 ↓
Metrics Server
 ↓
HPA
 ↓
Deployment desired replicas
 ↓
ReplicaSet
 ↓
More / Fewer Pods

Weekend Interview Questions

Students should prepare answers to these without reading from notes:

What is Kubernetes?
What is Amazon EKS?
What is the Control Plane?
What is a Worker Node?
What is a Pod?
What is a Container?
What does kubectl do?
What does the Scheduler do?
What is a Deployment?
What is a ReplicaSet?
What does replicas: 3 mean?
What is desired state?
What is reconciliation?
What happens when you delete a Deployment-managed Pod?
Why can Pod IP addresses change?
What problem does a Service solve?
What is ClusterIP?
What is LoadBalancer?
How does a Service find Pods?
What are labels?
What is a selector?
What are endpoints?
What happens when Service selector does not match Pod labels?
What is a ConfigMap?
What is a Secret?
ConfigMap vs Secret?
What are resource requests?
What are resource limits?
Who uses requests during scheduling?
What is 100m CPU?
What is 1000m CPU?
What can happen when CPU reaches its limit?
What can happen when Memory exceeds its limit?
What does OOMKilled mean?
What is a Readiness Probe?
What is a Liveness Probe?
Can a Pod be Running but 0/1 Ready?
What is Metrics Server?
What does kubectl top pods show?
What is HPA?
What does Horizontal scaling mean?
What is minReplicas?
What is maxReplicas?
What does averageUtilization: 50 mean?
Why are CPU requests important for CPU-utilization HPA?
Does HPA directly create Pods?
Does HPA create Worker Nodes?
What happens if HPA wants more Pods but the Node is full?
What does Pending mean?
How do you find why a Pod is Pending?
What does FailedScheduling mean?
What does Too many pods mean?
What's the difference between CPU capacity and Pod capacity?
What do you check when Pods are Running but Service has no endpoints?
What commands would you use if an application is down?

Required Submission

Each student should submit screenshots showing:

1. kubectl get nodes
2. kubectl get pods
3. kubectl get deployment
4. kubectl get rs
5. kubectl get svc
6. kubectl get endpoints nginx-service
7. kubectl get configmap
8. kubectl get secret
9. kubectl describe pod showing Requests/Limits
10. kubectl top nodes
11. kubectl top pods
12. kubectl get hpa
13. HPA before load
14. Load generator Running
15. HPA during CPU load
16. Additional Nginx Pods created by scaling
17. HPA after load is removed
18. One troubleshooting example and its Events

Production Kubernetes Troubleshooting Lab

Aisalkyn Aidarova — Thu, 08 Oct 2026 01:37:58 +0000

Observability, Incident Response & Root-Cause Analysis

Scenario: You are the DevOps/SRE on-call team for an e-commerce company.

Architecture:

                    CUSTOMER
                        │
                        ▼
                   order-service
                        │
                        ▼
                    Service
                        │
              ┌─────────┼─────────┐
              ▼         ▼         ▼
            Pod-1     Pod-2     Pod-3
                        │
                        ▼
                  Dependencies

Today we will deliberately create these incidents:

INCIDENT 1 → New deployment → CrashLoopBackOff → Rollback

INCIDENT 2 → Production slow → High traffic / CPU → HPA investigation

INCIDENT 3 → Running Pods but application unavailable → Service selector

INCIDENT 4 → Running 0/1 → Readiness failure

INCIDENT 5 → OOMKilled → Memory investigation

INCIDENT 6 → Application dependency problem → Logs

LAB 0 — Check the Cluster

Everybody starts here.

kubectl cluster-info

Then:

kubectl get nodes

Expected:

NAME                         STATUS   ROLES    AGE
ip-192-168-10-10.ec2...      Ready    <none>   ...
ip-192-168-20-20.ec2...      Ready    <none>   ...

Check:

kubectl get nodes -o wide

Now create our production namespace:

kubectl create namespace production

Check:

kubectl get namespace production

LAB 1 — Build Healthy Production First

Before troubleshooting production, we need healthy production.

Create:

nano production.yaml

Paste:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
  namespace: production
spec:
  replicas: 3

  selector:
    matchLabels:
      app: order-service

  template:
    metadata:
      labels:
        app: order-service

    spec:
      containers:
      - name: order-service
        image: nginx:1.27

        ports:
        - containerPort: 80

        resources:
          requests:
            cpu: "100m"
            memory: "64Mi"
          limits:
            cpu: "500m"
            memory: "256Mi"

        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 5

        livenessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 10
          periodSeconds: 10

---
apiVersion: v1
kind: Service
metadata:
  name: order-service
  namespace: production
spec:
  selector:
    app: order-service

  ports:
  - port: 80
    targetPort: 80

  type: ClusterIP

Apply:

kubectl apply -f production.yaml

Watch:

kubectl get pods -n production -w

Eventually:

NAME                             READY   STATUS
order-service-xxxxxxxxxx-abcde   1/1     Running
order-service-xxxxxxxxxx-fghij   1/1     Running
order-service-xxxxxxxxxx-klmno   1/1     Running

Press:

Ctrl+C

Check the Entire Production Environment

kubectl get all -n production

Check labels:

kubectl get pods -n production --show-labels

You should see:

app=order-service

Check Service:

kubectl get svc -n production

Now:

kubectl get endpoints order-service -n production

Depending on Kubernetes version, you may also use:

kubectl get endpointslice -n production

You should have backend addresses.

Concept:

Service
   │
   │ selector:
   │ app=order-service
   │
   ▼
Pod        Pod        Pod
app=       app=       app=
order      order      order
-service   -service   -service

Everything matches.

Production is healthy.

Verify the Application

Create a temporary troubleshooting Pod:

kubectl run test-client \
  --image=curlimages/curl \
  --restart=Never \
  -n production \
  -- sleep 3600

Check:

kubectl get pod test-client -n production

Then:

kubectl exec -n production test-client -- \
curl -s http://order-service

You should receive the Nginx HTML page.

Now tell students:

This is our known-good production baseline.

That's important.

INCIDENT 1 — New Deployment Breaks Production

STUDENTS ONLY RECEIVE THIS MESSAGE

🚨 INCIDENT: Customers cannot place orders. The Order Service started failing immediately after today's deployment.

Ask students:

What changed?

Answer:

New deployment.

First check:

kubectl get pods -n production

But currently everything works.

INSTRUCTOR — Break Production

Don't show students the command.

Run:

kubectl set image deployment/order-service \
order-service=nginx:this-version-does-not-exist \
-n production

Now students run:

kubectl get pods -n production

They should eventually see something similar to:

order-service-old        1/1   Running
order-service-new        0/1   ImagePullBackOff
order-service-new        0/1   ImagePullBackOff

Important teaching point:

Because this is a rolling update, old healthy Pods may remain available while new Pods fail. So production may be degraded rather than completely unavailable.

Student Investigation

First:

kubectl rollout status deployment/order-service -n production

It will not complete normally.

Then:

kubectl get pods -n production

Pick one failing Pod:

kubectl describe pod <NEW-POD-NAME> -n production

Look at Events.

Students should find something similar to:

Failed to pull image

and:

ErrImagePull

then:

ImagePullBackOff

Now ask:

Is ImagePullBackOff the root cause?

More precisely, it's the Kubernetes symptom/state. The useful cause in the Events is that the requested image/tag cannot be pulled.

Check Deployment:

kubectl get deployment order-service \
-n production \
-o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

It should show:

nginx:this-version-does-not-exist

Check Rollout History

kubectl rollout history deployment/order-service -n production

You'll have revisions.

Now ask:

Production worked before the release. The new rollout cannot start. Should we sit in production and experiment with the image?

No.

Mitigate first.

kubectl rollout undo deployment/order-service -n production

Then:

kubectl rollout status deployment/order-service -n production

Check:

kubectl get pods -n production

Eventually:

READY   STATUS
1/1     Running
1/1     Running
1/1     Running

Check image:

kubectl get deployment order-service \
-n production \
-o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

Should be back to:

nginx:1.27

Verify from customer path:

kubectl exec -n production test-client -- \
curl -s http://order-service

Incident #1 conclusion

New Deployment
      ↓
New Pods fail
      ↓
ImagePullBackOff
      ↓
Events
      ↓
Invalid/unavailable image tag
      ↓
ROLLBACK
      ↓
Previous known-good release
      ↓
VERIFY

INCIDENT 2 — Production Is Slow

Now we make the lab more professional.

Tell students only:

🚨 INCIDENT: Production is slow. Customers report that Order Service responses are taking much longer than normal. There has been no new deployment.

Ask:

What's the cause?

Students should NOT answer:

“Traffic.”

They should say:

We need evidence.

First Establish Baseline

Check:

kubectl get pods -n production

All should be:

1/1 Running

Now:

kubectl top pods -n production

If this command works, save/observe the CPU usage.

If it returns something like:

error: Metrics API not available

that itself is an observability problem: Metrics Server isn't available/configured. Don't confuse that with the application incident.

Check HPA:

kubectl get hpa -n production

There probably isn't one yet.

Create HPA

kubectl autoscale deployment order-service \
--cpu-percent=50 \
--min=3 \
--max=6 \
-n production

Check:

kubectl get hpa -n production

Then:

kubectl describe hpa order-service -n production

If Metrics Server is functioning, after metrics become available you'll see current/target CPU utilization.

Generate Traffic

Run a load-generator Pod:

kubectl run load-generator \
--image=busybox:1.36 \
--restart=Never \
-n production \
-- /bin/sh -c \
'while true; do wget -q -O- http://order-service >/dev/null; done'

One loop may not produce enough CPU to demonstrate scaling. Create several:

for i in 1 2 3 4 5; do
  kubectl run load-$i \
  --image=busybox:1.36 \
  --restart=Never \
  -n production \
  -- /bin/sh -c \
  'while true; do wget -q -O- http://order-service >/dev/null; done'
done

Now watch:

kubectl top pods -n production

Then:

kubectl get hpa -n production -w

And in another terminal:

kubectl get pods -n production -w

What should students investigate?

Request/load increased
        ↓
CPU may increase
        ↓
HPA observes CPU relative to requests
        ↓
Desired replicas may increase
        ↓
Deployment creates additional Pods

Important: don't promise a particular CPU percentage or replica count. It depends on your cluster, Nginx workload, Metrics Server, timing, and how much load the generators produce.

That's actually good for the class: students must observe evidence rather than copy an expected number.

Stop watchers with:

Ctrl+C

Make Scaling Intentionally Too Restrictive

Now create a more interesting scenario.

kubectl patch hpa order-service \
-n production \
-p '{"spec":{"maxReplicas":3}}'

Check:

kubectl get hpa order-service -n production

Now you have:

MINPODS = 3
MAXPODS = 3

The HPA cannot scale beyond three.

Keep load running.

Check:

kubectl top pods -n production

Then:

kubectl describe hpa order-service -n production

Now ask:

What evidence would you need before saying traffic/capacity is the reason for slowness?

Students should correlate:

Load/request rate ↑
CPU/resource pressure ↑
Latency ↑
HPA target exceeded
Replicas already at configured maximum

Not:

Production slow
→ probably traffic.

Stop Traffic

kubectl delete pod load-generator -n production --ignore-not-found

And:

kubectl delete pod \
load-1 load-2 load-3 load-4 load-5 \
-n production \
--ignore-not-found

Restore HPA:

kubectl patch hpa order-service \
-n production \
-p '{"spec":{"maxReplicas":6}}'

INCIDENT 3 — All Pods Running, Application Doesn't Work

This is one of the best incidents.

Tell students:

🚨 INCIDENT: All Order Service Pods show Running 1/1, but requests to the Order Service fail.

Ask:

If Pods are Running, does that prove production is working?

No.

INSTRUCTOR — Break Service

Run:

kubectl patch service order-service \
-n production \
-p '{"spec":{"selector":{"app":"wrong-order-service"}}}'

Students don't see this command.

Student Investigation

They start:

kubectl get pods -n production

Result:

Running
Running
Running

Then try:

kubectl exec -n production test-client -- \
curl --max-time 5 http://order-service

It fails/times out depending on the exact networking behavior.

Now:

kubectl get svc -n production

Service exists.

kubectl get endpoints order-service -n production

Likely:

ENDPOINTS   <none>

On newer clusters also inspect:

kubectl get endpointslice -n production

Now compare:

kubectl describe service order-service -n production

with:

kubectl get pods -n production --show-labels

Service selector:

app=wrong-order-service

Pod label:

app=order-service

There is your evidence.

Pods Running ✓
      ↓
Service exists ✓
      ↓
Endpoints ✗
      ↓
Selector does not match labels
      ↓
ROOT CAUSE

Fix It

kubectl patch service order-service \
-n production \
-p '{"spec":{"selector":{"app":"order-service"}}}'

Check:

kubectl get endpoints order-service -n production

Addresses should return.

Verify:

kubectl exec -n production test-client -- \
curl -s http://order-service

Production works again.

INCIDENT 4 — Running 0/1

Tell students:

🚨 INCIDENT: Pods are running, but they are not becoming Ready.

Instructor breaks readiness:

kubectl patch deployment order-service \
-n production \
--type='json' \
-p='[
  {
    "op":"replace",
    "path":"/spec/template/spec/containers/0/readinessProbe/httpGet/path",
    "value":"/health-does-not-exist"
  }
]'

Watch:

kubectl rollout status deployment/order-service -n production

Students check:

kubectl get pods -n production

New Pods may show:

0/1 Running

Ask:

What's the difference between STATUS and READY?

Now:

kubectl describe pod <POD-NAME> -n production

They should find readiness probe failures such as an HTTP failure status.

Now check:

kubectl get endpoints order-service -n production

Ready backends can disappear as the bad rollout progresses.

Concept:

Container
   ↓
Running
   ↓
Readiness check
   ↓
FAIL
   ↓
Pod not Ready
   ↓
Not a healthy Service backend

What Do We Do in Production?

Again ask:

This happened immediately after a new configuration/deployment. Previous revision worked. What is our fastest safe mitigation?

Rollback.

kubectl rollout history deployment/order-service -n production

Then:

kubectl rollout undo deployment/order-service -n production

Verify:

kubectl rollout status deployment/order-service -n production

Then:

kubectl get pods -n production

And:

kubectl get endpoints order-service -n production

Finally:

kubectl exec -n production test-client -- \
curl -s http://order-service

INCIDENT 5 — OOMKilled

For this one use a separate workload so we don't destroy the main application.

Create:

nano memory-app.yaml

Paste:

apiVersion: v1
kind: Pod
metadata:
  name: memory-app
  namespace: production
spec:
  containers:
  - name: memory-app
    image: python:3.12-alpine

    resources:
      requests:
        memory: "20Mi"
      limits:
        memory: "40Mi"

    command:
      - python
      - -c
      - |
        import time
        data = []
        while True:
            data.append(bytearray(10 * 1024 * 1024))
            print("Allocated more memory", flush=True)
            time.sleep(1)

Apply:

kubectl apply -f memory-app.yaml

Watch:

kubectl get pod memory-app -n production -w

The container should exceed its memory limit and be terminated/restarted. Depending on timing, you may see OOMKilled most clearly in describe rather than as the STATUS column.

Run:

kubectl describe pod memory-app -n production

Look for:

Last State:
  Terminated
Reason:
  OOMKilled

Now:

kubectl logs memory-app -n production

And critically:

kubectl logs memory-app -n production --previous

This is a very good place to teach --previous.

Ask:

Was there a deployment?

No.

Should we automatically rollback?

No.

Evidence points toward memory behavior/resource limit.

Remove:

kubectl delete pod memory-app -n production

INCIDENT 6 — Dependency Failure

Now we'll simulate the idea of:

Order Service
     ↓
Kafka / Database / API

without requiring Kafka installation.

Create:

nano dependency-app.yaml

Paste:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
  namespace: production
spec:
  replicas: 1

  selector:
    matchLabels:
      app: payment-service

  template:
    metadata:
      labels:
        app: payment-service

    spec:
      containers:
      - name: payment-service
        image: busybox:1.36

        command:
        - /bin/sh
        - -c
        - |
          echo "Starting payment-service"
          echo "Connecting to Kafka..."
          sleep 2
          echo "ERROR: Cannot connect to kafka.production.svc:9092"
          echo "Application exiting"
          exit 1

Apply:

kubectl apply -f dependency-app.yaml

Watch:

kubectl get pods -n production -w

Eventually:

CrashLoopBackOff

Now tell students only:

🚨 INCIDENT: Payment Service is unavailable.

They should investigate:

kubectl get pods -n production

Then:

kubectl logs deployment/payment-service -n production

Eventually use the actual Pod name:

kubectl get pods -n production \
-l app=payment-service

Then:

kubectl logs <PAYMENT-POD> \
-n production \
--previous

They'll see:

Starting payment-service
Connecting to Kafka...
ERROR: Cannot connect to kafka.production.svc:9092
Application exiting

Now ask:

Have we proven Kafka itself is down?

No.

This is very important.

We have proven:

Payment application reports that it cannot connect to the Kafka endpoint.

We haven't yet proven why.

Potential investigation branches:

                 Kafka connection failure
                           │
         ┌─────────────────┼─────────────────┐
         ↓                 ↓                 ↓
      DNS/name           Network           Kafka
      wrong?             blocked?          unhealthy?
         │                 │                 │
         ↓                 ↓                 ↓
      Service?        NetworkPolicy?      Brokers?
         │
         ↓
       Port?

That's production thinking.

Delete:

kubectl delete deployment payment-service -n production

FINAL INCIDENT — Students Get No Hints

Now combine everything.

Tell them:

🚨 SEV-1 INCIDENT

Customers report that the Order Service is unavailable.

You are the on-call DevOps team. Investigate and restore service.

Don't tell them anything else.

Instructor secretly breaks:

kubectl patch service order-service \
-n production \
-p '{"spec":{"selector":{"app":"orders"}}}'

Then sit back.

Students should develop this process themselves:

What's affected?
       ↓
Order Service
       ↓
Any deployment?
       ↓
No
       ↓
Pods?
       ↓
Running / Ready
       ↓
Resources?
       ↓
Healthy
       ↓
Service?
       ↓
Exists
       ↓
Endpoints?
       ↓
NONE
       ↓
Why?
       ↓
Selector vs labels
       ↓
Mismatch
       ↓
ROOT CAUSE

They fix:

kubectl patch service order-service \
-n production \
-p '{"spec":{"selector":{"app":"order-service"}}}'

But don't accept:

“Fixed.”

Ask:

PROVE IT.

They need to verify:

kubectl get endpoints order-service -n production

Then:

kubectl exec -n production test-client -- \
curl -s http://order-service

Then:

kubectl get pods -n production

Only now:

Production restored.

The Troubleshooting Command Sheet

This is the part I would have every student save.

# =====================================================
# 1. WORKLOAD
# =====================================================

kubectl get pods -n production

kubectl get pods -n production -o wide

kubectl describe pod <pod> -n production


# =====================================================
# 2. LOGS
# =====================================================

kubectl logs <pod> -n production

kubectl logs <pod> -n production --previous


# =====================================================
# 3. EVENTS
# =====================================================

kubectl get events \
-n production \
--sort-by=.metadata.creationTimestamp


# =====================================================
# 4. METRICS
# =====================================================

kubectl top pods -n production

kubectl top nodes


# =====================================================
# 5. DEPLOYMENT
# =====================================================

kubectl get deployment -n production

kubectl describe deployment <deployment> -n production

kubectl rollout status deployment/<deployment> -n production

kubectl rollout history deployment/<deployment> -n production

kubectl rollout undo deployment/<deployment> -n production


# =====================================================
# 6. NETWORK
# =====================================================

kubectl get svc -n production

kubectl describe svc <service> -n production

kubectl get endpoints -n production

kubectl get endpointslice -n production

kubectl get ingress -n production

kubectl get networkpolicy -n production


# =====================================================
# 7. STORAGE
# =====================================================

kubectl get pvc -n production

kubectl get pv

kubectl get storageclass

kubectl describe pvc <pvc> -n production


# =====================================================
# 8. SCALING
# =====================================================

kubectl get hpa -n production

kubectl describe hpa <hpa> -n production


# =====================================================
# 9. SECURITY
# =====================================================

kubectl auth can-i get pods -n production

kubectl auth can-i delete pods -n production


# =====================================================
# 10. CONFIGURATION
# =====================================================

kubectl get configmap -n production

kubectl get secret -n production

The Production Rule Students Need to Remember

Don't teach them this:

Production Down
     ↓
kubectl get pods
     ↓
restart everything

Teach them this:

                    INCIDENT
                       │
                       ▼
              WHAT IS THE IMPACT?
                       │
                       ▼
               WHAT CHANGED?
                       │
                       ▼
          IDENTIFY AFFECTED COMPONENT
                       │
                       ▼
         ┌─────────────┼─────────────┐
         ▼             ▼             ▼
       LOGS          METRICS       EVENTS
         │             │             │
         └─────────────┼─────────────┘
                       ▼
                  HYPOTHESIS
                       │
                       ▼
                    EVIDENCE
                       │
                       ▼
                  ROOT CAUSE
                       │
             ┌─────────┴─────────┐
             ▼                   ▼
         MITIGATE               FIX
       e.g. rollback
             │
             ▼
           VERIFY
             │
             ▼
      PRODUCTION RESTORED
             │
             ▼
       ROOT CAUSE ANALYSIS

And throughout the lab, whenever a student says:

“I think it's traffic.”

Your response should be:

“Show me the evidence.”

If they say:

“It's Kubernetes.”

Ask:

“Which Kubernetes component? Show me the evidence.”

If they say:

“Kafka is down.”

Ask:

“Have you proven Kafka is down, or have you only proven that your application cannot connect to Kafka?”

If they say:

“I fixed it.”

Pods Deployments ReplicaSets StatefulSets Storage Troubleshooting

Aisalkyn Aidarova — Tue, 06 Oct 2026 00:49:16 +0000

🚀 JumpToTech Kubernetes Core Hands-On Lab

Objective

By the end of this lab, you should understand:

Control Plane
Worker Nodes
Pods
Containers
Deployment
ReplicaSet
Self-healing
Scaling
Labels and selectors
StatefulSet
Stable Pod identity
PersistentVolume (PV)
PersistentVolumeClaim (PVC)
StorageClass
Basic Kubernetes troubleshooting

PART 1 — Understand Your Kubernetes Cluster

Before creating anything, check the cluster.

Run:

kubectl cluster-info

Then:

kubectl get nodes

Now:

kubectl get nodes -o wide

Question

What are these nodes?

Answer

Nodes are machines participating in the Kubernetes cluster.

Application workloads normally run on Worker Nodes.

A simplified architecture looks like this:

                    KUBERNETES CLUSTER

                       CONTROL PLANE
                 ┌─────────────────────┐
                 │ API Server          │
                 │ Scheduler           │
                 │ Controller Manager  │
                 │ etcd                │
                 └──────────┬──────────┘
                            │
                 ───────────┼──────────
                            │
                ┌───────────┴───────────┐
                ↓                       ↓
          WORKER NODE 1           WORKER NODE 2
          ┌────────────┐          ┌────────────┐
          │ kubelet    │          │ kubelet    │
          │ Pod        │          │ Pod        │
          │ Container  │          │ Container  │
          └────────────┘          └────────────┘

Important components

API Server

Receives Kubernetes API requests.

For example:

kubectl get pods

and:

kubectl apply -f deployment.yaml

communicate with the Kubernetes API.

Scheduler

Decides which Worker Node should run a newly created unscheduled Pod.

Controller Manager

Runs controllers that continuously work to make the actual state match the desired state.

etcd

Stores Kubernetes cluster state.

kubelet

Runs on each Worker Node and makes sure assigned Pods are running.

PART 2 — Create a Standalone Pod

Create a new directory:

mkdir kubernetes-core-lab
cd kubernetes-core-lab

Create:

nano pod.yaml

Add:

apiVersion: v1
kind: Pod

metadata:
  name: nginx-pod
  labels:
    app: nginx

spec:
  containers:
    - name: nginx
      image: nginx:1.27
      ports:
        - containerPort: 80

Save the file.

Apply it:

kubectl apply -f pod.yaml

Check:

kubectl get pods

You should see:

NAME        READY   STATUS    RESTARTS   AGE
nginx-pod   1/1     Running   0          ...

Now:

kubectl get pods -o wide

Question

What additional information do you see?

You should see information including:

IP
NODE

The NODE column tells you which Worker Node is running the Pod.

PART 3 — Inspect the Pod

Run:

kubectl describe pod nginx-pod

Find:

Node:
Labels:
Status:
IP:
Containers:
Events:

Important

kubectl describe is one of the most important troubleshooting commands.

When something is wrong, always look at:

Events

near the bottom.

Now check logs:

kubectl logs nginx-pod

PART 4 — Delete the Standalone Pod

Run:

kubectl delete pod nginx-pod

Check:

kubectl get pods

Question

Did the Pod come back?

Answer

No.

Why?

Because we created a standalone Pod.

There is no higher-level workload controller responsible for maintaining that Pod.

This demonstrates an important production concept:

Standalone Pod

DELETE POD
    ↓
Pod disappears
    ↓
Nothing recreates it

This is one reason production applications are normally managed through higher-level workload resources such as Deployments rather than manually creating individual Pods.

PART 5 — Create a Deployment

Now create:

nano deployment.yaml

Add:

apiVersion: apps/v1
kind: Deployment

metadata:
  name: nginx-deployment

spec:
  replicas: 3

  selector:
    matchLabels:
      app: nginx

  template:
    metadata:
      labels:
        app: nginx

    spec:
      containers:
        - name: nginx
          image: nginx:1.27
          ports:
            - containerPort: 80

Apply:

kubectl apply -f deployment.yaml

Check:

kubectl get deployments

Then:

kubectl get rs

Then:

kubectl get pods

PART 6 — Understand What Kubernetes Created

We created only:

Deployment

But Kubernetes created:

Deployment
     ↓
ReplicaSet
     ↓
Pods

Run:

kubectl get deployment

Then:

kubectl get rs

Then:

kubectl get pods

Question

Did we manually create the ReplicaSet?

Answer

No.

The Deployment created and manages the ReplicaSet.

Question

Did we manually create the individual Pods?

Answer

No.

The ReplicaSet maintains the required Pod replicas.

Conceptually:

Deployment
     │
     │ manages
     ↓
ReplicaSet
     │
     │ maintains
     ↓
┌─────────┐
│ Pod #1  │
├─────────┤
│ Pod #2  │
├─────────┤
│ Pod #3  │
└─────────┘

PART 7 — Desired State

Look inside deployment.yaml:

replicas: 3

This tells Kubernetes:

DESIRED STATE = 3 Pods

Check:

kubectl get deployment nginx-deployment

You may see something similar to:

NAME               READY   UP-TO-DATE   AVAILABLE
nginx-deployment   3/3     3            3

Kubernetes continuously tries to make:

ACTUAL STATE
     =
DESIRED STATE

This process is called reconciliation.

PART 8 — Test Kubernetes Self-Healing

Get Pods:

kubectl get pods

Copy one Deployment Pod name.

Example:

nginx-deployment-7c79c4bf97-abcd1

Delete it:

kubectl delete pod nginx-deployment-7c79c4bf97-abcd1

Immediately run:

kubectl get pods

Better:

kubectl get pods -w

Watch carefully.

You should see a replacement Pod.

Press:

CTRL+C

to stop watching.

What happened?

Before:

Desired = 3
Actual  = 3

We deleted one:

Desired = 3
Actual  = 2

Kubernetes detects the difference.

The ReplicaSet works to restore:

Desired = 3
Actual  = 3

Interview Question

Why did the Pod come back?

Answer

Because the Deployment's ReplicaSet maintains the desired number of replicas. When one Pod disappeared, the ReplicaSet created a replacement.

PART 9 — Scale the Deployment

Currently:

replicas: 3

Scale to 6:

kubectl scale deployment nginx-deployment --replicas=6

Check:

kubectl get deployment

Then:

kubectl get pods

Question

How many Pods?

Answer:

Now:

kubectl get rs

The ReplicaSet should show the desired/current replica counts.

Now scale down:

kubectl scale deployment nginx-deployment --replicas=2

Watch:

kubectl get pods -w

Kubernetes terminates extra Pods.

Stop with:

CTRL+C

PART 10 — Labels and Selectors

Run:

kubectl get pods --show-labels

You should see labels including:

app=nginx

Now:

kubectl get pods -l app=nginx

Why are labels important?

Labels allow Kubernetes resources to identify/select other resources.

Our Deployment contains:

selector:
  matchLabels:
    app: nginx

and the Pod template contains:

labels:
  app: nginx

These must match for this Deployment.

Think:

Deployment selector
        ↓
    app=nginx
        ↓
   Matching Pods

PART 11 — Deployment Update

Check the current image:

kubectl get deployment nginx-deployment \
  -o jsonpath='{.spec.template.spec.containers[0].image}'

Now update Nginx:

kubectl set image deployment/nginx-deployment \
  nginx=nginx:1.28

Watch:

kubectl get pods -w

Stop with:

CTRL+C

Check rollout:

kubectl rollout status deployment/nginx-deployment

Check history:

kubectl rollout history deployment/nginx-deployment

What happened?

Kubernetes performed a rollout.

A Deployment allows us to update an application while Kubernetes manages replacement of Pods according to its rollout strategy.

PART 12 — Rollback

Suppose our new application version has a problem.

Rollback:

kubectl rollout undo deployment/nginx-deployment

Check:

kubectl rollout status deployment/nginx-deployment

Then:

kubectl get pods

Interview Question

How do you rollback a Deployment?

Answer:

kubectl rollout undo deployment/<deployment-name>

PART 13 — Deployment vs StatefulSet

Now we move to StatefulSet.

Deployment Pods might look like:

nginx-deployment-76f8b987d8-x8j2p
nginx-deployment-76f8b987d8-82ksa
nginx-deployment-76f8b987d8-p91ka

The exact generated suffixes can change.

StatefulSet Pods look like:

web-0
web-1
web-2

StatefulSet gives Pods stable ordinal identities.

Concept:

DEPLOYMENT

Pod A
Pod B
Pod C

Pods are generally interchangeable.

Compared with:

STATEFULSET

web-0
web-1
web-2

Each Pod has a stable identity.

PART 14 — Create a StorageClass

First check existing StorageClasses:

kubectl get storageclass

If your cluster already has a default StorageClass, record its name:

kubectl get storageclass

Example on an AWS EKS cluster might be:

gp2

or:

gp3

depending on the cluster configuration.

Do not blindly create a new StorageClass if your cluster already has the correct one.

Set the StorageClass name you actually have:

export STORAGE_CLASS=gp2

Replace gp2 if your cluster uses a different StorageClass.

Verify:

kubectl get storageclass "$STORAGE_CLASS"

PART 15 — Create a StatefulSet

Create:

nano statefulset.yaml

Add the following.

Replace gp2 with your cluster's StorageClass if necessary.

apiVersion: v1
kind: Service
metadata:
  name: nginx-headless
spec:
  clusterIP: None
  selector:
    app: nginx-stateful
  ports:
    - port: 80
      targetPort: 80
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nginx-stateful
spec:
  serviceName: nginx-headless
  replicas: 3

  selector:
    matchLabels:
      app: nginx-stateful

  template:
    metadata:
      labels:
        app: nginx-stateful

    spec:
      containers:
        - name: nginx
          image: nginx:1.27

          ports:
            - containerPort: 80

          volumeMounts:
            - name: nginx-data
              mountPath: /usr/share/nginx/html

  volumeClaimTemplates:
    - metadata:
        name: nginx-data

      spec:
        accessModes:
          - ReadWriteOnce

        storageClassName: gp2

        resources:
          requests:
            storage: 1Gi

Apply:

kubectl apply -f statefulset.yaml

Watch:

kubectl get pods -w

You should eventually see:

nginx-stateful-0
nginx-stateful-1
nginx-stateful-2

PART 16 — Observe StatefulSet Identity

Run:

kubectl get statefulset

Then:

kubectl get pods

Notice:

nginx-stateful-0
nginx-stateful-1
nginx-stateful-2

Compare:

DEPLOYMENT

nginx-deployment-xxxxx-abc12
nginx-deployment-xxxxx-def34

with:

STATEFULSET

nginx-stateful-0
nginx-stateful-1
nginx-stateful-2

This stable identity is one of the most important StatefulSet concepts.

PART 17 — PVC and PV

Run:

kubectl get pvc

You may see:

nginx-data-nginx-stateful-0
nginx-data-nginx-stateful-1
nginx-data-nginx-stateful-2

Check:

kubectl get pv

Now:

kubectl get storageclass

Understand the relationship:

StatefulSet
     ↓
nginx-stateful-0
     ↓
PVC
     ↓
PV
     ↓
Storage

PVC

PersistentVolumeClaim.

The application requests storage.

PV

PersistentVolume.

Represents persistent storage available/bound for Kubernetes use.

StorageClass

Describes a class of storage and, with dynamic provisioning, tells Kubernetes which provisioner and parameters to use to create storage.

PART 18 — Write Data

Let's demonstrate why persistent storage matters.

Run:

kubectl exec nginx-stateful-0 -- \
  sh -c 'echo "Hello from JumpToTech StatefulSet" > /usr/share/nginx/html/index.html'

Check:

kubectl exec nginx-stateful-0 -- \
  cat /usr/share/nginx/html/index.html

Expected:

Hello from JumpToTech StatefulSet

PART 19 — Delete the StatefulSet Pod

Delete:

kubectl delete pod nginx-stateful-0

Watch:

kubectl get pods -w

You should see:

nginx-stateful-0

created again.

Notice the name.

It is still:

nginx-stateful-0

not a random new identity.

Once it is Running, check the file:

kubectl exec nginx-stateful-0 -- \
  cat /usr/share/nginx/html/index.html

If storage was correctly provisioned and reattached, you should still see:

Hello from JumpToTech StatefulSet

This demonstrates two different concepts

Stable identity
+
Persistent storage

Do not confuse them.

StatefulSet gives the Pod stable identity.

PVC/PV provide persistent storage.

PART 20 — Compare Deployment and StatefulSet

Feature	Deployment	StatefulSet
Common use	Stateless apps	Stateful apps
Pod identity	Interchangeable/generated	Stable ordinal identity
Example	Web/API	Databases/clustered stateful systems
Typical names	app-xxxxx-xxxxx	app-0, app-1
Scaling	Designed for interchangeable replicas	Can use ordered behavior
Storage	Can use persistent storage	Commonly paired with per-Pod PVCs
Rollout	Deployment strategy	StatefulSet strategy

Important:

A StatefulSet does not automatically make a database production-ready.

The database itself must still be configured correctly for replication, backups, failover, security, and recovery.

PART 21 — BREAK THE APPLICATION

Now we intentionally create problems.

This is the most important section.

A DevOps engineer does not only deploy applications.

A DevOps engineer must be able to troubleshoot them.

TROUBLESHOOTING LAB 1 — Wrong Image

Run:

kubectl set image deployment/nginx-deployment \
  nginx=nginx:this-image-does-not-exist

Wait:

kubectl get pods

You may see:

ErrImagePull

followed by:

ImagePullBackOff

What do you do?

First:

kubectl describe pod <problem-pod-name>

Go to:

Events

You should find information about the image pull failure.

Question

Is Kubernetes broken?

Answer

No.

The requested container image/tag cannot be successfully pulled.

Fix:

kubectl set image deployment/nginx-deployment \
  nginx=nginx:1.27

Check:

kubectl rollout status deployment/nginx-deployment

TROUBLESHOOTING LAB 2 — CrashLoopBackOff

Create:

nano broken-pod.yaml

Add:

apiVersion: v1
kind: Pod

metadata:
  name: crash-demo

spec:
  containers:
    - name: crash
      image: busybox:1.36
      command:
        - sh
        - -c
        - echo "Application starting"; sleep 3; exit 1

Apply:

kubectl apply -f broken-pod.yaml

Watch:

kubectl get pods -w

Eventually the container will repeatedly restart and may show:

CrashLoopBackOff

Stop watching:

CTRL+C

Now troubleshoot:

kubectl describe pod crash-demo

Then:

kubectl logs crash-demo

And:

kubectl logs crash-demo --previous

Important troubleshooting pattern

Something is wrong
        ↓
kubectl get pods
        ↓
kubectl describe pod
        ↓
Check Events
        ↓
kubectl logs
        ↓
kubectl logs --previous

TROUBLESHOOTING LAB 3 — Pending PVC

Do this as a demonstration rather than leaving broken resources in the cluster.

Create:

nano bad-pvc.yaml

Add:

apiVersion: v1
kind: PersistentVolumeClaim

metadata:
  name: broken-pvc

spec:
  accessModes:
    - ReadWriteOnce

  storageClassName: storage-class-does-not-exist

  resources:
    requests:
      storage: 1Gi

Apply:

kubectl apply -f bad-pvc.yaml

Check:

kubectl get pvc

You should see it remain:

Pending

Investigate:

kubectl describe pvc broken-pvc

Then:

kubectl get storageclass

Lesson

If you see:

Pod Pending

and that Pod needs a PVC, check:

kubectl get pvc
kubectl describe pvc <pvc-name>
kubectl get storageclass

Delete the intentionally broken claim:

kubectl delete pvc broken-pvc

PART 22 — The Kubernetes Troubleshooting Mindset

Do not randomly delete things.

Follow a process.

Step 1

Check:

kubectl get pods

Step 2

Get more information:

kubectl get pods -o wide

Step 3

Describe the problematic resource:

kubectl describe pod <pod-name>

Step 4

Read:

Events

Step 5

Check application logs:

kubectl logs <pod-name>

For a previously crashed container:

kubectl logs <pod-name> --previous

Step 6

If storage is involved:

kubectl get pvc
kubectl get pv
kubectl get storageclass
kubectl describe pvc <pvc-name>

Step 7

If Deployment is involved:

kubectl get deployment
kubectl get rs
kubectl rollout status deployment/<name>
kubectl rollout history deployment/<name>

PART 23 — Final Architecture

You should now understand this:

USER
 │
 │ kubectl apply
 ↓
API SERVER
 │
 ↓
Desired state stored/processed
 │
 ↓
CONTROLLERS
 │
 ↓
Deployment
 │
 ↓
ReplicaSet
 │
 ↓
Pods
 │
 ↓
SCHEDULER
 │
 │ chooses Node
 ↓
WORKER NODE
 │
 ↓
kubelet
 │
 ↓
Container Runtime
 │
 ↓
Container Running

For a StatefulSet:

StatefulSet
     │
     ├── nginx-stateful-0
     │         │
     │         └── PVC → PV → Storage
     │
     ├── nginx-stateful-1
     │         │
     │         └── PVC → PV → Storage
     │
     └── nginx-stateful-2
               │
               └── PVC → PV → Storage

PART 24 — Student Challenge

Do this section without instructor help.

Challenge 1

Create a Deployment named:

student-web

Requirements:

Image: nginx:1.27
Replicas: 4
Label: app=student-web
Container port: 80

Verify:

kubectl get deployment
kubectl get rs
kubectl get pods

Challenge 2

Delete one Pod.

Prove Kubernetes recreates it.

Challenge 3

Scale:

4 → 7

Then:

7 → 2

Challenge 4

Update Nginx to:

nginx:1.28

Watch the rollout.

Challenge 5

Break the image intentionally.

Use:

nginx:does-not-exist

Troubleshoot it.

Then repair it.

Challenge 6

Explain:

Deployment
    ↓
ReplicaSet
    ↓
Pod
    ↓
Container

Challenge 7

Explain the difference between:

Deployment

and:

StatefulSet

without looking at your notes.

FINAL INTERVIEW QUESTIONS

1. What is a Pod?

A Pod is Kubernetes' smallest deployable unit and contains one or more containers.

2. What is a Deployment?

A Deployment manages stateless application workloads and declaratively manages ReplicaSets and application updates.

3. What is a ReplicaSet?

A ReplicaSet maintains the desired number of matching Pod replicas.

4. What happens when a Deployment Pod dies?

Its ReplicaSet works to create a replacement so the desired replica count is restored.

5. What is StatefulSet?

A workload controller designed for applications that need stable Pod identities and often stable per-Pod storage.

6. Deployment vs StatefulSet?

Deployment replicas are generally interchangeable.

StatefulSet Pods have stable ordinal identities such as:

mysql-0
mysql-1
mysql-2

7. What does Scheduler do?

It chooses an appropriate Node for an unscheduled Pod.

8. What does kubelet do?

It runs on each Worker Node and works with the container runtime to ensure assigned Pods are running.

9. What does API Server do?

It exposes the Kubernetes API and is the central entry point for API operations.

10. What is etcd?

It is the distributed key-value store used by Kubernetes to store cluster state.

11. What is PV?

PersistentVolume — a Kubernetes resource representing persistent storage.

12. What is PVC?

PersistentVolumeClaim — a request for persistent storage.

13. What does PVC `Bound` mean?

The claim has been successfully bound to a PersistentVolume.

14. What does `Pending` mean?

The resource cannot yet proceed to its required running/bound state. You need to investigate why.

15. What is `ImagePullBackOff`?

Kubernetes cannot successfully pull the requested container image and is backing off between retry attempts.

16. What is `CrashLoopBackOff`?

A container repeatedly starts, fails, and Kubernetes delays subsequent restart attempts.

17. First command when a Pod has a problem?

Start with:

kubectl get pods

Then investigate:

kubectl describe pod <pod-name>

and:

kubectl logs <pod-name>

18. Where do you look in `kubectl describe`?

Especially:

Events

19. How do you see which Node runs a Pod?

kubectl get pods -o wide

20. Explain what happens after `kubectl apply -f deployment.yaml`.

A strong answer:

kubectl sends an API request to the Kubernetes API Server.

The requested desired state is persisted in the cluster.

The Deployment controller observes the Deployment and manages the appropriate ReplicaSet.

The ReplicaSet works to create the required Pods.

The Scheduler assigns unscheduled Pods to suitable Nodes.

The kubelet on each selected Worker Node observes the assigned Pod and works with the container runtime to start the required containers.

Kubernetes controllers continuously reconcile actual state with desired state.

CLEANUP

Delete the lab resources:

kubectl delete -f deployment.yaml
kubectl delete -f statefulset.yaml
kubectl delete -f broken-pod.yaml
kubectl delete pod nginx-pod --ignore-not-found
kubectl delete pvc broken-pvc --ignore-not-found
kubectl delete deployment student-web --ignore-not-found

Check:

kubectl get pods
kubectl get deployment
kubectl get statefulset
kubectl get pvc

What You Must Remember

CONTROL PLANE
     ↓
Manages the cluster

WORKER NODE
     ↓
Runs workloads

POD
     ↓
Runs container(s)

DEPLOYMENT
     ↓
Manages ReplicaSets and stateless application rollout

REPLICASET
     ↓
Maintains Pod replicas

STATEFULSET
     ↓
Stable Pod identity

PVC
     ↓
Requests storage

PV
     ↓
Represents persistent storage

And the most important Kubernetes idea:

        DESIRED STATE
              ↓
       Kubernetes checks
              ↓
         ACTUAL STATE
              ↓
      Something is wrong?
              ↓
        RECONCILIATION
              ↓
   Move actual state toward
       the desired state

JumpToTech — LIVE Production Troubleshooting Lab

Aisalkyn Aidarova — Fri, 02 Oct 2026 01:03:50 +0000

JumpToTech — LIVE Production Troubleshooting Lab

StatefulSet + PVC + Service + DNS + Failures + Argo CD

Duration: 3 hours

Rule for today:

Do NOT immediately delete the Pod.

For every incident:

OBSERVE → DESCRIBE → LOGS → EVENTS → DEPENDENCIES → FIX → VERIFY

PART 1 — CREATE THE LAB

Step 1 — Create the folder

mkdir stateful-production-lab
cd stateful-production-lab

mkdir k8s
cd k8s

Check:

pwd

Step 2 — Namespace

Create:

touch 00-namespace.yaml

Open it:

code 00-namespace.yaml

Add:

apiVersion: v1
kind: Namespace
metadata:
  name: production

Step 3 — Secret

Create:

touch 01-secret.yaml

Add:

apiVersion: v1
kind: Secret
metadata:
  name: mysql-secret
  namespace: production
type: Opaque
stringData:
  MYSQL_ROOT_PASSWORD: jumptotech123
  MYSQL_DATABASE: restaurant
  MYSQL_USER: appuser
  MYSQL_PASSWORD: apppassword

This is a classroom Secret.

Do not use plaintext production passwords in Git like this in a real environment.

Step 4 — Headless Service

Create:

touch 02-mysql-headless.yaml

Add:

apiVersion: v1
kind: Service
metadata:
  name: mysql-headless
  namespace: production
spec:
  clusterIP: None

  selector:
    app: mysql

  ports:
    - name: mysql
      port: 3306
      targetPort: 3306

Explain:

mysql-0.mysql-headless.production.svc.cluster.local

StatefulSet uses stable Pod identity.

Step 5 — StatefulSet

Create:

touch 03-mysql-statefulset.yaml

Add:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
  namespace: production

spec:
  serviceName: mysql-headless

  replicas: 1

  selector:
    matchLabels:
      app: mysql

  template:
    metadata:
      labels:
        app: mysql

    spec:
      terminationGracePeriodSeconds: 30

      containers:
        - name: mysql
          image: mysql:8.0

          ports:
            - name: mysql
              containerPort: 3306

          envFrom:
            - secretRef:
                name: mysql-secret

          resources:
            requests:
              cpu: 100m
              memory: 256Mi

            limits:
              cpu: 500m
              memory: 512Mi

          readinessProbe:
            exec:
              command:
                - sh
                - -c
                - mysqladmin ping -h 127.0.0.1 -uroot -p"$MYSQL_ROOT_PASSWORD"

            initialDelaySeconds: 15
            periodSeconds: 10
            timeoutSeconds: 5

          livenessProbe:
            exec:
              command:
                - sh
                - -c
                - mysqladmin ping -h 127.0.0.1 -uroot -p"$MYSQL_ROOT_PASSWORD"

            initialDelaySeconds: 30
            periodSeconds: 20
            timeoutSeconds: 5

          volumeMounts:
            - name: mysql-data
              mountPath: /var/lib/mysql

  volumeClaimTemplates:
    - metadata:
        name: mysql-data

      spec:
        accessModes:
          - ReadWriteOnce

        resources:
          requests:
            storage: 2Gi

Step 6 — Apply

Return to the project root:

cd ..

Apply:

kubectl apply -f k8s/

Check:

kubectl get pods -n production

Wait until:

NAME      READY   STATUS
mysql-0   1/1     Running

Check StatefulSet:

kubectl get sts -n production

Check PVC:

kubectl get pvc -n production

Check PV:

kubectl get pv

Check Service:

kubectl get svc -n production

STOP HERE AND EXPLAIN

Ask students:

What did Kubernetes create?

Draw:

StatefulSet
     |
     v
 mysql-0
     |
     v
mysql-data-mysql-0
     |
     v
    PV
     |
     v
Persistent Storage

Now:

kubectl get pod mysql-0 \
  -n production \
  -o jsonpath='{.metadata.name}'

Output:

mysql-0

Stable identity.

PART 2 — PROVE THAT DATA SURVIVES

Step 7 — Create actual data

Enter MySQL:

kubectl exec -it mysql-0 \
  -n production \
  -- mysql -u root -pjumptotech123

Run:

USE restaurant;

Create table:

CREATE TABLE customers (
    id INT AUTO_INCREMENT PRIMARY KEY,
    name VARCHAR(100),
    email VARCHAR(100)
);

Insert data:

INSERT INTO customers (name, email)
VALUES
('Emma', 'emma@jumptotech.com');

Check:

SELECT * FROM customers;

Expected:

+----+-------+----------------------+
| id | name  | email                |
+----+-------+----------------------+
|  1 | Emma  | emma@jumptotech.com |
+----+-------+----------------------+

Exit:

exit;

Step 8 — Delete the Pod

Now deliberately delete the database Pod:

kubectl delete pod mysql-0 -n production

Immediately:

kubectl get pods -n production -w

StatefulSet creates:

mysql-0

again.

Wait until:

mysql-0   1/1   Running

Step 9 — Did we lose the database?

Connect again:

kubectl exec -it mysql-0 \
  -n production \
  -- mysql -u root -pjumptotech123

Run:

USE restaurant;

SELECT * FROM customers;

The data should still exist.

Explain:

POD DELETED
    X

PVC NOT DELETED
    ✓

PV NOT DELETED
    ✓

DATA NOT DELETED
    ✓


New mysql-0
     |
     v
same PVC
     |
     v
persistent data

This is the foundation of today's incidents.

INCIDENT 1 — CLIENT CANNOT CONNECT TO DATABASE

Time: approximately 35 minutes into class

Tell students only:

Production alert:

Application cannot connect to database.

Find the problem.

Do NOT tell them what is broken.

First:

kubectl get pods -n production

Output:

mysql-0   1/1   Running

Ask:

Is production healthy just because the Pod is Running?

No.

Check Service:

kubectl get svc -n production

Check endpoints:

kubectl get endpoints -n production

Check EndpointSlices:

kubectl get endpointslices -n production

Now create a troubleshooting Pod:

kubectl run troubleshoot \
  --image=busybox:1.36 \
  --restart=Never \
  -n production \
  -- sleep 3600

Wait:

kubectl get pod troubleshoot -n production

Test DNS

kubectl exec troubleshoot \
  -n production \
  -- nslookup mysql-headless

Then:

kubectl exec troubleshoot \
  -n production \
  -- nslookup mysql-headless.production.svc.cluster.local

If DNS works, say:

Stop.

We just proved Kubernetes DNS can resolve the Service.

Therefore, don't continue saying “maybe DNS.”

Move to the next layer.

Test the Port

BusyBox commonly provides nc.

Run:

kubectl exec troubleshoot \
  -n production \
  -- nc -zv mysql-headless 3306

We want connectivity to:

mysql-headless:3306

Architecture:

Troubleshooting Pod
        |
        | DNS
        v
mysql-headless
        |
        | TCP 3306
        v
     mysql-0

NOW BREAK IT

Open:

code k8s/02-mysql-headless.yaml

Change:

selector:
  app: mysql

to:

selector:
  app: mysql-broken

Apply:

kubectl apply -f k8s/02-mysql-headless.yaml

Check Pods:

kubectl get pods -n production

Still:

mysql-0   1/1   Running

Now:

kubectl get endpoints mysql-headless -n production

Students should notice the Service no longer has the expected backend.

Check labels:

kubectl get pods \
  -n production \
  --show-labels

Compare:

SERVICE SELECTOR

app=mysql-broken


POD LABEL

app=mysql

Root cause:

Service
   |
   X
Endpoint
   |
   X
mysql-0

FIX INCIDENT 1

Change back:

selector:
  app: mysql

Apply:

kubectl apply -f k8s/02-mysql-headless.yaml

Verify:

kubectl get endpoints mysql-headless -n production

Test again:

kubectl exec troubleshoot \
  -n production \
  -- nc -zv mysql-headless 3306

Lesson:

Pod Running ≠ Application Reachable

INCIDENT 2 — WRONG PORT

Tell students:

New incident.

Database Pod is healthy.

DNS resolves.

Application still cannot connect.

Break Service.

Change:

targetPort: 3306

to:

targetPort: 3307

Apply:

kubectl apply -f k8s/02-mysql-headless.yaml

Now:

kubectl get pods -n production

Healthy.

DNS:

kubectl exec troubleshoot \
  -n production \
  -- nslookup mysql-headless

DNS works.

Now connectivity:

kubectl exec troubleshoot \
  -n production \
  -- nc -zv mysql-headless 3306

It should fail to reach the actual MySQL listener through the misconfigured Service path.

Ask:

DNS works. Pod works. What is between them?

Answer:

SERVICE

Inspect:

kubectl describe svc mysql-headless -n production

Look at:

Port:        3306
TargetPort:  3307

But MySQL listens on:

Therefore:

Application
     |
     | 3306
     v
Service
     |
     | WRONG → 3307
     X
MySQL
     |
     | listening 3306

FIX INCIDENT 2

Restore:

targetPort: 3306

Apply:

kubectl apply -f k8s/02-mysql-headless.yaml

Verify:

kubectl exec troubleshoot \
  -n production \
  -- nc -zv mysql-headless 3306

INCIDENT 3 — DATABASE POD RUNNING BUT NOT READY

Tell students:

Monitoring says database Pod exists, but it is not Ready.

Break readiness probe.

Open:

code k8s/03-mysql-statefulset.yaml

Find:

readinessProbe:

Change its command to intentionally use the wrong port:

readinessProbe:
  exec:
    command:
      - sh
      - -c
      - mysqladmin ping -h 127.0.0.1 -P 3307 -uroot -p"$MYSQL_ROOT_PASSWORD"
  initialDelaySeconds: 15
  periodSeconds: 10
  timeoutSeconds: 5

Apply:

kubectl apply -f k8s/03-mysql-statefulset.yaml

Watch:

kubectl get pods -n production -w

You may see:

mysql-0   0/1   Running

Ask:

Is the container Running?

Yes.

Is Kubernetes considering it Ready?

No.

Investigate:

kubectl describe pod mysql-0 -n production

Go to:

Events:

Then:

kubectl get events \
  -n production \
  --sort-by=.lastTimestamp

Check logs too:

kubectl logs mysql-0 -n production

Explain:

STATUS
Running

READY
0/1

means:

PROCESS EXISTS

but

POD IS NOT READY FOR NORMAL SERVICE TRAFFIC

FIX INCIDENT 3

Restore:

readinessProbe:
  exec:
    command:
      - sh
      - -c
      - mysqladmin ping -h 127.0.0.1 -uroot -p"$MYSQL_ROOT_PASSWORD"

Apply:

kubectl apply -f k8s/03-mysql-statefulset.yaml

Watch:

kubectl get pods -n production -w

Wait for:

mysql-0   1/1   Running

INCIDENT 4 — IMAGEPULLBACKOFF

Tell students:

Developer released a new database version.

Production deployment started failing.

Break:

image: mysql:8.0

Change to:

image: mysql:jumptotech-does-not-exist

Apply:

kubectl apply -f k8s/03-mysql-statefulset.yaml

Watch:

kubectl get pods -n production -w

Possible:

ErrImagePull

ImagePullBackOff

Ask students:

What is your first investigation command?

Use:

kubectl describe pod mysql-0 -n production

Look at Events.

Then:

kubectl get events \
  -n production \
  --sort-by=.lastTimestamp

Root cause:

Requested image/tag doesn't exist.

Possible real causes:

wrong repository
wrong tag
private registry authentication
imagePullSecret
registry/network issue

FIX INCIDENT 4

Restore:

image: mysql:8.0

Apply:

kubectl apply -f k8s/03-mysql-statefulset.yaml

Watch recovery:

kubectl get pods -n production -w

Then verify the data again:

kubectl exec -it mysql-0 \
  -n production \
  -- mysql -u root -pjumptotech123

USE restaurant;

SELECT * FROM customers;

Your data should still exist.

10-MINUTE BREAK

At this point students have diagnosed:

✓ DNS

✓ Service selector

✓ Endpoints

✓ Wrong port

✓ Readiness

✓ ImagePullBackOff

✓ Stateful persistence

Now move specifically into storage.

INCIDENT 5 — STORAGE TROUBLESHOOTING

Start healthy:

kubectl get pvc -n production

Expected:

mysql-data-mysql-0   Bound

Ask:

If the Pod becomes Pending or ContainerCreating because of storage, where do we look?

First:

kubectl describe pod mysql-0 -n production

Then:

kubectl get pvc -n production

Then:

kubectl describe pvc mysql-data-mysql-0 \
  -n production

Then:

kubectl get pv

Then:

kubectl get sc

Then Events:

kubectl get events \
  -n production \
  --sort-by=.lastTimestamp

Teach the dependency chain:

POD
 |
 v
PVC
 |
 v
STORAGE CLASS
 |
 v
CSI / PROVISIONER
 |
 v
PV
 |
 v
ACTUAL STORAGE

Do not delete the working PVC just to manufacture a failure, because it contains the database created earlier.

Instead create a safe broken claim.

Create:

cat > k8s/04-broken-pvc.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: broken-storage
  namespace: production
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: storage-class-does-not-exist
  resources:
    requests:
      storage: 1Gi
EOF

Apply:

kubectl apply -f k8s/04-broken-pvc.yaml

Check:

kubectl get pvc -n production

Students should see:

broken-storage   Pending

Now ask:

Don't tell me the answer. Show me how you find it.

Run:

kubectl describe pvc broken-storage -n production

Then:

kubectl get sc

Students discover:

storage-class-does-not-exist

is invalid.

Delete only the training PVC:

kubectl delete pvc broken-storage -n production

Delete the lab file:

rm k8s/04-broken-pvc.yaml

Important:

WE DID NOT TOUCH:

mysql-data-mysql-0

INCIDENT 6 — CRASHLOOPBACKOFF

Now create a separate failure so the database stays safe.

Create:

cat > k8s/05-crash.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: broken-application
  namespace: production
spec:
  containers:
    - name: application
      image: busybox:1.36
      command:
        - sh
        - -c
        - |
          echo "Starting JumpToTech application..."
          sleep 3
          echo "ERROR: Database configuration invalid"
          exit 1
EOF

Apply:

kubectl apply -f k8s/05-crash.yaml

Watch:

kubectl get pod broken-application \
  -n production \
  -w

Eventually:

CrashLoopBackOff

Now ask:

What do you do?

Logs:

kubectl logs broken-application -n production

Then the very important command:

kubectl logs broken-application \
  -n production \
  --previous

Then:

kubectl describe pod broken-application \
  -n production

Explain:

APPLICATION STARTS
       |
       v
APPLICATION FAILS
       |
       v
CONTAINER EXITS
       |
       v
KUBERNETES RESTARTS
       |
       v
FAILS AGAIN
       |
       v
CrashLoopBackOff

Clean up:

kubectl delete -f k8s/05-crash.yaml
rm k8s/05-crash.yaml

PART 3 — ARGO CD

Now tell students:

Up to now we intentionally changed the cluster ourselves.

In a GitOps production environment, we want Git to define desired state.

Argo CD's automated sync can reconcile Git changes, and selfHeal: true can reconcile supported live-state drift back toward the desired state in Git. :chatgpt-content-reference{index="1"}

First make sure the healthy manifests are restored.

Check:

kubectl get pods -n production
kubectl get svc -n production
kubectl get endpoints -n production
kubectl get pvc -n production

We want healthy state.

Create Git Repository

From project root:

git init
git branch -M main

git add .
git commit -m "add production stateful application"

Create an empty GitHub repository for the lab.

Then:

git remote add origin <YOUR-REPOSITORY-URL>

git push -u origin main

Create Argo CD Application

Create:

mkdir -p argocd

Then:

cat > argocd/application.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: stateful-production
  namespace: argocd
spec:
  project: default

  source:
    repoURL: YOUR_REPOSITORY_URL
    targetRevision: main
    path: k8s

  destination:
    server: https://kubernetes.default.svc
    namespace: production

  syncPolicy:
    automated:
      prune: true
      selfHeal: true

    syncOptions:
      - CreateNamespace=true
EOF

Replace:

YOUR_REPOSITORY_URL

with your repository.

Commit:

git add .
git commit -m "add Argo CD application"
git push

Bootstrap:

kubectl apply -f argocd/application.yaml

Check:

kubectl get applications -n argocd

Argo CD's Application spec supports automated sync, pruning, and self-healing in this configuration. :chatgpt-content-reference{index="2"}

INCIDENT 7 — ARGO CD DRIFT

Now the class becomes very interesting.

Git says:

replicas: 1

Check:

grep -n "replicas" \
  k8s/03-mysql-statefulset.yaml

Now somebody manually changes production:

kubectl scale sts mysql \
  --replicas=2 \
  -n production

Immediately:

kubectl get sts mysql -n production

You may briefly see:

DESIRED: 2

But Git says:

Ask:

Who owns desired state?

GIT

Watch:

kubectl get sts mysql \
  -n production \
  -w

With self-healing enabled, Argo CD should reconcile the managed StatefulSet back toward the Git-defined state. :chatgpt-content-reference{index="3"}

Explain:

              GIT
          replicas: 1
               |
               v
            Argo CD
               |
          compare state
               |
               v
          Kubernetes
          replicas: 2
               |
               v

             DRIFT

               |
               v
         RECONCILIATION

               |
               v
          replicas: 1

INCIDENT 8 — BAD PRODUCTION DEPLOYMENT THROUGH GIT

This is the FINAL incident.

Tell students:

It is 2:00 PM.

Production is working.

Developer pushes a change.

Argo CD deploys it.

Five minutes later users report an outage.

You are on call.

Now deliberately introduce the bad image through Git.

Change:

image: mysql:8.0

to:

image: mysql:broken-production-release

Do NOT run:

kubectl apply

Instead:

git add .
git commit -m "release database v2"
git push

Now Argo CD sees Git change and syncs according to its automated-sync configuration. :chatgpt-content-reference{index="4"}

Watch:

kubectl get pods -n production -w

Production begins failing.

STUDENTS TROUBLESHOOT WITHOUT HELP

Give them 5–10 minutes.

Their process should be:

kubectl get pods -n production

Then:

kubectl describe pod mysql-0 -n production

Then:

kubectl get events \
  -n production \
  --sort-by=.lastTimestamp

Then ask:

What changed?

Check:

git log --oneline -5

Now they should connect:

HEALTHY PRODUCTION
       |
       v
NEW GIT COMMIT
       |
       v
ARGO CD SYNC
       |
       v
BAD IMAGE
       |
       v
ImagePullBackOff
       |
       v
DATABASE UNAVAILABLE
       |
       v
PRODUCTION INCIDENT

PRODUCTION RECOVERY

Don't fix production manually with:

kubectl set image...

because Git would still contain the bad desired state.

For today's simple lab, restore the known-good image in the manifest:

image: mysql:8.0

Then:

git add .
git commit -m "fix: restore known-good mysql image"
git push

Watch:

kubectl get pods -n production -w

Wait:

mysql-0   1/1   Running

Now verify DATA, not just Pod status:

kubectl exec -it mysql-0 \
  -n production \
  -- mysql -u root -pjumptotech123

Run:

USE restaurant;

SELECT * FROM customers;

We should still have:

Emma

That is your production verification.

FINAL INCIDENT REVIEW

Ask students:

Incident 1

Pod = Running
Client cannot connect

Root cause:

Service selector mismatch

How did we find it?

Service → Endpoints → Labels

Incident 2

DNS works
Connection doesn't work

Root cause:

Wrong targetPort

How did we find it?

DNS test
     ↓
Port test
     ↓
Service inspection

Incident 3

mysql-0

STATUS = Running
READY = 0/1

Root cause:

Readiness probe

Incident 4

ImagePullBackOff

Root cause:

Invalid image

Primary evidence:

kubectl describe pod
Events

Incident 5

PVC = Pending

Root cause in our exercise:

Invalid StorageClass

Investigation:

PVC
 ↓
describe PVC
 ↓
StorageClass
 ↓
Events

Incident 6

CrashLoopBackOff

Investigation:

logs
logs --previous
describe
events

Incident 7

Git = 1 replica

Cluster = 2 replicas

Problem:

Configuration Drift

Argo CD:

reconciles desired state

Incident 8

Production was working

Git change

Argo CD sync

Production down

Investigation:

What changed?
    ↓
Pods
    ↓
Describe
    ↓
Events
    ↓
Git history
    ↓
Root cause
    ↓
Restore known-good desired state
    ↓
Verify application AND data

FINAL INTERVIEW QUESTION

Your interviewer says:

“Our stateful production application suddenly became unavailable. How would you troubleshoot it?”

Answer:

“First I would clarify the scope of the incident and determine what changed around the time of failure. I would check the workload state and then use describe, logs and events to collect evidence. I would verify the application's dependencies, including Services, endpoints, DNS, ports, StatefulSet replicas, PVC/PV status and storage. If the environment uses GitOps, I would also compare the Git desired state with the live cluster and review the latest Argo CD synchronization. After identifying the root cause, I would apply or revert the appropriate change through the normal deployment process and verify both service availability and data integrity.”

Commands Students Should Know After This Lab

kubectl get pods -n production -o wide

kubectl describe pod <pod> -n production

kubectl logs <pod> -n production

kubectl logs <pod> -n production --previous

kubectl get events \
  -n production \
  --sort-by=.lastTimestamp

kubectl get sts -n production

kubectl get svc -n production

kubectl describe svc <service> -n production

kubectl get endpoints -n production

kubectl get endpointslices -n production

kubectl get pods -n production --show-labels

kubectl get pvc -n production

kubectl describe pvc <pvc> -n production

kubectl get pv

kubectl get sc

The Rule to Remember

                 PRODUCTION DOWN
                        |
                        v
                  WHAT CHANGED?
                        |
                        v
                       GET
                        |
                        v
                    DESCRIBE
                        |
                        v
                      LOGS
                        |
                        v
                     EVENTS
                        |
                        v
                  DEPENDENCIES
                        |
        +---------------+---------------+
        |               |               |
        v               v               v
     NETWORK          STORAGE          CONFIG
        |               |               |
     Service           PVC           Secret
     Endpoint           PV            ConfigMap
     DNS                SC
     Port               CSI
        |               |
        +-------+-------+
                |
                v
            ROOT CAUSE
                |
                v
               FIX
                |
                v
             VERIFY

Do not restart first. Investigate first.

How I would run the three hours

First 30–40 minutes: build the MySQL StatefulSet, create real database data, delete mysql-0, and prove persistence.

Next 60 minutes: Service selector → endpoints → DNS → wrong targetPort → readiness → ImagePullBackOff. Make the students tell you which command to run before you run it.

10-minute break.

Next 35–40 minutes: PVC Pending + CrashLoopBackOff + storage troubleshooting.

Final 30–40 minutes: turn on Argo CD ownership, demonstrate drift, then finish with the bad Git release causing a production outage. Make the students diagnose and recover it themselves.

One important production distinction to say during class: a StatefulSet gives each Pod stable identity and can associate it with stable storage, but that alone does not give MySQL database replication or high availability. This lab deliberately uses one MySQL replica so students don't leave believing that replicas: 3 automatically creates a valid MySQL HA cluster. Kubernetes documents StatefulSet's stable network/storage identity behavior explicitly. :chatgpt-content-reference{index="5"}

JumpToTech DevOps — Production Troubleshooting Lab

Aisalkyn Aidarova — Fri, 02 Oct 2026 00:51:46 +0000

StatefulSet + PostgreSQL + Helm + Argo CD + Production Troubleshooting

Duration: 3 hours

Level: Intermediate → Advanced

Role: DevOps / SRE Engineer

🎯 LAB OBJECTIVE

You joined the production support rotation.

The Kubernetes platform already exists.

The company already uses:

Kubernetes
StatefulSet
Deployment
PostgreSQL
PV/PVC
Helm
Argo CD
GitOps

You are NOT building everything from scratch.

A developer released a new application version.

After the release, production started experiencing problems.

Your responsibility is to:

OBSERVE
   ↓
IDENTIFY THE SYMPTOM
   ↓
COLLECT EVIDENCE
   ↓
ISOLATE THE LAYER
   ↓
FIND ROOT CAUSE
   ↓
FIX IN GIT / HELM
   ↓
ARGO CD SYNC
   ↓
VERIFY

⚠️ PRODUCTION RULE

Do NOT immediately run:

kubectl delete pod

Do NOT immediately restart everything.

Do NOT immediately change YAML.

First ask:

WHAT IS THE SYMPTOM?

WHAT CHANGED?

WHICH LAYER IS FAILING?

WHAT EVIDENCE DO I HAVE?

🏗️ PRODUCTION ARCHITECTURE

                        USER
                         |
                         v
                   Load Balancer
                         |
                         v
                    Web Service
                         |
                         v
                 Web Deployment
                    3 replicas
                         |
                         v
                  Backend / API
                         |
                         v
                PostgreSQL Service
                         |
                         v
               PostgreSQL StatefulSet
                         |
                         v
                     postgres-0
                         |
                         v
                        PVC
                         |
                         v
                         PV
                         |
                         v
                Persistent Storage


Git Repository
      |
      v
    Helm
      |
      v
   Argo CD
      |
      v
 Kubernetes

🧠 FIRST INTERVIEW QUESTION

Why is the frontend a Deployment but PostgreSQL a StatefulSet?

Deployment

Frontend Pods are replaceable.

web-abc123 dies

        ↓

Kubernetes creates

web-xyz456

We normally do not care about the individual Pod identity.

StatefulSet

PostgreSQL has state.

postgres-0
     |
     v
PVC
     |
     v
PV
     |
     v
DATA

The Pod may disappear.

The data must NOT disappear.

StatefulSet also provides stable Pod identity and ordered behavior.

🧠 SECOND INTERVIEW QUESTION

Does every application need a PVC?

NO.

Example:

Frontend

User:

opens page
clicks button
browses products
changes page

The frontend itself may not need persistent local storage.

If the Pod disappears:

Pod gone
   ↓
new Pod
   ↓
application continues

But persistent business data such as:

orders
customers
inventory
transactions

must be stored somewhere durable.

That could be:

PostgreSQL StatefulSet + PVC

or an external managed database such as:

Amazon RDS

🛠️ TROUBLESHOOTING TOOLBOX

Before starting the incidents, remember these commands.

Cluster

kubectl get nodes

Pods

kubectl get pods -n production

kubectl get pods -n production -o wide

Describe

kubectl describe pod <POD> -n production

Logs

kubectl logs <POD> -n production

Previous crashed container:

kubectl logs <POD> -n production --previous

Events

kubectl get events \
  -n production \
  --sort-by=.metadata.creationTimestamp

Services

kubectl get svc -n production

Endpoints

kubectl get endpoints -n production

StatefulSet

kubectl get statefulset -n production

kubectl describe statefulset postgres -n production

Storage

kubectl get pvc -n production

kubectl get pv

kubectl get sc

Deployment

kubectl get deployment -n production

kubectl rollout status deployment/web -n production

kubectl rollout history deployment/web -n production

🚨 INCIDENT 1 — Developer Released a Bad Image

At 2:00 PM, the developer releases a new version.

Five minutes later:

PRODUCTION ALERT

Application unavailable.

Run:

kubectl get pods -n production

You discover:

web-7f8d9c      0/1    ImagePullBackOff

Your job

Do NOT immediately fix it.

Investigate.

kubectl describe pod <BROKEN-POD> -n production

Look at:

Events

Also check:

kubectl get events \
  -n production \
  --sort-by=.metadata.creationTimestamp

Interview Question

What does ImagePullBackOff mean?

Think about:

wrong image
wrong tag
private registry
authentication
imagePullSecret
ECR permissions
network access

🚨 INCIDENT 2 — Pod Is Running, Application Is Down

Developer says:

My Pod is Running. Kubernetes is fine.

You run:

kubectl get pods -n production

Result:

NAME                READY     STATUS
web-xxxxx            0/1      Running

Question:

Is this Pod healthy?

NO.

Remember:

RUNNING != READY

Investigate:

kubectl describe pod <POD> -n production

Look for:

Readiness probe failed

Possible developer mistake:

readinessProbe:
  httpGet:
    path: /health-does-not-exist
    port: 80

But the real application endpoint is:

🧠 INTERVIEW QUESTION

What is the difference between readiness and liveness?

Readiness

Can this Pod receive traffic?

Failure:

Pod remains Running
BUT
Service stops sending traffic to it

Liveness

Is the application still alive?

Repeated failure can cause Kubernetes to restart the container.

🚨 INCIDENT 3 — All Pods Are Running But Website Is Down

This is one of the most important production scenarios.

You run:

kubectl get pods -n production

Everything shows:

1/1 Running

But customers say:

WEBSITE DOWN

What do you check next?

kubectl get svc -n production

Then:

kubectl get endpoints -n production

You discover:

web
ENDPOINTS: <none>

Now compare:

kubectl get svc web -n production -o yaml

with:

kubectl get pods -n production --show-labels

Developer accidentally changed:

labels:
  app: frontend

while Service still expects:

selector:
  app: web

Traffic flow is broken:

User
 |
 v
Service
 |
 X
 |
Pods

🧠 INTERVIEW QUESTION

Pods are Running but Service has no endpoints. What do you check?

Answer:

Service selector
      ↓
Pod labels

They must match.

🚨 INCIDENT 4 — Application Cannot Connect to Database

Frontend works.

Application starts.

But when users try to save something:

ERROR

Database connection failed

First:

kubectl get pods -n production

PostgreSQL:

postgres-0    1/1    Running

Do NOT conclude:

Database is fine.

Check Service:

kubectl get svc -n production

Check endpoints:

kubectl get endpoints postgres -n production

Check DNS from another Pod:

kubectl run debug \
  --rm -it \
  --image=busybox:1.36 \
  -n production \
  -- sh

Inside:

nslookup postgres

Test port:

nc -zv postgres 5432

🚨 INCIDENT 5 — Developer Changed Database Configuration

Developer changes:

DB_HOST: postgres

to:

DB_HOST: postgres-production

But Kubernetes Service is actually:

postgres

Application logs show database connection failures.

Investigate:

kubectl logs <APPLICATION-POD> -n production

Then inspect configuration:

kubectl get configmap -n production

kubectl get secret -n production

kubectl describe pod <APPLICATION-POD> -n production

🧠 INTERVIEW QUESTION

Pod can start but cannot connect to database. What do you check?

Think layer by layer:

Application logs
       ↓
Environment variables
       ↓
Secret / ConfigMap
       ↓
DNS
       ↓
Service
       ↓
Endpoints
       ↓
Port
       ↓
NetworkPolicy / Firewall
       ↓
Database authentication

🚨 INCIDENT 6 — PostgreSQL Pod Is Pending

Run:

kubectl get pods -n production

Result:

postgres-0     0/1     Pending

Do NOT delete it.

Run:

kubectl describe pod postgres-0 -n production

Then:

kubectl get pvc -n production

You discover:

postgres-data-postgres-0     Pending

Now investigate:

kubectl describe pvc postgres-data-postgres-0 -n production

kubectl get sc

Possible developer mistake:

storageClassName: gp3-wrong

but the cluster does not have:

gp3-wrong

🧠 INTERVIEW QUESTION

Pod is Pending. What can cause it?

Do NOT immediately say storage.

Possible causes include:

Insufficient CPU
Insufficient memory
Node selector
Affinity
Taints / tolerations
PVC Pending
StorageClass problem
Volume topology
Scheduling constraints

Use:

kubectl describe pod <POD>

The Events section usually provides important evidence.

🚨 INCIDENT 7 — StatefulSet Pod Was Deleted

Someone runs:

kubectl delete pod postgres-0 -n production

Students may panic.

Watch:

kubectl get pods -n production -w

StatefulSet recreates:

postgres-0

Now check:

kubectl get pvc -n production

PVC remains.

Connect:

kubectl exec -it postgres-0 \
  -n production \
  -- psql \
  -U appuser \
  -d appdb

Run:

SELECT * FROM students;

The previous data should still exist.

🧠 INTERVIEW QUESTION

Why didn't deleting the Pod delete our database?

Because:

Pod lifecycle

and:

Persistent Volume lifecycle

are different.

Conceptually:

postgres-0 deleted
       ↓
StatefulSet recreates postgres-0
       ↓
PVC still exists
       ↓
Persistent storage still exists
       ↓
Database sees existing data

🚨 INCIDENT 8 — CrashLoopBackOff

Developer releases a configuration change.

Now:

kubectl get pods -n production

shows:

CrashLoopBackOff

First commands:

kubectl describe pod <POD> -n production

kubectl logs <POD> -n production

And extremely important:

kubectl logs <POD> \
  -n production \
  --previous

Possible causes:

missing environment variable
wrong command
application exception
database unavailable
bad configuration
permissions
bad Secret

🚨 INCIDENT 9 — OOMKilled

Developer changed memory limits:

resources:
  limits:
    memory: "50Mi"

Application requires more memory.

Pod starts restarting.

Investigate:

kubectl describe pod <POD> -n production

Look for:

Reason: OOMKilled

Check:

kubectl top pod -n production

if Metrics Server is installed.

🧠 INTERVIEW QUESTION

Requests vs Limits

Requests

Used primarily by the scheduler when deciding where the Pod can run.

Limits

Maximum resource usage allowed for the container.

For memory, exceeding the limit can result in:

OOMKilled

🚨 INCIDENT 10 — HELM PRODUCTION FAILURE

Now we move from raw Kubernetes YAML to the way many companies manage applications.

Structure:

restaurant-company/
│
├── Chart.yaml
├── values.yaml
│
└── templates/
    ├── deployment.yaml
    ├── service.yaml
    ├── statefulset.yaml
    └── configmap.yaml

Developer normally does NOT rewrite every YAML file.

They may change:

image:
  repository: nginx
  tag: "1.27"

replicaCount: 3

service:
  port: 80

database:
  host: postgres
  port: 5432

persistence:
  enabled: true
  size: 2Gi

Developer accidentally commits:

service:
  port: 8080

But the container listens on:

Before deploying, inspect rendered manifests:

helm template production-app .

Validate:

helm lint .

Compare values:

helm get values production-app -n production

Check releases:

helm list -n production

History:

helm history production-app -n production

🧠 INTERVIEW QUESTION

Why use Helm if Kubernetes YAML already works?

Because Helm allows teams to reuse templates and change environment-specific configuration through values.

Example:

Same Helm Chart
      |
      +------ values-dev.yaml
      |
      +------ values-stage.yaml
      |
      +------ values-prod.yaml

🚨 INCIDENT 11 — ARGO CD SHOWS OUTOFSYNC

Now imagine someone executes:

kubectl edit deployment web -n production

and changes:

replicas: 5

But Git says:

replicaCount: 3

Now:

GIT
3 replicas

KUBERNETES
5 replicas

Argo CD detects:

OutOfSync

🧠 INTERVIEW QUESTION

What is configuration drift?

Configuration drift occurs when the live environment differs from the desired configuration stored in Git.

With GitOps:

Git = desired state

The correct workflow is generally:

Change code/config
       ↓
git add
       ↓
git commit
       ↓
git push
       ↓
Argo CD detects change
       ↓
Sync
       ↓
Kubernetes updated

🚨 INCIDENT 12 — ARGO CD SYNCED BUT APPLICATION IS BROKEN

This is a critical interview concept.

Argo CD shows:

Synced

Students say:

Everything is working!

Wrong.

Synced means the live Kubernetes configuration matches the desired Git configuration.

Git itself may contain a bad configuration.

For example:

database:
  host: wrong-postgres

Argo CD can successfully deploy it.

Therefore:

Synced ≠ Application Healthy

Investigate:

kubectl get pods -n production

kubectl logs <POD> -n production

kubectl get svc -n production

kubectl get endpoints -n production

🚨 INCIDENT 13 — MANUAL FIX KEEPS DISAPPEARING

Engineer discovers a problem and runs:

kubectl edit deployment web -n production

They fix the problem.

Five minutes later:

THE PROBLEM RETURNS

Why?

Because Git still contains the incorrect desired state.

Argo CD reconciles the cluster back to Git.

🧠 GOLDEN GITOPS RULE

In an Argo CD managed production environment:

DO NOT ONLY FIX THE CLUSTER.

FIX THE SOURCE OF TRUTH.

Usually:

values.yaml
       ↓
Git commit
       ↓
Git push
       ↓
Argo CD
       ↓
Kubernetes

Emergency procedures vary by company, but the durable fix must be reconciled with the source of truth.

🚨 FINAL INCIDENT — 3:00 AM PRODUCTION OUTAGE

This is the final interview simulation.

You receive:

SEV-1 PRODUCTION INCIDENT

Customers cannot place orders.

Website opens successfully.

Browsing works.

Checkout fails.

Deployment happened 10 minutes ago.

You are the on-call DevOps/SRE engineer.

Nobody tells you what is broken.

STEP 1 — CHECK APPLICATION

kubectl get pods -n production

Ask:

Are Pods Running?

Are they Ready?

Any restarts?

STEP 2 — CHECK EVENTS

kubectl get events \
  -n production \
  --sort-by=.metadata.creationTimestamp

STEP 3 — CHECK APPLICATION LOGS

kubectl logs <APPLICATION-POD> -n production

Previous container:

kubectl logs <APPLICATION-POD> \
  -n production \
  --previous

STEP 4 — CHECK DATABASE

kubectl get statefulset -n production

kubectl get pods -n production

kubectl get pvc -n production

STEP 5 — CHECK NETWORKING

kubectl get svc -n production

kubectl get endpoints -n production

Test DNS:

nslookup postgres

Test port:

nc -zv postgres 5432

STEP 6 — CHECK CONFIGURATION

kubectl get configmap -n production

kubectl describe pod <APPLICATION-POD> -n production

Ask:

Did DB_HOST change?

Did DB_PORT change?

Did Secret reference change?

Did Service name change?

STEP 7 — CHECK HELM

helm list -n production

helm get values production-app -n production

helm history production-app -n production

Ask:

What changed between releases?

STEP 8 — CHECK ARGO CD

Ask:

Synced?

OutOfSync?

Healthy?

Degraded?

What commit was deployed?

Then inspect Git history:

git log --oneline -10

git show HEAD

or compare commits:

git diff HEAD~1 HEAD

🎯 STUDENT FINAL REPORT

Every student must report the incident using this structure:

1. SYMPTOM

2. IMPACT

3. WHAT CHANGED?

4. EVIDENCE

5. FAILED LAYER

6. ROOT CAUSE

7. FIX

8. VERIFICATION

9. PREVENTION

Example:

SYMPTOM:
Checkout failed after deployment.

IMPACT:
Customers could browse products but could not place orders.

EVIDENCE:
Application logs showed database hostname resolution failure.

ROOT CAUSE:
Developer changed database.host in values.yaml
from postgres to postgres-production.

FIX:
Corrected values.yaml and committed the change to Git.

DEPLOYMENT:
Argo CD synchronized the corrected desired state.

VERIFICATION:
Application became Ready.
Database connection succeeded.
Checkout test succeeded.

PREVENTION:
Add Helm validation and automated integration tests
before production deployment.

🎤 RAPID-FIRE INTERVIEW ROUND

Teacher asks students without giving them commands.

Question 1

Pod is Pending.

What do you run?

kubectl describe pod <pod>

Question 2

Pod is CrashLoopBackOff.

What do you check?

kubectl logs <pod>
kubectl logs <pod> --previous
kubectl describe pod <pod>

Question 3

Pod is Running but 0/1.

Think:

Readiness

Question 4

All Pods are Running but application is inaccessible.

Think:

Service
Endpoints
Selectors
Labels
Ingress / Load Balancer
Network

Question 5

Service has no endpoints.

Think:

selector != Pod labels

Question 6

StatefulSet Pod is Pending.

Think:

Scheduling
PVC
PV
StorageClass
Events

Question 7

Database Pod was deleted. Will data necessarily disappear?

No.

Check persistent storage and PVC/PV lifecycle.

Question 8

Argo CD says Synced. Does that prove the application works?

No.

Synced != Healthy application

Question 9

You manually fixed production, but Argo CD changed it back.

Why?

Git still contains the desired state Argo CD is reconciling.

Question 10

Where should the permanent fix go?

SOURCE OF TRUTH

Git / Helm values

🏆 FINAL PRODUCTION TROUBLESHOOTING MODEL

Memorize this:

USER REPORTS PROBLEM
        ↓
WHAT CHANGED?
        ↓
kubectl get
        ↓
kubectl describe
        ↓
kubectl logs
        ↓
events
        ↓
Service / Endpoints / DNS
        ↓
ConfigMap / Secret
        ↓
StatefulSet / PVC / PV
        ↓
Helm values
        ↓
Git diff
        ↓
Argo CD
        ↓
ROOT CAUSE
        ↓
FIX SOURCE OF TRUTH
        ↓
SYNC / DEPLOY
        ↓
VERIFY APPLICATION
        ↓
MONITOR
        ↓
POSTMORTEM / PREVENTION

🎓 WHAT YOU SHOULD BE ABLE TO EXPLAIN AFTER THIS LAB

By the end of this lab you should be comfortable explaining:

Deployment vs StatefulSet
Stateless vs stateful applications
StatefulSet Pod identity
PV vs PVC
StorageClass
Data persistence after Pod deletion
Headless Service
Service selectors
Pod labels
Kubernetes endpoints
Kubernetes DNS
Readiness vs liveness
Pending Pods
CrashLoopBackOff
ImagePullBackOff
OOMKilled
ConfigMap vs Secret
Database connectivity troubleshooting
Helm charts
values.yaml
helm lint
helm template
Helm release history
Argo CD
GitOps
Desired state
Configuration drift
Synced vs application health
Why manual production changes can be reverted
How to investigate a production incident
Root cause analysis
How to verify recovery

⭐ MOST IMPORTANT LESSON

A DevOps Engineer is not valuable because they can memorize:

kubectl apply -f deployment.yaml

The important skill is being able to answer:

The application worked yesterday.

A deployment happened.

Now production is broken.

WHAT CHANGED?

WHERE IS THE FAILURE?

WHAT EVIDENCE PROVES IT?

HOW DO I FIX IT SAFELY?

HOW DO I KNOW IT IS REALLY FIXED?

That is production troubleshooting.

This complements the lab you already posted particularly well because your existing article already establishes the healthy PostgreSQL StatefulSet, persistent-storage test, and initial incidents. :chatgpt-content-reference{index="1"}

One teaching change I'd make: you should introduce the mistakes secretly. Don't show students “change this line to the wrong value.” Give them only the incident ticket. That turns it from a Kubernetes tutorial into a realistic DevOps/SRE troubleshooting interview simulation.

statefulset prod lab

Aisalkyn Aidarova — Thu, 01 Oct 2026 01:15:41 +0000

JumpToTech DevOps Lab

Production-Style StatefulSet + Persistent Storage + Argo CD

Estimated time: 2–3 hours

Level: Intermediate

Technologies: Kubernetes, StatefulSet, PV, PVC, StorageClass, Nginx, Git, GitHub, Argo CD

1. Lab Objective

In this lab you will deploy a stateful workload using Kubernetes and manage it through Argo CD.

By the end of the lab, you should understand this architecture:

GitHub Repository
        |
        v
     Argo CD
        |
        v
   Kubernetes
        |
        v
   StatefulSet
        |
   +----+----+
   |    |    |
 web-0 web-1 web-2
   |    |    |
  PVC  PVC  PVC
   |    |    |
   PV   PV   PV
   |    |    |
   +----+----+
        |
        v
Persistent Storage

You will prove that:

Pod != Persistent Data

A Pod can be destroyed and recreated while its persistent data survives.

2. Important Concepts

Before starting, understand these components.

StatefulSet

A StatefulSet manages Pods that require stable identities.

Instead of names such as:

web-5d8478b7c4-xk29p
web-5d8478b7c4-8h2mm

a StatefulSet creates:

web-0
web-1
web-2

These identities are stable.

If:

web-1

is deleted, StatefulSet recreates:

web-1

rather than creating a randomly named replacement.

3. PV vs PVC vs Storage

Understand this chain:

Pod
 |
 v
PVC
 |
 v
PV
 |
 v
Actual Storage

PVC — PersistentVolumeClaim

PVC is a request for storage.

Example:

I need 2Gi of persistent storage.

PV — PersistentVolume

PV is the Kubernetes resource representing persistent storage.

Actual Storage

The real storage backend could be:

AWS EBS
AWS EFS
Azure Disk
Google Persistent Disk
NFS
Ceph
etc.

For example:

Pod
 ↓
PVC
 ↓
PV
 ↓
AWS EBS

4. StorageClass

StorageClass defines how a class of storage can be dynamically provisioned.

Check available StorageClasses:

kubectl get storageclass

Short version:

kubectl get sc

Look for the default StorageClass:

kubectl get sc

Example:

NAME                 PROVISIONER
standard (default)   ...

The exact output depends on your cluster.

Conceptually:

PVC
 |
 | requests storage
 v
StorageClass
 |
 v
CSI Provisioner
 |
 v
Actual Volume
 |
 v
PV
 |
 v
PVC becomes Bound

This is called:

Dynamic Provisioning

5. Check the Cluster

Run:

kubectl cluster-info

Then:

kubectl get nodes

Expected:

NAME       STATUS   ROLES
node-1     Ready    ...

Do not continue if your nodes are not Ready.

6. Create the Project

Create a directory:

mkdir statefulset-production-lab
cd statefulset-production-lab

Create directories:

mkdir -p k8s

Check:

tree

Expected:

statefulset-production-lab/
└── k8s/

If tree is unavailable:

find .

7. Create Namespace

Create:

vim k8s/namespace.yaml

Add:

apiVersion: v1
kind: Namespace
metadata:
  name: stateful-app
  labels:
    app.kubernetes.io/part-of: stateful-app

Apply:

kubectl apply -f k8s/namespace.yaml

Verify:

kubectl get ns

8. Create ConfigMap

Production applications should separate configuration from the container image.

Create:

vim k8s/configmap.yaml

Add:

apiVersion: v1
kind: ConfigMap
metadata:
  name: web-config
  namespace: stateful-app
data:
  APP_ENV: "production"
  LOG_LEVEL: "info"

Apply:

kubectl apply -f k8s/configmap.yaml

Check:

kubectl get configmap -n stateful-app

9. Create Secret

Create:

vim k8s/secret.yaml

For this training lab:

apiVersion: v1
kind: Secret
metadata:
  name: web-secret
  namespace: stateful-app
type: Opaque
stringData:
  APP_PASSWORD: "jumptotech-lab-password"

Apply:

kubectl apply -f k8s/secret.yaml

Check:

kubectl get secret -n stateful-app

Production note: committing plaintext secrets to Git is not a production secret-management strategy. Real GitOps environments commonly integrate an external secrets solution, cloud secret manager, or encrypted/sealed secret workflow. This file is included only to demonstrate Kubernetes Secret consumption.

10. Create Headless Service

StatefulSets commonly use a Headless Service to provide stable network identities.

Create:

vim k8s/headless-service.yaml

Add:

apiVersion: v1
kind: Service
metadata:
  name: web-headless
  namespace: stateful-app
spec:
  clusterIP: None

  selector:
    app: web

  ports:
    - name: http
      port: 80
      targetPort: 80

Notice:

clusterIP: None

This makes it a:

Headless Service

Architecture:

Headless Service
       |
       +---- web-0
       |
       +---- web-1
       |
       +---- web-2

11. Create Production-Style StatefulSet

Create:

vim k8s/statefulset.yaml

Add:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
  namespace: stateful-app

spec:
  serviceName: web-headless

  replicas: 3

  podManagementPolicy: OrderedReady

  updateStrategy:
    type: RollingUpdate

  selector:
    matchLabels:
      app: web

  template:
    metadata:
      labels:
        app: web

    spec:
      terminationGracePeriodSeconds: 30

      containers:
        - name: nginx
          image: nginx:1.27

          ports:
            - name: http
              containerPort: 80

          envFrom:
            - configMapRef:
                name: web-config

            - secretRef:
                name: web-secret

          resources:
            requests:
              cpu: "100m"
              memory: "128Mi"

            limits:
              cpu: "500m"
              memory: "512Mi"

          readinessProbe:
            httpGet:
              path: /
              port: 80

            initialDelaySeconds: 5
            periodSeconds: 10

          livenessProbe:
            httpGet:
              path: /
              port: 80

            initialDelaySeconds: 15
            periodSeconds: 20

          volumeMounts:
            - name: web-data
              mountPath: /usr/share/nginx/html

  volumeClaimTemplates:

    - metadata:
        name: web-data

      spec:
        accessModes:
          - ReadWriteOnce

        resources:
          requests:
            storage: 1Gi

12. Understand `volumeClaimTemplates`

This is one of the most important parts:

volumeClaimTemplates:

  - metadata:
      name: web-data

StatefulSet uses this template to create a PVC for each replica.

With:

replicas: 3

you should receive approximately:

web-data-web-0
web-data-web-1
web-data-web-2

Architecture:

web-0
 |
 +---- web-data-web-0
          |
          v
          PV

web-1
 |
 +---- web-data-web-1
          |
          v
          PV

web-2
 |
 +---- web-data-web-2
          |
          v
          PV

Each replica therefore maintains its own storage claim.

13. Create PodDisruptionBudget

Create:

vim k8s/pdb.yaml

Add:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
  namespace: stateful-app

spec:
  minAvailable: 2

  selector:
    matchLabels:
      app: web

The PDB protects application availability during voluntary disruptions, such as certain node-maintenance operations.

It does not prevent every possible Pod failure or deletion.

14. Check All Files

Run:

tree

Expected:

.
└── k8s
    ├── configmap.yaml
    ├── headless-service.yaml
    ├── namespace.yaml
    ├── pdb.yaml
    ├── secret.yaml
    └── statefulset.yaml

15. Validate Before Deployment

Use client-side parsing/validation first:

kubectl apply --dry-run=client -f k8s/

Then inspect:

kubectl diff -f k8s/

kubectl diff may return a non-zero exit code when differences exist; that does not necessarily mean the manifests are invalid.

16. Deploy

Apply:

kubectl apply -f k8s/

Watch:

kubectl get pods -n stateful-app -w

Expected:

web-0
web-1
web-2

Because OrderedReady is configured, StatefulSet normally progresses through ordinal replicas in order as they become Ready.

Press:

CTRL+C

when complete.

17. Inspect StatefulSet

Run:

kubectl get statefulset -n stateful-app

Short version:

kubectl get sts -n stateful-app

Then:

kubectl describe sts web -n stateful-app

Students should identify:

Replicas
Pod Management Policy
Update Strategy
Volume Claims
Selector
Containers

18. Inspect Pods

Run:

kubectl get pods -n stateful-app -o wide

Expected identities:

web-0
web-1
web-2

Compare this with a Deployment:

Deployment:

web-7d8bc9c87c-d7sk2
web-7d8bc9c87c-j82lm


StatefulSet:

web-0
web-1
web-2

19. Inspect PVCs

Run:

kubectl get pvc -n stateful-app

Expected:

NAME             STATUS
web-data-web-0   Bound
web-data-web-1   Bound
web-data-web-2   Bound

The critical status is:

Bound

Now inspect one:

kubectl describe pvc web-data-web-0 -n stateful-app

Look for:

Status
Volume
Capacity
Access Modes
StorageClass

20. Inspect PVs

Run:

kubectl get pv

Notice that PV is cluster-scoped, while PVC is namespaced.

Find the PV bound to web-data-web-0:

kubectl get pvc web-data-web-0 \
  -n stateful-app \
  -o jsonpath='{.spec.volumeName}{"\n"}'

Save it:

PV_NAME=$(kubectl get pvc web-data-web-0 \
  -n stateful-app \
  -o jsonpath='{.spec.volumeName}')

Inspect:

kubectl describe pv "$PV_NAME"

Architecture:

web-0
 |
 v
web-data-web-0
 |
 v
PV
 |
 v
Storage Backend

21. Write Persistent Data

Enter:

kubectl exec -it web-0 \
  -n stateful-app \
  -- sh

Inside:

echo "<h1>JumpToTech Stateful Application</h1>" \
> /usr/share/nginx/html/index.html

Add another file:

echo "This data must survive Pod deletion." \
> /usr/share/nginx/html/data.txt

Check:

cat /usr/share/nginx/html/index.html

Check:

cat /usr/share/nginx/html/data.txt

Exit:

exit

22. Verify Application

Run:

kubectl port-forward \
  pod/web-0 \
  8080:80 \
  -n stateful-app

Open another terminal:

curl localhost:8080

Expected:

<h1>JumpToTech Stateful Application</h1>

Stop port-forward with:

CTRL+C

23. Failure Test

Now deliberately destroy the Pod.

Before:

kubectl get pods -n stateful-app

Delete:

kubectl delete pod web-0 -n stateful-app

Watch:

kubectl get pods -n stateful-app -w

StatefulSet should restore:

web-0

Wait until:

web-0   1/1   Running

24. Verify Data Survived

Run:

kubectl exec web-0 \
  -n stateful-app \
  -- cat /usr/share/nginx/html/data.txt

Expected:

This data must survive Pod deletion.

This proves:

Pod lifecycle
      !=
Persistent storage lifecycle

The Pod was destroyed.

The persistent claim/storage remained available.

The recreated web-0 mounted its storage again.

25. Compare Data Between Replicas

Check web-1:

kubectl exec web-1 \
  -n stateful-app \
  -- ls -la /usr/share/nginx/html

Do not assume it contains the file written to web-0.

Why?

Because:

web-0 → PVC-0 → Storage-0

web-1 → PVC-1 → Storage-1

web-2 → PVC-2 → Storage-2

The replicas have independent persistent claims.

26. Scale Up

Scale:

kubectl scale statefulset web \
  --replicas=5 \
  -n stateful-app

Watch:

kubectl get pods -n stateful-app -w

Expected:

web-0
web-1
web-2
web-3
web-4

Check claims:

kubectl get pvc -n stateful-app

You should now see additional claims for the new replicas.

27. Scale Down

Scale back:

kubectl scale statefulset web \
  --replicas=3 \
  -n stateful-app

Check:

kubectl get pods -n stateful-app

Then:

kubectl get pvc -n stateful-app

Notice the important behavior:

Do not assume scaling down deletes the corresponding PVCs.

StatefulSet storage retention is intentionally conservative because automatically deleting persistent data could be dangerous. StatefulSet also supports PVC retention-policy configuration when a workload explicitly requires different behavior.

28. Examine Stable Network Identity

Run:

kubectl get svc -n stateful-app

You should see:

web-headless

with:

CLUSTER-IP: None

Check DNS from inside the cluster:

kubectl run dns-test \
  --image=busybox:1.36 \
  --restart=Never \
  -n stateful-app \
  -- sleep 3600

Wait:

kubectl wait \
  --for=condition=Ready \
  pod/dns-test \
  -n stateful-app \
  --timeout=60s

Resolve:

kubectl exec dns-test \
  -n stateful-app \
  -- nslookup web-0.web-headless.stateful-app.svc.cluster.local

Try:

kubectl exec dns-test \
  -n stateful-app \
  -- nslookup web-1.web-headless.stateful-app.svc.cluster.local

This demonstrates stable network identity.

Clean up the test Pod:

kubectl delete pod dns-test -n stateful-app

29. Check ConfigMap Environment Variables

Run:

kubectl exec web-0 \
  -n stateful-app \
  -- printenv APP_ENV

Expected:

production

Check:

kubectl exec web-0 \
  -n stateful-app \
  -- printenv LOG_LEVEL

Expected:

info

30. Check Resource Requests and Limits

Run:

kubectl describe pod web-0 -n stateful-app

Find:

Requests:
  cpu
  memory

Limits:
  cpu
  memory

These help Kubernetes make scheduling decisions and enforce container resource limits.

31. Check Probes

Run:

kubectl describe pod web-0 -n stateful-app

Find:

Liveness
Readiness

Readiness Probe

Answers:

Should this Pod currently receive traffic?

Liveness Probe

Answers:

Does Kubernetes consider this container healthy enough to keep running,
or should it restart it?

32. Check PDB

Run:

kubectl get pdb -n stateful-app

Then:

kubectl describe pdb web-pdb -n stateful-app

Remember:

PDB != replica controller
PDB != protection against every failure
PDB != backup

It primarily constrains voluntary disruptions.

PART II — GITOPS WITH ARGO CD

33. Why Argo CD?

So far deployment was performed manually:

Engineer
   |
kubectl apply
   |
   v
Kubernetes

Production GitOps introduces a different operating model:

Engineer
   |
   v
Git
   |
   v
GitHub
   |
   v
Argo CD
   |
   v
Kubernetes

Git becomes the desired-state source.

34. Important GitOps Rule

Once Argo CD owns this application, avoid treating manual kubectl apply as the normal deployment process.

Desired workflow:

Change YAML
   ↓
git add
   ↓
git commit
   ↓
git push
   ↓
Argo CD detects desired-state change
   ↓
Argo CD syncs
   ↓
Kubernetes changes

35. Create `.gitignore`

Create:

vim .gitignore

Add:

.DS_Store
*.log
.env

Do not put credentials, kubeconfigs, tokens, or private keys in the repository.

36. Git Repository

Initialize:

git init

Create branch:

git checkout -b main

Check:

git status

Add files:

git add .

Commit:

git commit -m "feat: add production-style stateful workload"

37. Push to GitHub

Create an empty repository in your GitHub account.

Example repository name:

statefulset-production-lab

Add your actual remote:

git remote add origin <YOUR-GITHUB-REPOSITORY-URL>

Verify:

git remote -v

Push:

git push -u origin main

The repository should contain:

statefulset-production-lab/
│
├── .gitignore
│
└── k8s/
    ├── namespace.yaml
    ├── configmap.yaml
    ├── secret.yaml
    ├── headless-service.yaml
    ├── statefulset.yaml
    └── pdb.yaml

Again: the demo secret.yaml is for training. Do not use plaintext Git secrets for real production credentials.

38. Install Argo CD

Check first:

kubectl get ns argocd

If Argo CD is already installed, do not reinstall it.

If it is not installed, install Argo CD using the installation method provided for your training cluster.

After installation verify:

kubectl get pods -n argocd

Wait until the required Argo CD components are running.

39. Access Argo CD

For a local/training environment:

kubectl port-forward \
  svc/argocd-server \
  -n argocd \
  8081:443

Then access:

https://localhost:8081

Retrieve the initial admin password when using the standard initial-secret setup:

kubectl -n argocd \
  get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" \
  | base64 -d

Then:

echo

Username:

admin

40. Create Argo CD Application Manifest

Create a new directory:

mkdir argocd

Create:

vim argocd/application.yaml

Add:

apiVersion: argoproj.io/v1alpha1
kind: Application

metadata:
  name: stateful-app
  namespace: argocd

spec:
  project: default

  source:
    repoURL: YOUR_GITHUB_REPOSITORY_URL
    targetRevision: main
    path: k8s

  destination:
    server: https://kubernetes.default.svc
    namespace: stateful-app

  syncPolicy:

    automated:
      prune: true
      selfHeal: true

    syncOptions:
      - CreateNamespace=true

Replace:

YOUR_GITHUB_REPOSITORY_URL

with your repository URL.

41. Important: Avoid the Argo CD Bootstrap Loop

The Argo CD Application manifest itself is stored under:

argocd/application.yaml

but the Application watches only:

path: k8s

Therefore Argo CD manages the Kubernetes workload manifests under k8s/, while the Application definition can be bootstrapped separately.

Commit it:

git add argocd/application.yaml
git commit -m "feat: add Argo CD application"
git push

Bootstrap the Argo CD Application:

kubectl apply -f argocd/application.yaml

42. Check Argo CD

Run:

kubectl get applications -n argocd

Expected eventually:

NAME           SYNC STATUS   HEALTH STATUS
stateful-app   Synced        Healthy

You can also inspect:

kubectl describe application stateful-app -n argocd

43. Understand Automated Sync

We configured:

automated:
  prune: true
  selfHeal: true

Automated sync

Argo CD can automatically synchronize Git desired state into Kubernetes.

`prune: true`

If a resource managed by the Application is removed from Git, Argo CD can remove that managed resource from the cluster during sync.

`selfHeal: true`

If a managed Kubernetes resource drifts from Git, Argo CD can restore the Git-defined desired state.

This creates:

              GIT
               |
               |
         desired state
               |
               v
            Argo CD
               |
        compare/reconcile
               |
               v
           Kubernetes

44. GitOps Scaling Test

Do not scale manually for this test.

Edit:

vim k8s/statefulset.yaml

Change:

replicas: 3

to:

replicas: 4

Commit:

git add k8s/statefulset.yaml
git commit -m "scale: increase StatefulSet to four replicas"
git push

Watch:

kubectl get pods -n stateful-app -w

Eventually:

web-0
web-1
web-2
web-3

Check PVC:

kubectl get pvc -n stateful-app

The new replica should receive its corresponding claim.

45. GitOps Self-Healing Test

Git says:

replicas: 4

Now deliberately introduce drift:

kubectl scale statefulset web \
  --replicas=2 \
  -n stateful-app

Check:

kubectl get sts web -n stateful-app

Argo CD should detect that live state differs from Git.

With automated self-healing enabled, it should reconcile toward the desired state stored in Git.

Check:

kubectl get pods -n stateful-app -w

The desired replica count should return to the Git-defined value.

This demonstrates:

Git = Desired State

Cluster = Live State

Argo CD = Reconciliation

46. GitOps Update Test

Change image:

image: nginx:1.27

to a newer approved image tag available for your environment.

Commit:

git add k8s/statefulset.yaml
git commit -m "chore: update nginx image"
git push

Watch:

kubectl rollout status \
  statefulset/web \
  -n stateful-app

Then:

kubectl get pods -n stateful-app

Inspect:

kubectl describe sts web -n stateful-app

47. Check Persistent Data After Update

Check:

kubectl exec web-0 \
  -n stateful-app \
  -- cat /usr/share/nginx/html/data.txt

The original persistent data should still exist if the same PVC/storage remains attached.

This demonstrates the separation between:

Application lifecycle
Container lifecycle
Pod lifecycle
Storage lifecycle

48. Production Architecture

The completed system conceptually looks like:

                         DEVELOPER
                             |
                             |
                         git push
                             |
                             v
                         GITHUB
                             |
                             |
                             v
                         ARGO CD
                             |
                    desired-state sync
                             |
                             v
                     KUBERNETES CLUSTER
                             |
                             v
                        STATEFULSET
                             |
                +------------+------------+
                |            |            |
                v            v            v
              web-0        web-1        web-2
                |            |            |
                v            v            v
              PVC-0        PVC-1        PVC-2
                |            |            |
                v            v            v
               PV-0         PV-1         PV-2
                |            |            |
                +------------+------------+
                             |
                             v
                     STORAGE BACKEND

In an AWS EKS environment using EBS, the lower portion may conceptually be:

StatefulSet
    |
    v
   Pod
    |
    v
   PVC
    |
    v
StorageClass
    |
    v
EBS CSI Driver
    |
    v
AWS EBS Volume

49. Troubleshooting

Pod Pending

Run:

kubectl describe pod web-0 -n stateful-app

Check events:

kubectl get events \
  -n stateful-app \
  --sort-by=.lastTimestamp

PVC Pending

Run:

kubectl get pvc -n stateful-app

Then:

kubectl describe pvc web-data-web-0 -n stateful-app

Check:

kubectl get sc

Potential causes include:

No suitable/default StorageClass
CSI driver unavailable
Provisioning failure
Topology/zone constraints
Storage quota
Access-mode mismatch

StatefulSet Not Ready

Run:

kubectl describe sts web -n stateful-app

Then:

kubectl describe pod web-0 -n stateful-app

Logs:

kubectl logs web-0 -n stateful-app

Argo CD OutOfSync

Check:

kubectl describe application stateful-app -n argocd

Confirm:

repository URL
branch
path
permissions
manifest validity
destination

50. Useful Commands Cheat Sheet

kubectl get sts -A

kubectl get pods -A

kubectl get pvc -A

kubectl get pv

kubectl get sc

kubectl get svc -A

kubectl get pdb -A

kubectl describe sts web -n stateful-app

kubectl describe pod web-0 -n stateful-app

kubectl describe pvc web-data-web-0 -n stateful-app

kubectl logs web-0 -n stateful-app

kubectl get events -n stateful-app --sort-by=.lastTimestamp

kubectl get applications -n argocd

51. Interview Questions

Students must be able to answer these after completing the lab.

What is a StatefulSet?

A StatefulSet is a Kubernetes workload controller designed for applications whose replicas require stable identities, stable network identities, ordered lifecycle semantics, or stable per-replica storage.

Deployment vs StatefulSet?

Deployment replicas are generally interchangeable.

StatefulSet replicas have stable ordinal identities such as:

web-0
web-1
web-2

and commonly use stable per-replica storage.

What is PVC?

A PersistentVolumeClaim is a request for persistent storage.

What is PV?

A PersistentVolume is a Kubernetes resource representing persistent storage available to workloads.

What is the difference between PV and actual storage?

PV is a Kubernetes resource.

The actual storage backend may be something such as:

AWS EBS
EFS
NFS
Azure Disk
Google Persistent Disk

What is StorageClass?

StorageClass describes a class of storage and how storage may be dynamically provisioned.

What is dynamic provisioning?

Dynamic provisioning allows storage to be provisioned automatically when a PVC requests it rather than requiring an administrator to manually pre-create every PV.

What does `volumeClaimTemplates` do?

It allows a StatefulSet to create a persistent storage claim for each replica.

Why does StatefulSet use stable identity?

Because individual replicas may need a consistent identity, network name, and association with their own persistent storage.

Does deleting a StatefulSet Pod delete its data?

Not necessarily. Pod lifecycle and persistent storage lifecycle are separate. PVC/PV retention and reclaim behavior determine what happens to persistent storage.

Does StatefulSet automatically configure database replication?

No.

Running:

mysql-0
mysql-1
mysql-2

does not automatically create a properly replicated MySQL cluster.

Database replication, leader election, failover, consistency, backups, and recovery require database-specific configuration or an appropriate Kubernetes Operator/application architecture.

Can Deployment use PVC?

Yes.

Persistent storage is not exclusive to StatefulSet.

StatefulSet is selected when stable per-replica identity and lifecycle/storage relationships are important.

Why use a Headless Service?

It supports direct service discovery and stable DNS identities for StatefulSet replicas rather than providing only one load-balanced Service virtual IP.

What does Argo CD do?

Argo CD continuously compares Kubernetes live state with the desired state stored in Git and can synchronize the cluster to that desired state.

52. Student Assignment

Complete all tasks without copying the finished StatefulSet from another student.

Your final environment must contain:

Namespace
ConfigMap
Secret
Headless Service
StatefulSet
3+ replicas
PVC per replica
PV-backed persistent storage
Resource requests
Resource limits
Readiness probe
Liveness probe
PodDisruptionBudget
Git repository
Argo CD Application
Automated synchronization
Self-healing
Persistent-data test

Perform and document these tests:

1. Delete web-0.
2. Prove web-0 is recreated.
3. Prove its persistent data survives.
4. Show all PVCs.
5. Identify the PV bound to web-0.
6. Scale through Git.
7. Show Argo CD synchronizing the change.
8. Introduce replica-count drift manually.
9. Show Argo CD self-healing.
10. Explain Pod → PVC → PV → actual storage.

53. Required Submission

Submit:

GitHub repository link

and screenshots/output showing:

kubectl get sts -n stateful-app

kubectl get pods -n stateful-app

kubectl get pvc -n stateful-app

kubectl get pv

kubectl get sc

kubectl get svc -n stateful-app

kubectl get pdb -n stateful-app

kubectl get applications -n argocd

Also submit:

kubectl exec web-0 \
  -n stateful-app \
  -- cat /usr/share/nginx/html/data.txt

after deleting and recreating web-0.

Final Knowledge Map

Every student should be able to reproduce this from memory:

                    GIT
                     |
                     v
                  ARGO CD
                     |
                     v
                 KUBERNETES
                     |
                     v
                STATEFULSET
                     |
          +----------+----------+
          |          |          |
        web-0      web-1      web-2
          |          |          |
          v          v          v
        PVC-0      PVC-1      PVC-2
          |          |          |
          v          v          v
         PV-0       PV-1       PV-2
          |          |          |
          +----------+----------+
                     |
                     v
              ACTUAL STORAGE

Memorize:

StatefulSet  = stable workload identity

PVC          = request for storage

PV           = Kubernetes persistent-storage resource

StorageClass = how/class through which storage is provisioned

CSI Driver   = Kubernetes-to-storage-system integration

EBS/EFS/etc. = actual storage backend

Argo CD      = GitOps reconciliation

Git          = desired state

Production principle:

Pods are replaceable.

Persistent data must not depend on
the lifetime of a particular container.

Git defines desired Kubernetes state.

Argo CD continuously reconciles
the cluster toward that state.

DevOps / SRE Interview Master Guide

Aisalkyn Aidarova — Wed, 30 Sep 2026 20:35:30 +0000

The memory method

For most technical questions, remember D-E-R:

D — Definition: What is it?

E — Example: Where did you use it?

R — Real production detail: How did you troubleshoot/manage it?

For troubleshooting questions, remember C-L-I-F:

C — Check symptoms/events

L — Logs and metrics

I — Identify root cause

F — Fix + verify

For behavioral questions, remember STAR:

Situation → Task → Action → Result

PART 1 — RECRUITER / HR SCREEN

1. Tell me about yourself.

Easy memory: Present → Tools → Responsibilities → Goal.

Answer:

“I’m a DevOps Engineer with several years of experience working with AWS and cloud-native infrastructure.

My main technologies are AWS, Kubernetes/EKS, Docker, Terraform, GitHub Actions, Jenkins, Argo CD, Helm, Linux, and monitoring tools such as Prometheus, Grafana, Datadog, and CloudWatch.

In my current environment, I automate infrastructure with Terraform, build CI/CD pipelines, manage Kubernetes deployments, implement GitOps with Argo CD, and troubleshoot production issues.

I work closely with developers and other engineering teams to make deployments reliable, repeatable, secure, and automated.

I'm looking for a DevOps/SRE opportunity where I can continue working with cloud infrastructure, Kubernetes, automation, and reliability.”

2. What do you do in your current role?

Memory: Infrastructure → CI/CD → Kubernetes → Monitoring → Support.

Answer:

“My responsibilities can be divided into five areas.

First, infrastructure: I provision AWS resources using Terraform.

Second, CI/CD: I maintain pipelines using GitHub Actions and Jenkins.

Third, Kubernetes: I deploy and support applications running on EKS.

Fourth, monitoring: I use tools such as CloudWatch, Prometheus, Grafana, and Datadog.

Finally, production support: I troubleshoot failed deployments, application issues, infrastructure problems, and alerts.”

3. Why are you looking for another opportunity?

“I've learned a lot in my current environment, but I'm looking for an opportunity where I can continue growing in cloud infrastructure, Kubernetes, automation, and SRE practices while taking on larger technical responsibilities.”

Avoid complaining about managers or coworkers.

4. Why DevOps?

“I enjoy automation and solving infrastructure problems. DevOps allows me to work across development, infrastructure, cloud, security, CI/CD, and operations. I especially enjoy taking a manual process and turning it into a reliable automated workflow.”

5. What are your strongest technologies?

“My strongest areas are AWS, Kubernetes/EKS, Terraform, Docker, CI/CD, Linux, and GitOps.

I've used GitHub Actions and Jenkins for CI/CD, Terraform for infrastructure automation, Argo CD and Helm for Kubernetes deployments, and CloudWatch, Prometheus, Grafana, and Datadog for observability.”

6. What is your biggest strength?

“One of my strongest skills is troubleshooting. When an incident happens, I try not to randomly change things. I first understand the symptoms, check logs, events and metrics, isolate the affected layer, identify the root cause, implement the fix, and verify the system is stable.”

7. What is your weakness?

“I used to spend too much time trying to solve every problem independently. I've learned that in production environments, escalation and collaboration are important. I now investigate first, document what I've found, and involve the appropriate team when necessary.”

8. Are you comfortable with on-call?

“Yes. I understand that SRE and DevOps positions can include production support and on-call responsibilities. During an incident my priority is restoring service safely, communicating clearly, and then completing root-cause analysis afterward.”

9. What salary are you looking for?

“I’m open to discussing compensation based on the responsibilities, level, overall package, and expectations for the position. Could you share the budgeted range for the role?”

10. Why should we hire you?

“My experience aligns well with cloud-native DevOps environments. I have hands-on experience with AWS, Terraform, Kubernetes, Docker, CI/CD, GitOps, monitoring, and production troubleshooting.

I can work across the entire delivery lifecycle—from infrastructure provisioning to application deployment and production support.”

PART 2 — DEVOPS FUNDAMENTALS

11. What is DevOps?

Remember: People + Process + Automation.

“DevOps is a combination of culture, practices, and automation that improves collaboration between development and operations teams.

The goal is to deliver software faster and more reliably through automation, CI/CD, Infrastructure as Code, monitoring, and shared responsibility.”

12. What does a DevOps engineer do?

“A DevOps engineer automates infrastructure and software delivery.

Typical responsibilities include cloud infrastructure, CI/CD pipelines, containers, Kubernetes, Infrastructure as Code, monitoring, security automation, and production troubleshooting.”

13. What is CI/CD?

CI = test. CD = deliver/deploy.

“Continuous Integration means developers frequently integrate code and automated pipelines validate it through builds, linting, tests, quality checks, and security scans.

Continuous Delivery or Deployment handles getting validated software into environments such as staging and production.”

14. Describe your complete CI/CD process.

MEMORIZE THIS FLOW:

Developer → Git → CI → Test → Scan → Build → ECR → GitOps → Argo CD → EKS → Monitor

Answer:

“A developer pushes code or opens a pull request.

That triggers GitHub Actions.

The pipeline checks out the repository, installs dependencies, runs linting and tests, performs SonarQube analysis and Trivy security scanning.

After the checks pass, we build a Docker image.

We tag the image using an immutable identifier such as the commit SHA and push it to Amazon ECR.

For deployment we use GitOps. Deployment configuration is updated in Git, and Argo CD detects the desired-state change.

Argo CD synchronizes the manifests with EKS.

Kubernetes performs the rollout and readiness probes determine when new pods can receive traffic.

After deployment we monitor logs, metrics, application health, and alerts.”

This answer alone can lead to 20+ follow-up questions.

PART 3 — LINUX

15. What Linux commands do you use every day?

“I commonly use:

ls, cd, pwd, cat, less, grep, find, tail, head, ps, top, df, du, free, curl, ss, chmod, chown, systemctl, journalctl, ssh, scp, kill and tar.”

16. How do you check disk space?

df -h

For directories:

du -sh *

17. How do you check memory?

free -h

And:

top

18. How do you check processes?

ps aux
top

Find specific process:

ps aux | grep nginx

19. How do you check ports?

ss -tulpn

20. How do you check service status?

systemctl status nginx

Logs:

journalctl -u nginx

21. How do you find errors in a log?

grep -i error application.log

Live logs:

tail -f application.log

22. Difference between `kill` and `kill -9`?

“Normal kill sends SIGTERM and gives the process an opportunity to shut down gracefully.

kill -9 sends SIGKILL and immediately terminates the process.

I prefer graceful termination first and use SIGKILL only when necessary.”

23. What is chmod 755?

Owner:

7 = read + write + execute

Group:

5 = read + execute

Others:

5 = read + execute

24. Hard link vs symbolic link?

“A hard link references the same inode/data.

A symbolic link stores a path pointing to another file and can cross filesystems.”

25. Server CPU is 100%. What do you do?

CLIF.

“I first confirm the CPU utilization and identify the processes consuming CPU using top or ps.

Then I check application and system logs, recent deployments, traffic changes and monitoring metrics.

I determine whether the problem is application code, abnormal traffic, a runaway process, or insufficient capacity.

Then I mitigate the immediate impact and investigate the root cause.”

PART 4 — GIT / GITHUB

26. What is Git?

“Git is a distributed version-control system used to track source-code changes and collaborate safely.”

27. Git vs GitHub?

“Git is the version-control technology.

GitHub is a platform that hosts Git repositories and provides features such as pull requests, Actions, reviews, branch protection, and collaboration.”

28. Describe your Git workflow.

“Normally I pull the latest main branch, create a feature branch, make changes, commit them, push the branch, and open a pull request.

The PR triggers CI checks and code review.

Once approvals and required checks pass, the PR is merged according to the team's branching policy.”

29. `git fetch` vs `git pull`?

“git fetch downloads remote changes without modifying my working branch.

git pull fetches changes and then integrates them into my current branch.”

30. Merge vs rebase?

“Merge combines histories and normally creates a merge commit.

Rebase moves commits onto another base and creates a cleaner linear history.

I follow the team's Git policy, particularly because rebasing already-shared history can cause problems.”

31. What is a merge conflict?

“A merge conflict happens when Git cannot automatically determine how to combine competing changes.

I inspect the conflicting files, determine the intended final content, resolve the conflict, test the result, stage it and continue the merge or rebase.”

32. `git revert` vs `git reset`?

“git revert creates a new commit that reverses a previous commit and is safer for shared branches.

git reset moves the branch pointer and can rewrite history, so I use it carefully.”

PART 5 — DOCKER

33. What is Docker?

“Docker packages an application and its dependencies into a portable container image so it can run consistently across environments.”

34. Image vs container?

Image = blueprint. Container = running instance.

“An image is an immutable package containing the application and dependencies.

A container is a running instance of an image.”

35. What is Dockerfile?

“A Dockerfile contains instructions for building a Docker image.”

Example:

FROM nginx:alpine
COPY dist /usr/share/nginx/html
EXPOSE 80

36. Build an image.

docker build -t application:v1 .

37. Run container.

docker run -d -p 8080:80 application:v1

38. `CMD` vs `ENTRYPOINT`?

“Both define container startup behavior.

ENTRYPOINT normally defines the executable.

CMD commonly provides the default command or arguments that can be overridden.”

39. `COPY` vs `ADD`?

“COPY simply copies files into the image and is usually preferred.

ADD has additional functionality such as automatic local tar extraction.”

40. Why multi-stage builds?

“To reduce final image size and attack surface by separating build dependencies from runtime dependencies.”

41. Container keeps crashing. What do you check?

docker ps -a
docker logs <container>
docker inspect <container>

“I check exit code, logs, environment variables, command/entrypoint, resource constraints, configuration and application dependencies.”

PART 6 — KUBERNETES

42. What is Kubernetes?

“Kubernetes is a container orchestration platform that manages deployment, scaling, networking, availability, and lifecycle of containerized applications.”

43. Explain Kubernetes architecture.

Control Plane:

API Server

Scheduler

Controller Manager

etcd

Worker:

kubelet

container runtime

kube-proxy/networking components

Pods

44. What is a Pod?

“A Pod is Kubernetes' smallest deployable unit. It contains one or more containers that share networking and can share storage.”

45. Pod vs Deployment?

“A Pod runs containers.

A Deployment manages replicated application Pods through ReplicaSets and supports declarative updates, scaling and rollouts.”

46. Deployment vs StatefulSet?

“Deployment is generally used for stateless applications where pod identity isn't important.

StatefulSet is designed for stateful workloads requiring stable pod identities, ordered behavior, and persistent storage relationships.”

47. Deployment vs DaemonSet?

“A Deployment controls a desired number of application replicas.

A DaemonSet normally ensures a pod runs on each eligible node, making it useful for node-level agents such as logging or monitoring components.”

48. What is a Service?

“A Service provides a stable network endpoint for a changing set of Pods selected by labels.”

49. Service types?

ClusterIP — internal access.

NodePort — exposes a port through nodes.

LoadBalancer — integrates with an external load balancer where supported.

ExternalName — DNS mapping to an external hostname.

50. What is Ingress?

“Ingress defines HTTP/HTTPS routing rules into services.

An Ingress Controller actually implements those rules.

In AWS environments, the AWS Load Balancer Controller can create and manage ALB resources based on Kubernetes resources.”

51. ConfigMap vs Secret?

“ConfigMap stores non-sensitive configuration.

Secret is intended for sensitive values.

For production secrets, I prefer integration with an appropriate secret-management system rather than treating base64-encoded Kubernetes Secrets alone as sufficient protection.”

52. What is Namespace?

“A namespace provides logical isolation and organization of Kubernetes resources.”

53. What is HPA?

“HPA automatically changes replica counts based on observed metrics such as CPU, memory, or supported custom/external metrics.”

54. What is PDB?

“A PodDisruptionBudget limits how many replicas can be voluntarily disrupted simultaneously, helping preserve application availability during operations such as node maintenance.”

55. HPA vs PDB?

Easy memory:

HPA = how many Pods.

PDB = how many can safely disappear during voluntary disruption.

56. Readiness vs liveness?

Readiness = traffic.

Liveness = restart.

“Readiness determines whether a pod should receive traffic.

Liveness determines whether Kubernetes should restart an unhealthy container.”

57. What is a startup probe?

“It protects slow-starting applications by delaying liveness/readiness behavior until startup has successfully completed.”

58. Pod is Pending. What do you do?

kubectl get pods
kubectl describe pod <pod>

“I look at Events.

Common causes include insufficient CPU/memory, scheduling constraints, taints/tolerations, affinity rules, unbound PVCs, or unavailable nodes.”

59. What is CrashLoopBackOff?

“It means the container repeatedly starts, fails, and Kubernetes backs off before restarting it again.”

Check:

kubectl logs <pod>
kubectl logs <pod> --previous
kubectl describe pod <pod>

60. ImagePullBackOff?

“Kubernetes cannot successfully pull the requested container image.”

Check:

kubectl describe pod <pod>

Then investigate image/tag, registry authentication, IAM, network connectivity, and registry availability.

61. Pod is Running but application doesn't work.

Remember the layers:

Pod → Application → Readiness → Service → Endpoints → Ingress → LB/DNS.

Check:

kubectl logs <pod>
kubectl describe pod <pod>
kubectl get svc
kubectl get endpoints
kubectl get ingress

62. What happens if a Pod dies?

“If it is managed by a controller such as a Deployment, the controller works to restore the desired replica count by creating a replacement Pod.”

63. What if a Kubernetes node dies?

“Pods on that node become unavailable. Kubernetes eventually reschedules controller-managed workloads onto healthy nodes when capacity and scheduling constraints permit.”

64. What is etcd?

“etcd is Kubernetes' distributed key-value store containing cluster state used by the control plane.”

65. What is kubelet?

“Kubelet is the node agent responsible for ensuring containers described by assigned PodSpecs are running on that node.”

66. What does Scheduler do?

“The scheduler selects appropriate nodes for unscheduled Pods based on resources, constraints, affinity, taints/tolerations and other scheduling factors.”

67. What are requests and limits?

“Requests influence scheduling and reserve the resource amount considered necessary.

Limits constrain resource usage.

For CPU, exceeding the limit can cause throttling. Exceeding a memory limit can result in an OOM kill.”

68. What is OOMKilled?

“The container exceeded its available memory limit or encountered a memory-related termination condition.”

I investigate:

kubectl describe pod <pod>
kubectl top pod

Then check memory usage patterns and application behavior before simply increasing resources.

69. What are labels and selectors?

“Labels are metadata attached to Kubernetes resources.

Selectors allow other resources such as Services and Deployments to identify matching resources.”

70. Service has no endpoints. Why?

“First I check whether the Service selector matches the labels on the Pods and whether the Pods are ready.”

kubectl get svc
kubectl get pods --show-labels
kubectl get endpoints

71. How do you rollback Kubernetes deployment?

kubectl rollout history deployment/app
kubectl rollout undo deployment/app

PART 7 — AWS

72. What AWS services have you used?

“EC2, VPC, IAM, EKS, ECR, S3, RDS, Route 53, CloudWatch, ALB, Auto Scaling, Secrets Manager, KMS and related networking/security services.”

73. What is EC2?

“EC2 provides virtual compute instances in AWS.”

74. What is VPC?

“A VPC is a logically isolated network in AWS where we define IP ranges, subnets, routing and connectivity.”

75. Public vs private subnet?

“A public subnet has a route that enables internet access through an Internet Gateway for appropriately configured resources.

Private subnets don't directly route outbound internet traffic through an Internet Gateway. When outbound internet access is required, they can use mechanisms such as a NAT Gateway.”

76. Security Group vs NACL?

SG = instance/resource level + stateful.

NACL = subnet level + stateless.

“Security Groups are stateful and apply to supported network interfaces/resources.

Network ACLs operate at the subnet boundary and are stateless.”

77. What is IAM?

“IAM controls authentication and authorization for AWS resources through identities, roles and policies.”

78. User vs role?

“An IAM user represents a long-lived IAM identity.

A role is assumed to obtain temporary credentials and is preferred for workloads and service-to-service access.”

79. What is least privilege?

“Grant only the permissions necessary to perform the required task and no more.”

80. What is EKS?

“Amazon EKS is AWS's managed Kubernetes service. AWS manages the Kubernetes control plane while customers manage their workloads and, depending on the configuration, compute and associated infrastructure.”

81. ECR?

“Amazon ECR is AWS's managed container registry used to store and distribute container images.”

82. S3?

“S3 is AWS object storage used for objects such as artifacts, logs, backups and Terraform state.”

83. ALB vs NLB?

“ALB operates at Layer 7 and provides HTTP/HTTPS-aware routing capabilities.

NLB operates primarily at Layer 4 and is designed for high-performance TCP/UDP/TLS traffic.”

84. What is Route 53?

“AWS's DNS service. It manages DNS records and provides routing and health-check capabilities.”

85. What is Auto Scaling?

“It dynamically adjusts compute capacity based on configured policies, metrics and desired capacity.”

PART 8 — TERRAFORM

86. What is Terraform?

“Terraform is an Infrastructure as Code tool that allows us to declaratively define and manage infrastructure.”

87. Main Terraform workflow?

Write → Init → Plan → Apply.

terraform init
terraform fmt
terraform validate
terraform plan
terraform apply

88. What is Terraform state?

“Terraform state maps resources in configuration to the infrastructure Terraform manages and stores important resource metadata.”

89. Why remote state?

“For team environments, remote state provides centralized storage and enables safer collaboration.

For AWS, S3 is commonly used for remote state, with appropriate state-locking mechanisms and versioning/protection.”

90. What is a Terraform module?

“A reusable collection of Terraform configuration representing infrastructure components.”

Example:

modules/
  vpc/
  eks/
  rds/

91. Variables vs outputs?

“Variables provide inputs to configuration.

Outputs expose useful values from configuration or modules.”

92. What is `terraform plan`?

“It calculates and displays the proposed changes without intentionally applying them.”

93. What is Terraform drift?

“Drift occurs when actual infrastructure differs from the configuration/state expectations, often because something was changed outside the normal IaC workflow.”

94. Someone manually changed AWS. What do you do?

“I first determine what changed and why.

I run terraform plan to understand the difference.

Then we decide whether the manual change should be represented in code or reverted so the environment again matches the desired configuration.

I avoid blindly applying Terraform without reviewing the plan.”

95. Terraform state is lost. What do you do?

“I don't immediately run terraform apply.

First I stop changes to the environment.

I check the remote backend, S3 version history/backups and locking configuration.

If a valid state version exists, I restore it carefully.

If state cannot be recovered, existing resources may need to be imported into Terraform.

Then I run terraform plan and verify that Terraform isn't planning to recreate or destroy production infrastructure unexpectedly.”

96. What is `terraform import`?

“It associates an existing infrastructure resource with a Terraform resource address/state so Terraform can manage it. Configuration still needs to accurately describe the imported resource.”

97. `count` vs `for_each`?

“Both create multiple resource instances.

count uses numeric indexes.

for_each uses keys from a map or set and can provide more stable resource identity when managing named objects.”

98. What is `depends_on`?

“It explicitly defines a dependency when Terraform cannot infer the dependency from resource references.”

PART 9 — JENKINS / GITHUB ACTIONS

99. What is Jenkins?

“Jenkins is an automation server commonly used to implement CI/CD pipelines.”

100. What is GitHub Actions?

“GitHub Actions is GitHub's workflow automation platform. Workflows are defined in YAML and triggered by events such as pushes and pull requests.”

101. Workflow vs job vs step?

Workflow → Jobs → Steps.

“A workflow is the overall automation.

It contains one or more jobs.

Each job contains steps.”

102. Why run jobs in parallel?

“To reduce pipeline duration when jobs don't depend on each other.”

For example:

Lint | Tests | SonarQube | Trivy

can run concurrently when appropriate.

103. How do you store CI secrets?

“I don't hardcode credentials in the repository.

I use the CI platform's secrets mechanism, OIDC/workload identity where supported, and external secret-management solutions depending on the environment.”

104. Pipeline fails. What do you do?

“I identify the failing job and exact step, inspect its logs, determine whether the failure is code, tests, dependencies, credentials, infrastructure or the pipeline itself, reproduce locally when practical, fix the root cause, rerun the affected workflow and verify the result.”

PART 10 — SONARQUBE / TRIVY / SECURITY

105. What is SonarQube?

“SonarQube performs static code analysis and helps identify bugs, vulnerabilities, security hotspots, code smells and maintainability issues.”

106. What is a Quality Gate?

“A Quality Gate defines conditions that code analysis must satisfy before we consider the code acceptable according to the organization's policy.”

107. What is Trivy?

“Trivy is a security scanner that can detect vulnerabilities and certain misconfigurations across container images, filesystems, repositories and IaC depending on how it's used.”

108. SonarQube vs Trivy?

Easy memory:

SonarQube → primarily code quality/static analysis.

Trivy → vulnerabilities/configuration/container/IaC scanning.

“They overlap somewhat in security-related capabilities, but they solve different parts of the software-security workflow.”

109. Trivy finds CRITICAL vulnerability. What happens?

“I don't ignore it just to make CI green.

I identify the affected package and available remediation, assess exploitability and organizational policy, upgrade or replace the vulnerable dependency/base image when possible, rebuild and scan again.

Any exception should follow the organization's documented risk-acceptance process.”

PART 11 — ARGO CD / GITOPS

110. What is GitOps?

“GitOps uses Git as the version-controlled source of desired configuration. Changes go through Git workflows, and an automated controller reconciles the environment toward that desired state.”

111. What is Argo CD?

“Argo CD is a GitOps continuous-delivery controller for Kubernetes. It compares the desired state in Git with the live cluster and can synchronize differences.”

112. Why Argo CD if we already have GitHub Actions?

MEMORIZE:

GitHub Actions builds. Argo CD deploys/reconciles.

“A common separation is:

GitHub Actions handles CI—testing, scanning, building and publishing artifacts.

Argo CD handles Kubernetes CD by reconciling cluster configuration with Git.

That separation keeps deployment state declarative and auditable.”

113. What is OutOfSync?

“It means the live Kubernetes state differs from the desired state Argo CD sees in Git.”

114. What is self-healing?

“When configured, Argo CD can detect certain live-state drift and reconcile the cluster back toward the desired Git state.”

PART 12 — HELM

115. What is Helm?

“Helm is a Kubernetes package manager and templating system used to package, configure, install and upgrade Kubernetes applications.”

116. What is a Helm chart?

“A chart is a package containing Kubernetes templates, default values and chart metadata.”

Typical:

Chart.yaml
values.yaml
templates/

117. Why Helm?

“It reduces duplicated YAML and makes Kubernetes configuration reusable and configurable across environments.”

118. Helm vs Argo CD?

“Helm renders/packages Kubernetes configuration.

Argo CD manages GitOps reconciliation and deployment.

They can work together: Argo CD can deploy an application represented by a Helm chart.”

PART 13 — NETWORKING

119. What is DNS?

“DNS translates names such as app.company.com into addresses/resources clients can connect to.”

120. TCP vs UDP?

“TCP is connection-oriented and provides ordered, reliable delivery.

UDP is connectionless with lower protocol overhead but doesn't provide TCP's delivery guarantees.”

121. HTTP vs HTTPS?

“HTTPS is HTTP protected using TLS, providing encryption and server authentication, and potentially client authentication depending on configuration.”

122. What happens when you type a URL?

Excellent interview question.

Remember: DNS → TCP/TLS → HTTP → LB → App → Response.

“First the client resolves the hostname through DNS.

It establishes network connectivity to the destination.

For HTTPS, a TLS handshake establishes the secure connection.

The client sends an HTTP request.

The request may pass through a load balancer or ingress to the application.

The application processes it, potentially communicating with databases or other services, and returns an HTTP response.”

123. What is a load balancer?

“It distributes traffic across multiple healthy application targets to improve availability and scalability.”

124. 502 vs 503 vs 504?

Easy memory:

502 → bad upstream response.

503 → service unavailable.

504 → upstream timeout.

“But the exact source depends on which proxy/load balancer generated the status code, so I check the relevant logs and metrics rather than assuming the cause.”

PART 14 — MONITORING / OBSERVABILITY

125. What is monitoring?

“Monitoring collects and evaluates system/application signals so teams can understand health and detect problems.”

126. Metrics vs logs vs traces?

Metrics = numbers.

Logs = events/details.

Traces = request journey.

127. What do you monitor in Kubernetes?

“Node CPU/memory/disk, pod health, resource utilization, restart counts, request latency, error rates, application throughput, deployment health and important business/application metrics.”

128. What is Prometheus?

“Prometheus is a monitoring and time-series system commonly used for collecting metrics and evaluating alerting rules.”

129. What is Grafana?

“Grafana provides visualization and dashboards over data sources such as Prometheus and others.”

130. What is CloudWatch?

“AWS's monitoring and observability service for metrics, logs, alarms, dashboards and related AWS telemetry.”

131. What is Datadog?

“Datadog is an observability platform providing capabilities around infrastructure monitoring, metrics, logs, traces, dashboards and alerting.”

PART 15 — SRE QUESTIONS

132. What is SRE?

“Site Reliability Engineering applies software-engineering approaches to operations and reliability.”

133. What is SLI?

“SLI is the actual measured reliability indicator.”

Example:

99.95% successful requests.

134. What is SLO?

“SLO is the reliability target for an SLI.”

Example:

99.9% successful requests over the defined period.

135. What is SLA?

“An SLA is a formal service agreement that can define service commitments and consequences when commitments aren't met.”

136. What is error budget?

“An error budget represents the amount of unreliability allowed by an SLO.

For a 99.9% availability objective, the remaining 0.1% represents the basic availability error budget over the measurement period.”

137. What is toil?

“Toil is repetitive operational work that is manual, automatable, tactical and scales poorly.

An SRE objective is to reduce unnecessary toil through engineering and automation.”

PART 16 — PRODUCTION INCIDENTS

138. Production is down. What do you do?

THIS IS CRITICAL.

Remember:

Confirm → Impact → Changes → Logs/Metrics → Mitigate → Fix → Verify → RCA

Answer:

“First I confirm the incident and determine customer impact.

I check monitoring, alerts, logs and metrics.

I look for recent deployments or infrastructure/configuration changes.

My first operational priority is restoring service safely. If a recent deployment caused the incident, rollback may be the fastest mitigation.

Once service is stable, we identify and fix the root cause, verify recovery, communicate status and document the incident.

Afterward we conduct a blameless postmortem and implement preventive actions.”

139. Deployment caused 500 errors.

“I check application metrics and logs and correlate the start of errors with the deployment.

If customer impact is significant and the previous release is known-good, I prioritize safe rollback.

After restoring service, I investigate the new release offline and identify the root cause.”

140. Deployment caused 504 errors.

“I investigate the complete request path:

Load balancer → Ingress → Service → Pod → downstream dependencies.

I check target health, readiness, endpoints, application logs, response latency, database/network dependencies and recent configuration changes.

If the release introduced the problem, rollback may be appropriate while we investigate.”

141. Database connection suddenly fails.

“I check application errors, database availability, network connectivity, DNS, security rules, credentials/secrets, connection pool behavior and recent changes.

I avoid assuming the database itself is down until I identify which layer is failing.”

142. Application suddenly becomes slow.

“First I establish whether latency affects all requests or a specific endpoint.

Then I correlate latency with CPU, memory, traffic, error rate, database latency and downstream-service metrics.

Distributed tracing can help identify which component is consuming the time.

Then I mitigate the bottleneck and investigate its root cause.”

143. Disk is full.

df -h
du -sh /*

“I identify which filesystem and directories are consuming space, check logs and temporary/application files, and determine why growth occurred.

I don't blindly delete files from production.

After safely recovering capacity, I fix retention, log rotation, storage sizing, or the application behavior that caused it.”

PART 17 — SCENARIO QUESTIONS

144. Developer says: “It works on my machine.”

“I compare environmental differences: image version, environment variables, dependencies, configuration, secrets, networking and runtime versions.

Containers and reproducible build processes help reduce these inconsistencies.”

145. CI passed but production failed. Why?

“CI success doesn't guarantee production success.

Potential differences include production configuration, secrets, traffic patterns, infrastructure, databases, network policies, resource limits and external dependencies.

I investigate production telemetry and compare the deployed artifact and configuration against what CI validated.”

146. Application works directly through Pod IP but not Service.

“I check Service selectors, Pod labels, endpoints, ports, targetPort, readiness and network policies.”

147. Service works but domain doesn't.

“I move outward:

Service → Ingress → Load Balancer → DNS → TLS.

I verify ingress configuration, load-balancer health/listeners, DNS records and certificate/TLS configuration.”

148. One Pod works, another doesn't.

“I compare the failing Pod with a healthy Pod:

image digest/tag, node, environment, mounted configuration/secrets, logs, events, resource usage and network connectivity.

That comparison usually narrows the problem quickly.”

PART 18 — SECURITY QUESTIONS

149. How do you secure a CI/CD pipeline?

“I use least-privilege permissions, protected branches, code reviews, short-lived credentials/OIDC when available, secret management, dependency and image scanning, SAST, artifact integrity controls, restricted runners and auditable deployment processes.”

150. How do you secure Kubernetes?

“RBAC and least privilege, workload identity, namespace and network isolation, Pod Security controls, image scanning, secret management, resource limits, admission policies where appropriate, audit logging and regular patching/upgrades.”

151. How do you manage secrets?

“I never intentionally store plaintext production secrets in Git.

Depending on the platform, I use solutions such as AWS Secrets Manager and appropriate Kubernetes integrations, with IAM/RBAC controls and rotation policies.”

152. Secret was committed to Git. What do you do?

“First I treat it as compromised.

I revoke or rotate the credential immediately.

Then I determine its exposure and review relevant audit logs.

We remove it from the repository/history as appropriate, but deleting the Git line alone isn't enough because the credential may already have been copied.

Finally, I add controls to prevent recurrence.”

PART 19 — BEHAVIORAL QUESTIONS

Students should use STAR.

153. Tell me about a production incident you handled.

Situation: A production application started returning errors after deployment.

Task: Restore service quickly and determine the cause.

Action:

“I checked monitoring and application logs and correlated the incident with the new deployment. We rolled back to the previous stable release, which restored service. Further investigation identified an incorrect environment configuration affecting the application's database connectivity.”

Result:

“Service was restored, we corrected the configuration, improved readiness validation and documented the incident in a postmortem.”

154. Tell me about a disagreement with a developer.

“I focus on technical evidence rather than personalities.

For example, if a developer believes infrastructure caused an issue, I compare logs, metrics, deployment history and configuration with them.

The goal isn't to prove who is wrong. The goal is to identify the failing layer and restore the service.”

155. Tell me about a mistake you made.

“I explain the mistake clearly, take responsibility for my part, explain how I corrected it and focus heavily on what process or automation I introduced so the same type of mistake is less likely to happen again.”

Students should prepare a genuine example rather than invent one.

156. How do you prioritize multiple incidents?

“I prioritize based on customer/business impact, severity, scope, security risk and service dependencies.

A production outage affecting customers takes priority over a lower-severity development issue.”

157. How do you work under pressure?

“I rely on structured troubleshooting. I confirm impact, use logs and metrics, make one controlled change at a time, communicate clearly and prioritize restoring service over trying experimental fixes in production.”

PART 20 — SENIOR DEVOPS QUESTIONS

158. How would you design a highly available application in AWS?

“I would distribute workloads across multiple Availability Zones.

A typical architecture could use Route 53, an ALB, application workloads across multiple AZs, autoscaling, a highly available database architecture, caching where appropriate, durable object storage, monitoring and automated recovery.

For Kubernetes, EKS worker capacity should span multiple AZs, applications should have multiple replicas, topology-aware scheduling where appropriate, health probes, PDBs and autoscaling.

The exact architecture depends on the application's availability and recovery requirements.”

159. How do you achieve zero/minimal-downtime deployments?

“Multiple replicas, rolling updates, readiness probes, appropriate termination handling, graceful shutdown, capacity during rollout, PDBs where appropriate and backward-compatible application/database changes.

For higher-risk releases we can also use strategies such as canary or blue/green deployments.”

160. Blue/green vs canary?

Blue/Green: two environments/releases; switch traffic.

Canary: gradually expose a new version to a subset of traffic/users.

161. How would you reduce AWS costs?

“I start with measurements rather than arbitrary downsizing.

I review utilization and right-size compute, use appropriate autoscaling, identify idle resources, optimize storage, lifecycle old data, evaluate Savings Plans/Reserved capacity for predictable workloads, consider Spot for suitable fault-tolerant workloads, and analyze network/data-transfer costs.

Then I verify that cost optimizations don't violate reliability requirements.”

162. How do you design disaster recovery?

“I start with business requirements, particularly RTO and RPO.

Then I design backups, replication, infrastructure recovery through IaC, cross-region or multi-region strategies where justified, documented recovery procedures and—very importantly—regular recovery testing.”

163. RTO vs RPO?

RTO = How quickly must we recover?

RPO = How much data can we afford to lose?

PART 21 — RAPID-FIRE QUESTIONS

Students should answer these in one or two sentences.

164. What is YAML?

A human-readable data serialization format heavily used for configuration.

165. What is JSON?

A structured data-interchange format using objects, arrays and primitive values.

166. What is API?

A defined interface that allows software components to communicate.

167. What is REST?

An architectural style for networked APIs commonly using HTTP resources and methods.

168. GET vs POST?

GET normally retrieves a resource. POST commonly submits data or requests creation/processing.

169. What is SSH?

A secure protocol commonly used for remote shell access and tunneling.

170. What is TLS?

A protocol providing encrypted and authenticated communications.

171. What is port 22?

Common default SSH port.

172. Port 80?

HTTP.

173. Port 443?

HTTPS.

174. What is cron?

A Unix/Linux facility for scheduling recurring commands/jobs.

175. What is Bash?

A Unix shell and scripting language commonly used for automation.

176. What is a reverse proxy?

A server/proxy that accepts client requests and forwards them to backend services.

177. Horizontal vs vertical scaling?

Horizontal = add more instances.

Vertical = increase resources of an instance.

178. Authentication vs authorization?

Authentication = who are you?

Authorization = what can you do?

179. Encryption at rest vs in transit?

At rest protects stored data.

In transit protects data moving across networks.

PART 22 — INTERVIEWER WILL DIG DEEPER

This is where students often get caught.

If student says:

“I use Kubernetes.”

Interviewer asks:

“What happens internally when you run kubectl apply?”

Strong answer:

“kubectl sends the desired resource definition to the Kubernetes API server.

The API server authenticates and authorizes the request, validates it and persists the desired state through the control plane.

Controllers observe the desired state and reconcile resources.

If new Pods need scheduling, the scheduler selects nodes.

Kubelets on those nodes observe their assigned workloads and work with the container runtime to start containers.

Other controllers and networking components establish related resources as required.”

If student says:

“I use Terraform.”

Interviewer:

“Explain what happens when you run terraform apply.”

“Terraform loads configuration and provider information, evaluates dependencies and current state, refreshes/reads necessary infrastructure information, determines required changes and invokes provider APIs to create, update or delete resources according to the approved plan.”

If student says:

“I use Docker.”

Interviewer:

“Why not just deploy the source code?”

“Containers package the application with its runtime dependencies into a versioned artifact. That improves consistency between CI, testing and production and makes releases easier to reproduce and roll back.”

PART 23 — THE MOST IMPORTANT TROUBLESHOOTING TREE

Have students memorize this.

Website is down

Start from the outside and move inward:

DNS

↓

Load Balancer

↓

Ingress

↓

Service

↓

Endpoints

↓

Pod

↓

Container

↓

Application

↓

Database / dependencies

Useful commands:

nslookup app.company.com

curl -v https://app.company.com

kubectl get ingress

kubectl describe ingress <name>

kubectl get svc

kubectl get endpoints

kubectl get pods

kubectl describe pod <pod>

kubectl logs <pod>

kubectl logs <pod> --previous

kubectl top pod

kubectl get events --sort-by=.metadata.creationTimestamp

This is much stronger than randomly restarting Pods.

PART 24 — QUESTIONS STUDENTS SHOULD ASK THE INTERVIEWER

Students should always have questions.

Good examples:

“What does your current CI/CD architecture look like?”

“How is Kubernetes used within the organization?”

“How do you manage Infrastructure as Code?”

“What monitoring and observability platforms does the team use?”

“How does the team handle production incidents and on-call?”

“What would you expect the person in this role to accomplish during the first 90 days?”

“What are the biggest reliability or infrastructure challenges the team is currently working on?”

PART 25 — THE 10 ANSWERS EVERY STUDENT MUST KNOW

If they cannot memorize everything immediately, start here:

1. Tell me about yourself

2. Explain your current project

3. Explain your CI/CD pipeline

4. Explain Kubernetes architecture

5. Troubleshoot a failing Pod

6. Explain Terraform and state

7. Explain AWS VPC/networking

8. Explain Docker

9. Explain a production incident

10. Explain monitoring and troubleshooting

production statefulset argocd

Aisalkyn Aidarova — Wed, 30 Sep 2026 02:11:07 +0000

GitHub
  ↓
Argo CD
  ↓
Kubernetes
  ↓
StatefulSet
  ├── Headless Service
  ├── ConfigMap
  ├── ServiceAccount
  ├── StatefulSet
  ├── PVC per Pod
  ├── PDB
  └── StorageClass
        ↓
      AWS EBS

Repository students should create

restaurant-statefulset-gitops/
│
├── README.md
├── argocd/
│   └── application.yaml
│
└── k8s/
    ├── namespace.yaml
    ├── serviceaccount.yaml
    ├── configmap.yaml
    ├── storageclass.yaml
    ├── headless-service.yaml
    ├── service.yaml
    ├── statefulset.yaml
    ├── pdb.yaml
    └── kustomization.yaml

`k8s/namespace.yaml`

apiVersion: v1
kind: Namespace
metadata:
  name: restaurant

`k8s/serviceaccount.yaml`

apiVersion: v1
kind: ServiceAccount
metadata:
  name: restaurant
  namespace: restaurant
automountServiceAccountToken: false

`k8s/configmap.yaml`

apiVersion: v1
kind: ConfigMap
metadata:
  name: restaurant-config
  namespace: restaurant
data:
  APP_ENV: "production"
  APP_NAME: "restaurant-company"

`k8s/storageclass.yaml`

For EKS production, use the EBS CSI provisioner:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: restaurant-gp3
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
parameters:
  type: gp3
  encrypted: "true"

Explain:

PVC
 ↓
StorageClass
 ↓
EBS CSI Driver
 ↓
AWS creates EBS volume
 ↓
PV

Your students must verify:

kubectl get csidriver

They need:

ebs.csi.aws.com

before this storage can work.

`k8s/headless-service.yaml`

StatefulSet uses this for stable network identity:

apiVersion: v1
kind: Service
metadata:
  name: restaurant-headless
  namespace: restaurant
spec:
  clusterIP: None

  selector:
    app: restaurant

  ports:
    - name: http
      port: 80
      targetPort: http

Now you can teach:

restaurant-0.restaurant-headless.restaurant.svc.cluster.local
restaurant-1.restaurant-headless.restaurant.svc.cluster.local
restaurant-2.restaurant-headless.restaurant.svc.cluster.local

`k8s/service.yaml`

This is the normal Service clients use:

apiVersion: v1
kind: Service
metadata:
  name: restaurant
  namespace: restaurant
spec:
  type: ClusterIP

  selector:
    app: restaurant

  ports:
    - name: http
      port: 80
      targetPort: http

`k8s/statefulset.yaml`

Using the ECR image you've been running:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: restaurant
  namespace: restaurant

spec:
  serviceName: restaurant-headless

  replicas: 3

  podManagementPolicy: OrderedReady

  updateStrategy:
    type: RollingUpdate

  selector:
    matchLabels:
      app: restaurant

  template:
    metadata:
      labels:
        app: restaurant

    spec:
      serviceAccountName: restaurant

      terminationGracePeriodSeconds: 30

      securityContext:
        fsGroup: 101

      containers:
        - name: restaurant

          image: 584612873567.dkr.ecr.us-east-1.amazonaws.com/resto-comp:latest

          imagePullPolicy: Always

          ports:
            - name: http
              containerPort: 80

          envFrom:
            - configMapRef:
                name: restaurant-config

          resources:
            requests:
              cpu: 100m
              memory: 128Mi

            limits:
              cpu: 500m
              memory: 512Mi

          readinessProbe:
            httpGet:
              path: /
              port: http
            initialDelaySeconds: 5
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 3

          livenessProbe:
            httpGet:
              path: /
              port: http
            initialDelaySeconds: 15
            periodSeconds: 20
            timeoutSeconds: 3
            failureThreshold: 3

          volumeMounts:
            - name: restaurant-data
              mountPath: /data

  volumeClaimTemplates:
    - metadata:
        name: restaurant-data

      spec:
        accessModes:
          - ReadWriteOnce

        storageClassName: restaurant-gp3

        resources:
          requests:
            storage: 5Gi

This creates:

restaurant-0
   ↓
restaurant-data-restaurant-0
   ↓
EBS-0


restaurant-1
   ↓
restaurant-data-restaurant-1
   ↓
EBS-1


restaurant-2
   ↓
restaurant-data-restaurant-2
   ↓
EBS-2

`k8s/pdb.yaml`

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: restaurant
  namespace: restaurant

spec:
  minAvailable: 2

  selector:
    matchLabels:
      app: restaurant

Explain that PDB is about voluntary disruptions, such as node maintenance/draining. It doesn't guarantee that two Pods can never fail.

`k8s/kustomization.yaml`

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - namespace.yaml
  - serviceaccount.yaml
  - configmap.yaml
  - storageclass.yaml
  - headless-service.yaml
  - service.yaml
  - statefulset.yaml
  - pdb.yaml

Students can test the complete manifest before Argo CD:

kubectl kustomize k8s/

Then:

kubectl apply --dry-run=server -k k8s/

Argo CD Application

Now create:

argocd/application.yaml

Use:

apiVersion: argoproj.io/v1alpha1
kind: Application

metadata:
  name: restaurant-statefulset
  namespace: argocd

spec:
  project: default

  source:
    repoURL: https://github.com/jumptotechschooldevops/restaurant-statefulset-gitops.git
    targetRevision: main
    path: k8s

  destination:
    server: https://kubernetes.default.svc
    namespace: restaurant

  syncPolicy:
    automated:
      prune: true
      selfHeal: true

    syncOptions:
      - CreateNamespace=true
      - PruneLast=true

Students need to replace repoURL if they're using their own repository.

Now teach the GitOps flow:

Developer
   ↓
git push

GitHub
   ↓
Argo CD watches repository
   ↓
desired state
   ↓
Kubernetes
   ↓
StatefulSet
   ↓
Pods + PVCs

Apply the Argo application:

kubectl apply -f argocd/application.yaml

Then:

kubectl get applications -n argocd

Watch:

kubectl get pods -n restaurant -w

Then:

kubectl get statefulset -n restaurant

kubectl get pvc -n restaurant

kubectl get pv

kubectl get service -n restaurant

Student StatefulSet test

Once everything is Running and PVCs are Bound:

kubectl exec -it restaurant-0 -n restaurant -- sh

Inside:

echo "DATA FROM RESTAURANT-0" > /data/student.txt
cat /data/student.txt
exit

Now:

kubectl delete pod restaurant-0 -n restaurant

Watch:

kubectl get pods -n restaurant -w

Once restaurant-0 is back:

kubectl exec restaurant-0 -n restaurant -- \
  cat /data/student.txt

Expected:

DATA FROM RESTAURANT-0

Then compare Pod 1:

kubectl exec restaurant-1 -n restaurant -- \
  cat /data/student.txt

It should not contain Pod 0's file because Pod 1 has its own PVC.

Argo CD self-healing lab

This is the part I'd definitely include.

Repository says:

replicas: 3

Student manually changes Kubernetes:

kubectl scale statefulset restaurant \
  --replicas=1 \
  -n restaurant

Then:

kubectl get statefulset -n restaurant

Ask:

Git says 3. Cluster says 1. Which one wins?

Because you enabled:

automated:
  selfHeal: true

Argo CD should reconcile the cluster back toward the Git desired state.

Teach:

Git = desired state

Cluster = actual state

       ↓

Argo CD compares them

Desired != Actual
       ↓
Argo CD reconciles
       ↓
Actual → Desired

Git change lab

Now tell students don't use kubectl to make the desired change.

Change:

replicas: 3

to:

replicas: 2

Then:

git add .
git commit -m "Scale restaurant StatefulSet to 2 replicas"
git push

Argo CD:

Git changed
   ↓
Argo detects change
   ↓
Sync
   ↓
StatefulSet changes

Check:

kubectl get pods -n restaurant

Production lesson: never put passwords in Git

Do not teach students to commit this:

DB_PASSWORD: mypassword123

into ConfigMap or a plaintext Secret in a public repository.

Explain:

ConfigMap
   ↓
non-sensitive configuration

APP_ENV
APP_NAME
LOG_LEVEL

Secrets should come from an appropriate secret-management workflow, such as AWS Secrets Manager with an external-secrets integration, rather than plaintext Git.

Very important architecture note for students

Your Restaurant Company application is a frontend served by Nginx.

In a real production architecture, I would teach them:

Frontend
Restaurant Company
      ↓
Deployment
      ↓
3 interchangeable Pods


Database
PostgreSQL/MySQL/etc.
      ↓
Stateful workload
      ↓
stable identity
      +
persistent storage

So tell them:

Мы запускаем Restaurant Company как StatefulSet для лаборатории, чтобы увидеть identity, PVC, PV, EBS и Argo CD.

Это не означает, что каждый frontend должен быть StatefulSet.

That's a very important production distinction.

Final student architecture

                    GITHUB
                       │
                       │ git push
                       ▼
                    ARGO CD
                       │
                  sync/self-heal
                       │
                       ▼
                EKS KUBERNETES
                       │
                 StatefulSet
                       │
       ┌───────────────┼───────────────┐
       ▼               ▼               ▼
 restaurant-0     restaurant-1     restaurant-2
       │               │               │
       ▼               ▼               ▼
    PVC-0            PVC-1           PVC-2
       │               │               │
       ▼               ▼               ▼
    EBS-0            EBS-1           EBS-2

           Headless Service
                  │
           Stable DNS Identity

statefulset vs deployment

Aisalkyn Aidarova — Wed, 30 Sep 2026 00:57:00 +0000

We can deploy the exact same ECR image twice:

Restaurant Company ECR Image
             |
      ┌──────┴──────┐
      ↓             ↓
 Deployment      StatefulSet
      ↓             ↓
 restaurant-*    restaurant-0
 random names    restaurant-1
                 restaurant-2

This lets them see the difference, instead of only hearing the theory.

Part 1 — First find your exact ECR image

On EC2 run:

aws ecr describe-repositories --region us-east-1

Then:

aws ecr list-images \
  --repository-name restaurant-company \
  --region us-east-1

Your image should look something like:

459532536656.dkr.ecr.us-east-1.amazonaws.com/restaurant-company:v1

Use your actual tag below if it is not v1.

LAB 1 — Deployment

Tell students:

Сначала запускаем наш Restaurant Company как Deployment.

Потом удалим Pod и посмотрим, что Kubernetes создаст новый Pod с другим именем.

Create:

mkdir statefulset-lab
cd statefulset-lab

vim deployment.yaml

Put:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: restaurant-deployment

spec:
  replicas: 3

  selector:
    matchLabels:
      app: restaurant

  template:
    metadata:
      labels:
        app: restaurant

    spec:
      containers:
        - name: restaurant
          image: 459532536656.dkr.ecr.us-east-1.amazonaws.com/restaurant-company:v1
          ports:
            - containerPort: 80

Run:

kubectl apply -f deployment.yaml

Then:

kubectl get pods

Students should see something like:

restaurant-deployment-7d8c96b7d4-x7p2m
restaurant-deployment-7d8c96b7d4-k9t5z
restaurant-deployment-7d8c96b7d4-m4q8n

STOP HERE and ask them

Ребята, какое имя у первого Pod?

They'll give the long name.

Then:

Как будет называться следующий Pod?

They cannot know.

Explain:

DEPLOYMENT

restaurant-deployment-xxxxx-A
restaurant-deployment-xxxxx-B
restaurant-deployment-xxxxx-C

Identity не важна.

Delete one Deployment Pod

Get the names:

kubectl get pods

Copy one and delete it:

kubectl delete pod restaurant-deployment-XXXXXXXXXX-XXXXX

Immediately:

kubectl get pods -w

Show them:

OLD:

restaurant-deployment-abc123-x7p2m ❌


NEW:

restaurant-deployment-abc123-j8k4q ✅

Ask:

То же имя вернулось?

Нет.

That's the lesson:

Deployment говорит:

"Мне всё равно, какой Pod. Просто дай мне 3 работающих Pod."

LAB 2 — Same ECR image, but StatefulSet

Don't add storage yet.

Students need to understand identity first.

Create:

vim statefulset.yaml

Use:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: restaurant-statefulset

spec:
  serviceName: restaurant
  replicas: 3

  selector:
    matchLabels:
      app: restaurant-stateful

  template:
    metadata:
      labels:
        app: restaurant-stateful

    spec:
      containers:
        - name: restaurant
          image: 459532536656.dkr.ecr.us-east-1.amazonaws.com/restaurant-company:v1
          ports:
            - containerPort: 80

Apply:

kubectl apply -f statefulset.yaml

Watch it:

kubectl get pods -w

Now something very important happens.

Students should see:

restaurant-statefulset-0
restaurant-statefulset-1
restaurant-statefulset-2

Tell them:

STOP. Смотрите на имена.

Compare:

DEPLOYMENT

restaurant-deployment-7d8c9-x8p2m
restaurant-deployment-7d8c9-k3j9q
restaurant-deployment-7d8c9-p7w4a


STATEFULSET

restaurant-statefulset-0
restaurant-statefulset-1
restaurant-statefulset-2

Now ask:

Если я поставлю replicas: 4, как будет называться следующий Pod?

Students should answer:

restaurant-statefulset-3

That's when they'll start understanding StatefulSet.

Now delete StatefulSet Pod

Run:

kubectl delete pod restaurant-statefulset-1

Then:

kubectl get pods -w

They should see:

restaurant-statefulset-1

come back.

Tell them:

Я удалил restaurant-statefulset-1.

Kubernetes не сказал: "Дам тебе какой-нибудь новый Pod."

Он сказал:

"Мне нужен именно номер 1 обратно."

Draw:

STATEFULSET

restaurant-statefulset-0
restaurant-statefulset-1  ❌
restaurant-statefulset-2

             ↓

restaurant-statefulset-0
restaurant-statefulset-1  ✅
restaurant-statefulset-2

This is stable identity.

Now the question that makes the class understand

Ask:

Хорошо. Имя вернулось. Но если внутри restaurant-statefulset-1 были важные данные — вернутся ли данные?

Answer:

Not automatically.

Now introduce storage.

Identity:
restaurant-statefulset-1
        +
Storage:
PVC-1

        =

Stateful application

LAB 3 — StatefulSet + Persistent Storage

For your Restaurant Company app, Nginx normally serves static files, so it does not actually need StatefulSet in production.

Tell students this clearly:

Мы используем Restaurant Company с StatefulSet для обучения.

В реальном проекте этот frontend лучше запускать Deployment.

Database — MySQL, PostgreSQL, MongoDB — гораздо более естественный пример StatefulSet.

But Restaurant Company makes the mechanics easy to see.

Let's give each Pod its own /data.

First check:

kubectl get storageclass

Then modify statefulset.yaml:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: restaurant-statefulset

spec:
  serviceName: restaurant
  replicas: 3

  selector:
    matchLabels:
      app: restaurant-stateful

  template:
    metadata:
      labels:
        app: restaurant-stateful

    spec:
      containers:
        - name: restaurant
          image: 459532536656.dkr.ecr.us-east-1.amazonaws.com/restaurant-company:v1

          ports:
            - containerPort: 80

          volumeMounts:
            - name: restaurant-data
              mountPath: /data

  volumeClaimTemplates:
    - metadata:
        name: restaurant-data

      spec:
        accessModes:
          - ReadWriteOnce

        resources:
          requests:
            storage: 1Gi

Because an existing StatefulSet may not accept this structural change cleanly, for the classroom the simplest route is to recreate the lab StatefulSet:

kubectl delete statefulset restaurant-statefulset

Then:

kubectl apply -f statefulset.yaml

Watch:

kubectl get pods -w

Then:

kubectl get pvc

Now the students should see the big idea:

restaurant-data-restaurant-statefulset-0
restaurant-data-restaurant-statefulset-1
restaurant-data-restaurant-statefulset-2

Draw:

restaurant-statefulset-0
           |
           ↓
restaurant-data-restaurant-statefulset-0


restaurant-statefulset-1
           |
           ↓
restaurant-data-restaurant-statefulset-1


restaurant-statefulset-2
           |
           ↓
restaurant-data-restaurant-statefulset-2

Now PROVE the data survives

This is the most important part of today's class.

Enter Pod 1:

kubectl exec -it restaurant-statefulset-1 -- sh

Inside:

echo "HELLO FROM POD NUMBER 1" > /data/student.txt

Check:

cat /data/student.txt

You should get:

HELLO FROM POD NUMBER 1

Exit:

exit

Now tell students:

Сейчас я уничтожу Pod.

kubectl delete pod restaurant-statefulset-1

Watch:

kubectl get pods -w

Wait until:

restaurant-statefulset-1   1/1   Running

Now:

kubectl exec -it restaurant-statefulset-1 -- sh

Then:

cat /data/student.txt

And:

HELLO FROM POD NUMBER 1

This is your wow moment.

Tell them:

Pod умер.

Container умер.

Новый container был создан.

Но данные остались.

Because:

OLD POD
restaurant-statefulset-1 ❌
           |
           ↓
        PVC-1 ✅
           |
           ↓
         DATA ✅


NEW POD
restaurant-statefulset-1 ✅
           |
           ↓
        SAME PVC-1
           |
           ↓
         DATA ✅

Then prove each Pod has its OWN storage

Run:

kubectl exec -it restaurant-statefulset-0 -- sh

Inside:

cat /data/student.txt

It should not contain Pod 1's file.

Then:

echo "I AM POD ZERO" > /data/student.txt
exit

Check Pod 0:

kubectl exec restaurant-statefulset-0 -- cat /data/student.txt

Result:

I AM POD ZERO

Check Pod 1:

kubectl exec restaurant-statefulset-1 -- cat /data/student.txt

Result:

HELLO FROM POD NUMBER 1

Now draw this:

             STATEFULSET

       restaurant-statefulset

       /          |          \
      /           |           \

    POD-0        POD-1        POD-2
      |            |            |
      ↓            ↓            ↓
    PVC-0        PVC-1        PVC-2
      |            |            |
      ↓            ↓            ↓
"I AM 0"      "HELLO 1"      DATA-2

The sentence I want your students to repeat

Have the whole class say this:

Deployment сохраняет количество Pod.

StatefulSet сохраняет identity Pod.

PVC сохраняет данные.

And then:

Deployment:
"I need 3 Pods."


StatefulSet:
"I need Pod 0,
 I need Pod 1,
 I need Pod 2,
 and each can have its own storage."

Two-Day Homework — CI `needs` + Build + Docker

Aisalkyn Aidarova — Tue, 22 Sep 2026 18:52:03 +0000

The goal is to reproduce this entire architecture independently:

Developer
   ↓
Feature Branch
   ↓
Pull Request
   ↓
┌──────── Lint ────────┐
│                      │
├──────── SonarQube ────┼──→ ALL PASS
│                      │       ↓
└──────── Trivy ────────┘     needs
                               ↓
                             Build
                               ↓
                             dist/
                               ↓
                           Dockerfile
                               ↓
                         Docker Image
                               ↓
                           Container
                               ↓
                      Host 8080 → 80
                               ↓
                             Nginx
                               ↓
                     Restaurant Company

Part 1 — Git preparation

Start from the repository:

cd ~/restaurant-company

git status
git branch

git switch main
git pull origin main

git switch -c homework-needs-docker

Verify:

git branch
git status

Students must explain: What is main? Why do we create a feature branch? Why do we run git pull before creating the branch? What does git status tell us?

Part 2 — Review the three parallel CI jobs

Open:

cat .github/workflows/ci.yml

Students must identify:

lint
sonarqube
trivy

They should write answers to these questions:

What is a Job? What is a Runner? What is a Step? Why can Lint, SonarQube and Trivy run in parallel? What is the difference between parallel jobs and sequential steps? What does runs-on: ubuntu-latest mean?

They should draw:

               ┌── Lint
               │
Pull Request ──┼── SonarQube
               │
               └── Trivy

Part 3 — Yesterday's main topic: `needs`

Students must add/review:

build:
  name: Build
  runs-on: ubuntu-latest

  needs:
    - lint
    - sonarqube
    - trivy

They must explain every line.

Especially:

needs:
  - lint
  - sonarqube
  - trivy

Written questions:

What does needs mean? What is a dependent job? What is an independent job? Are lint, sonarqube, and trivy job IDs or display names? Why shouldn't Build start immediately?

Then explain:

Lint       ✅
SonarQube  ✅
Trivy      ✅
              ↓
Build      ▶ RUN

versus:

Lint       ✅
SonarQube  ✅
Trivy      ❌
              ↓
Build      ⏭ SKIPPED

Students must answer:

What is the difference between FAILED and SKIPPED?

Part 4 — Build Job

Complete/review:

build:
  name: Build
  runs-on: ubuntu-latest

  needs:
    - lint
    - sonarqube
    - trivy

  steps:
    - name: Checkout code
      uses: actions/checkout@v4

    - name: Setup Node.js
      uses: actions/setup-node@v4
      with:
        node-version: 22
        cache: npm

    - name: Install dependencies
      run: npm ci

    - name: Build application
      run: npm run build

    - name: Verify build
      run: ls -la dist

Students explain:

Checkout
   ↓
Setup Node
   ↓
npm ci
   ↓
npm run build
   ↓
dist/
   ↓
Verify

Written questions: What does npm ci do? Which file locks exact dependencies? Where does npm run build come from? What does Vite create? What is dist/? Why do we run ls -la dist?

Part 5 — Commit, Push and Pull Request

Before committing:

git status
git diff

Then:

git add .github/workflows/ci.yml

git status

git commit -m "Add dependent build job"

git push -u origin homework-needs-docker

Create a Pull Request:

homework-needs-docker
        ↓
   Pull Request
        ↓
       main

Open GitHub Actions.

Take a screenshot showing the jobs.

Students should be able to explain:

        ┌── Lint ────────┐
        │                 │
PR ─────┼── SonarQube ────┼──→ Build
        │                 │
        └── Trivy ────────┘

Part 6 — Docker Fundamentals

Now today's material.

Before touching commands, students write definitions for:

Docker, Dockerfile, Docker Image, Container, Container Runtime.

Then explain:

Dockerfile
    ↓
docker build
    ↓
Image
    ↓
docker run
    ↓
Container

They must clearly answer:

What is the difference between an Image and a Container?

Part 7 — Study the Restaurant Company Dockerfile

Run:

cat Dockerfile

Study:

FROM node:22-alpine AS builder

WORKDIR /app

COPY package*.json ./

RUN npm ci

COPY . .

RUN npm run build


FROM nginx:alpine

COPY nginx.conf /etc/nginx/conf.d/default.conf

COPY --from=builder /app/dist /usr/share/nginx/html

EXPOSE 80

CMD ["nginx", "-g", "daemon off;"]

Students must explain every instruction:

FROM
AS builder
WORKDIR
COPY
RUN
COPY --from
EXPOSE
CMD

Part 8 — Multi-stage Build

Students draw:

STAGE 1 — BUILDER
────────────────────

node:22-alpine
      ↓
package.json
package-lock.json
      ↓
npm ci
      ↓
Source Code
      ↓
npm run build
      ↓
/app/dist


          ↓ COPY


STAGE 2 — RUNTIME
────────────────────

nginx:alpine
      ↓
/usr/share/nginx/html
      ↓
Nginx
      ↓
Port 80

Questions:

Why do we have two FROM instructions? What is a multi-stage build? Why do we need Node.js during Build? Why can Nginx serve the final frontend? What does COPY --from=builder mean?

Part 9 — Build the Docker Image

Run:

docker --version

docker build -t restaurant-company:homework .

Then:

docker images

Take a screenshot.

Students explain:

docker build
-t
restaurant-company
:homework
.

Especially:

What does . mean?

Answer: current directory as the build context.

Part 10 — Run the Container

Run:

docker run -d \
  --name restaurant-homework \
  -p 8080:80 \
  restaurant-company:homework

Verify:

docker ps

Students explain:

docker run
-d
--name
-p
8080:80
restaurant-company:homework

Draw:

EC2 HOST
Port 8080
    ↓
Docker
    ↓
CONTAINER
Port 80
    ↓
Nginx

They must explain:

In 8080:80, which is Host and which is Container?

Part 11 — Go Inside the Container

Run:

docker exec -it restaurant-homework sh

Now inside:

cat /etc/os-release

hostname

pwd

Students answer:

My EC2 is Ubuntu. Why does the container show Alpine?

Then:

cd /usr/share/nginx/html

pwd

ls -la

Inspect:

cat index.html

ls assets

Then:

cat /etc/nginx/conf.d/default.conf

Students must connect this to Dockerfile:

COPY --from=builder /app/dist /usr/share/nginx/html

They should explain:

/app/dist
     ↓
COPY
     ↓
/usr/share/nginx/html

Part 12 — Exit Without Stopping the Container

Run:

exit

Then:

docker ps

Students answer:

Why is the container still running after I typed exit?

They need to understand that they exited their interactive shell; the container's main Nginx process is still running.

Part 13 — Test with curl

From EC2:

curl -I http://localhost:8080

Then:

curl http://localhost:8080

Students should identify an HTTP response such as:

HTTP/1.1 200 OK
Server: nginx

Explain the request:

curl
 ↓
localhost:8080
 ↓
Docker Port Mapping
 ↓
Container :80
 ↓
Nginx
 ↓
Website

Take a screenshot.

Part 14 — Open Restaurant Company in Browser

Check AWS Security Group.

For the lab, allow the needed TCP 8080 access, preferably from the student's own IP when practical.

Then:

http://EC2-PUBLIC-IP:8080

Take a screenshot showing the Restaurant Company website.

Students must draw the complete architecture:

Laptop
   ↓
Browser
   ↓
Internet
   ↓
EC2 Public IP :8080
   ↓
AWS Security Group
   ↓
EC2 Host Port 8080
   ↓
Docker Port Mapping
   ↓
Container Port 80
   ↓
Nginx
   ↓
/usr/share/nginx/html
   ↓
Restaurant Company

Part 15 — Troubleshooting Incident

Now deliberately break the application:

docker stop restaurant-homework

Try:

curl -I http://localhost:8080

Then:

docker ps

Students should investigate:

docker ps -a

Find the problem.

Fix:

docker start restaurant-homework

Verify:

docker ps

curl -I http://localhost:8080

Refresh the browser.

Students write:

CHECK
 ↓
docker ps

FIND
 ↓
Container stopped

FIX
 ↓
docker start restaurant-homework

VERIFY
 ↓
docker ps
curl
browser

This should connect to your troubleshooting model:

CHECK → FIND → FIX → VERIFY

Part 16 — Command Challenge Without Notes

Students should be able to explain these commands without copying definitions:

docker build -t restaurant-company:homework .

docker images

docker run -d --name restaurant-homework -p 8080:80 restaurant-company:homework

docker ps

docker ps -a

docker exec -it restaurant-homework sh

docker stop restaurant-homework

docker start restaurant-homework

curl -I localhost:8080

Part 17 — Interview Questions

Students should write answers in their own words to these questions:

What is Docker? What problem does Docker solve? What is a Dockerfile? What is a Docker Image? What is a Container? Image vs Container? What does docker build do? What does docker run do? What does docker exec do? docker ps vs docker ps -a? What does -p 8080:80 mean? What is a multi-stage Docker build? Why does our first stage use Node.js? Why does our second stage use Nginx? What is dist/? What does COPY --from=builder do? Why is the EC2 Ubuntu but the container Alpine? What is needs in GitHub Actions? Why do Lint/SonarQube/Trivy run in parallel? Why does Build wait? What happens to Build when a required job fails?

Interview answer to practice

“In our CI pipeline, Lint, SonarQube and Trivy run as independent jobs in parallel. Our Build job depends on those required checks using needs. After the checks pass, we build the Vite application and produce the dist directory. We then use a multi-stage Dockerfile: Node.js builds the application, and the final Nginx stage serves the built files. We build a Docker image and run it as a container with host port 8080 mapped to container port 80.”