CD Security Lab

Posted on Oct 30

Nobody knew which server the app was on. Kubernetes, explained with an apartment building

#devops #beginners #linux #docker

Sources: CD's devops wiki 03-kubernetes/01-ecosystem-simple-explanation.md, 02-full-overview.md and 08-lab-webapp-mongodb.md (minikube), cross-checked against kubernetes.io, the minikube release notes and the CNCF 2025 survey on 2026-10-28. Lab names only; screens are drawings.

Saturday, 02:10. One of Contoso's three app servers reboots after a disk error. The on-call engineer opens the wiki page that lists which containers run where. It was last edited in March. Four of the eleven containers on that server are not on the list. One of them was the payment callback. For 50 minutes customers pay and get no confirmation, because nobody knew the callback existed, let alone which server to start it on.

Nothing was hacked. The company simply ran 30 containers by hand, with docker run commands in a shell script and a wiki page that fell behind.

TL;DR

Problem: containers started by hand have no owner, no desired state and no memory. When a server dies, a human has to remember what ran there, start it again, and hope the version is right.
Who it hurts: customers (the thing that was not restarted), the on-call engineer (every outage is archaeology), the business (hours of downtime that a machine could have fixed in seconds), and security (secrets in .env files, nobody knows what is running).
Fix: describe what you want ("3 copies of the web app, this version, this config") and let Kubernetes keep it true. One file replaces the script and the wiki page.
Cost: on a laptop, free (minikube). Managed control planes cost about $0.10 an hour on Azure, AWS and Google, roughly $73 a month, plus the worker VMs you would have paid for anyway.
Prove it: delete a pod on purpose. A new one is running before you finish reading the output.

1. The problem
2. Who gets hurt
3. Cost and value
4. Kubernetes as an apartment building
5. Build it: a cluster on your laptop in 10 minutes
6. Prove it
Gotchas
Checklist
What changed recently
Sources

1. The problem

Running containers by hand works for five containers on one server. It stops working somewhere around the second server, because three questions have no good answer:

Question	By hand	What you actually need
What should be running?	A script, a wiki page, someone's memory	A file that says "3 copies of `web`, version 2.4, with this config"
Who notices when it is not?	A customer, then the on-call phone	Something that checks every few seconds and fixes it
How do other services find it?	A fixed IP that changes when the container moves	A name that stays the same while the containers behind it come and go

Kubernetes answers those three with one idea: you declare the desired state, a controller compares it to the actual state forever, and fixes the difference. Everything else in this post is a name for a piece of that idea.

2. Who gets hurt

Who	What they lose with hand-run containers
Customers	The one container nobody restarted: payments, notifications, logins
On-call engineer	02:00 archaeology: which server, which version, which environment file
The business	50 minutes of downtime where the fix takes a machine 5 seconds
Security	Secrets in `.env` files on disk, no record of what runs, no access control on who starts what
The next hire	Nothing is written down in a form a machine can check, so nothing is written down

The CNCF's 2025 survey (published January 2026) found 82% of organisations that use containers now run Kubernetes in production, up from 66% in 2023. The top challenge was no longer technical ("cultural change with the development team", 47%), ahead of training (36%), security (36%) and complexity (34%). Translation: the tool has settled; the hard part is teaching people how it thinks. That is what this post is for.

3. Cost and value

What it costs to learn (checked 2026-10-28):

Option	Price	Notes
minikube on your laptop (this post)	$0	minikube 1.39.0, Kubernetes 1.37, needs 2 CPUs, 2 GB RAM, 20 GB disk and Docker
Azure Kubernetes Service, Free tier	$0 control plane	No uptime SLA; you pay only for the worker VMs
AKS Standard / Amazon EKS / Google GKE	about $0.10 per cluster-hour, about $73 a month	Control plane only. Worker VMs extra. GKE gives one zonal cluster free via a $74.40 monthly credit
AKS Premium / EKS extended support	about $0.60 per cluster-hour, about $438 a month	Long-term support for old Kubernetes versions. The price of not upgrading

The worker VMs cost the same whether a script or Kubernetes starts the containers. The control plane is the only new line, and it is $0 to $73 a month for a small team.

What it is worth: the Saturday outage above is 50 minutes of lost payments plus a weekend for one engineer. With a Deployment that says replicas: 3, the same disk error costs nothing visible: the scheduler puts the lost pods on the other two nodes within seconds, and the Service keeps the same name throughout. The product you are really buying is the controller that never sleeps.

Where the real cost goes: the CNCF numbers say it is training, not licences. Budget a week per engineer to learn the ten words below, and you have spent more than the first year of control-plane fees.

4. Kubernetes as an apartment building

My notes describe it this way, and it has survived every explanation I have given since:

Building	Kubernetes	What it does
The building	Cluster	Nodes (servers) plus the control plane that manages them
A room	Pod	The smallest thing Kubernetes runs: one or more containers that share an IP and can share a disk
Building manager	Deployment (via a ReplicaSet)	"Keep 3 rooms occupied with version 2.4." Replaces a pod that dies, rolls out new versions, rolls back
Receptionist	Service	One stable name and IP; sends visitors to whichever rooms are occupied right now
Main gate	Ingress or Gateway	Lets HTTP traffic in from outside and routes by host name and path
Floor	Namespace	A logical area: `team-a`, `production`, `staging`. Keeps names and permissions apart
Notice board	ConfigMap	Non-secret settings: URLs, ports, feature flags
Safe	Secret	Passwords, tokens, keys. Not in the image, not in `.env`
Storage room	Volume, PersistentVolumeClaim	Data that must outlive a pod
Thermostat	HorizontalPodAutoscaler	More rooms when CPU is high, fewer when it is low
Key cards	RBAC: ServiceAccount, Role, RoleBinding	Who (or which pod) may do what

A visitor's path is always the same: Ingress → Service → Pod → Volume. The building manager (Deployment) is not on that path; it works in the background making sure the rooms exist.

Under the floor, the control plane has its own parts (from my full-overview notes): the API server takes every request, etcd stores the state, the scheduler picks a node for each pod, the controller manager runs the loops that fix drift, and on each node the kubelet talks to the container runtime (containerd) while kube-proxy wires Services. You rarely touch these; you should know they exist so error messages make sense.

5. Build it: a cluster on your laptop in 10 minutes

Docker Desktop (or Podman) must be running. Then:

minikube start --driver=docker
kubectl get nodes

One node, Ready, version v1.37.x. minikube 1.39 uses containerd as the runtime by default; you do not install anything else.

Create the namespace and the three objects that cover 80% of daily work. One file, shop.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: contoso-shop
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: web-config
  namespace: contoso-shop
data:
  SHOP_NAME: "Contoso Shop (lab)"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: contoso-shop
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: nginx:1.27-alpine
          ports:
            - containerPort: 80
          envFrom:
            - configMapRef:
                name: web-config
          resources:
            requests: { cpu: "50m", memory: "64Mi" }
            limits:   { cpu: "200m", memory: "128Mi" }
---
apiVersion: v1
kind: Service
metadata:
  name: web
  namespace: contoso-shop
spec:
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 80

kubectl apply -f shop.yaml
kubectl get all -n contoso-shop

Three things in that file matter more than the rest:

replicas: 3 is a promise, not a command. You are not starting three containers. You are telling the building manager to keep three rooms occupied, forever.
The label app: web appears twice (selector and template). They must match. If they do not, the Deployment manages nothing and kubectl apply tells you so. My notes call this the number one beginner error, and it still is.
A pinned image and resource requests. nginx:1.27-alpine, not latest. Requests let the scheduler place the pod on a node that has room; limits stop one pod from eating the node.

Reach it from your laptop:

kubectl port-forward -n contoso-shop service/web 8080:80
curl http://127.0.0.1:8080

6. Prove it

The Saturday test, on purpose:

kubectl get pods -n contoso-shop -w
# in a second terminal
kubectl delete pod -n contoso-shop -l app=web --field-selector=status.phase=Running | head -1

Watch the first terminal: one pod goes Terminating, and within a few seconds a new pod with a new name is ContainerCreating, then Running. You did nothing. The receptionist (service/web) never changed its name or IP; curl keeps working through the whole thing.

Two more questions the building manager answers for you:

kubectl scale deployment web -n contoso-shop --replicas=5     # Saturday traffic
kubectl set image deployment/web web=nginx:1.28-alpine -n contoso-shop
kubectl rollout status deployment/web -n contoso-shop          # rolling update, zero downtime
kubectl rollout undo deployment/web -n contoso-shop            # the version was wrong: back in 10 seconds

Gotchas

A bare Pod does not come back. kubectl run or a kind: Pod manifest creates a room with no manager. Delete it and it stays deleted. Always wrap production pods in a Deployment (or StatefulSet, DaemonSet, Job).
Selector and labels must match, or the Deployment manages nothing.
latest is not a version. Two nodes can pull two different images called latest.
No requests, no scheduling sense. Without resources.requests, the scheduler packs pods blind and the node runs out of memory at the worst time.
A Service is not a gate. ClusterIP (the default) is internal only. To let the internet in you need a LoadBalancer Service, an Ingress controller, or the Gateway API.
Ingress NGINX is retired. The community controller most tutorials use stopped receiving fixes, including security fixes, in March 2026. New clusters should use the Gateway API (v1.6, June 2026) or another maintained controller (Traefik, Contour, cloud-native ones). minikube 1.39 ships a Traefik addon for this reason.
.env files are not Secrets. A Kubernetes Secret is still only base64 unless you enable encryption at rest or an external secret store, but it is already better than a file on a server nobody remembers.

Checklist

[ ] minikube start, kubectl get nodes shows Ready
[ ] One manifest: Namespace, ConfigMap, Deployment (replicas: 3), Service
[ ] Labels match between selector and template
[ ] Image pinned, resources.requests and limits set
[ ] kubectl delete pod → a new pod is Running within seconds, curl never fails
[ ] kubectl scale to 5 and back; rollout undo works
[ ] The wiki page that listed containers per server is deleted

What changed recently

Checked on 2026-10-28:

Kubernetes 1.37 "Garhwal" was released 2026-08-26 (patch 1.37.1 on 2026-09-15). My notes were written on 1.35; nothing in this post changed between the two.
minikube 1.39.0 (2026-09-01) defaults to Kubernetes 1.37, uses containerd as the runtime, ships Helm 4 as the guest client and adds a Traefik ingress addon.
Ingress NGINX was retired in March 2026 by the Kubernetes project: no more bug or security fixes. The Kubernetes blog describes it as critical infrastructure for about half of cloud-native environments. Use the Gateway API (v1.6 released 2026-06-30; ingress2gateway converts existing rules) or a maintained controller.
CNCF annual survey (2026-01-20): 82% run Kubernetes in production; the top challenge is now organisational, not technical.
Managed control-plane prices are unchanged at about $0.10 per cluster-hour on AKS Standard, EKS and GKE; the Azure pricing page itself renders the figures client-side, so the number here comes from third-party comparisons dated February and July 2026. Worth re-checking before you quote it.

Sources

Thanks for reading! I'm Cuong, and on CD Security Lab I rebuild real Microsoft security cases in a safe lab and explain them simply. Follow along:

▶️ YouTube (daily Shorts + explainers): CD Security Lab
💼 LinkedIn: Cuong Do
📘 Facebook: CD Security Lab
📸 Instagram: @cdsecuritylab
📝 More posts like this: Dev.to

Got a question about your own tenant? Drop it in the comments.

DEV Community