DEV Community: Adeoye Malumi

The Kubernetes Weekend That Humbled Me — Part 2: From One VM to a Real Cluster

Adeoye Malumi — Thu, 30 Jul 2026 15:18:01 +0000

Quick recap in case you're jumping in here: in Part 1 I fought Multipass, broken cloud-init, a 2.1 GB disk, and a bunch of other nonsense just to get one Ubuntu VM with a healthy Kubernetes control plane on it. That was the "get anything to boot" fight. This part is the fight I actually signed up for — giving pods a real network and turning one lonely VM into an actual cluster.

The CNI Wars

Quick context for anyone newer to this: a CNI plugin is the thing responsible for giving every pod its own IP and making sure traffic can actually get routed between pods, including pods on different nodes. Without one, kubectl works, your control plane is healthy, but nothing you deploy can actually talk to anything else. CoreDNS staying pending at the end of Part 1 was a symptom of exactly this — no CNI, no pod network, no CoreDNS.

First attempt was Flannel, since it's the "default" answer most tutorials give you:

kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

The flannel pod went straight into CrashLoopBackOff with:

failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory

That file gets written by flanneld itself, after it successfully reaches the API server and learns what pod CIDR the cluster is using. In my setup, flannel's own pod couldn't reach the API server at 10.96.0.1:443 — the cluster-internal service IP for the API — so it never got far enough to write that file in the first place. Chicken and egg: the CNI needs API connectivity to bootstrap the network, but with a single-NIC NAT setup like mine, that internal routing wasn't reliably working yet.

Tried Weave next, thinking a different implementation might sidestep it:

kubectl apply -f https://github.com/weaveworks/weave/releases/download/v2.8.1/weave-daemonset-k8s.yaml

Images pulled fine this time, but the readiness probe failed:

Readiness probe failed: Get "http://127.0.0.1:6784/status": dial tcp 127.0.0.1:6784: connect: connection refused

Same root cause wearing a different outfit — Weave's own daemon also needs to reach the API server to establish its network bridge, and it was hitting the same wall Flannel did.

At that point I stopped trying "real" CNIs and just wanted pods to schedule so I could keep moving. I hand-wrote a plain bridge network config that doesn't depend on talking to the API server at all:

sudo apt-get install -y containernetworking-plugins

sudo mkdir -p /etc/cni/net.d
sudo tee /etc/cni/net.d/10-bridge.conf <<EOF
{
  "cniVersion": "1.0.0",
  "name": "bridge",
  "type": "bridge",
  "bridge": "cni0",
  "isGateway": true,
  "ipMasq": true,
  "ipam": {
    "type": "host-local",
    "subnet": "10.244.0.0/16",
    "routes": [{ "dst": "0.0.0.0/0" }]
  }
}
EOF

sudo systemctl restart containerd

The node flipped to Ready and nginx deployed. Worth being straight with you about what this actually is, though: it's a local bridge, not an overlay network. It's fine for unblocking a single node and proving pod scheduling works, but it doesn't give you real cross-node pod routing the way Flannel or Calico would once properly configured. Good enough to keep learning on, not something I'd trust for anything beyond that.

Adding a Worker Node

Rather than repeating the whole cloud-image nightmare from Part 1, I cloned the master's disk — it already had containerd, kubeadm, kubelet, and kubectl installed, which saved me from doing that setup twice:

& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" clonehd "k8s-master\disk.vdi" "k8s-worker-1\disk.vdi" --format VDI

& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" createvm --name k8s-worker-1 --ostype Ubuntu_64 --register
& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifyvm k8s-worker-1 --cpus 2 --memory 4096
& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" storagectl k8s-worker-1 --name "SATA" --add sata
& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" storageattach k8s-worker-1 --storagectl "SATA" --port 0 --device 0 --type hdd --medium "k8s-worker-1\disk.vdi"
& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifyvm k8s-worker-1 --nic1 nat
& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifyvm k8s-worker-1 --natpf1 "ssh,tcp,,2223,,22"

But because I cloned the disk, this new VM also thinks it's still the master. Had to strip that identity out before it could join as anything:

sudo hostnamectl set-hostname k8s-worker-1

sudo sed -i 's/k8s-master/k8s-worker-1/g' /etc/hosts

sudo kubeadm reset -f
sudo rm -rf /etc/kubernetes/ /var/lib/kubelet/ /var/lib/dockershim/ /var/run/kubernetes/ ~/.kube/

kubeadm reset tears down the control-plane state and cluster certificates the clone inherited, so the node starts clean instead of trying to act as a second control plane.

The Network Both VMs Actually Needed

Here's something that trips people up: each VM's default NAT adapter can reach the internet, but NAT networking isolates VMs from each other by design — that's the whole point of NAT, it's not meant for VM-to-VM traffic. Master and worker literally couldn't see each other over NAT alone. Both needed a second network adapter, set to VirtualBox's internal network mode, purely for talking to one another:

& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifyvm k8s-master --nic2 intnet --intnet2 "k8s-net"
& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifyvm k8s-worker-1 --nic2 intnet --intnet2 "k8s-net"

Then static IPs on that internal network so the addresses wouldn't shift around:

Master:

sudo tee /etc/netplan/02-internal.yaml <<EOF
network:
  version: 2
  ethernets:
    enp0s8:
      dhcp4: no
      addresses:
        - 192.168.56.10/24
EOF
sudo netplan apply

Worker:

sudo tee /etc/netplan/02-internal.yaml <<EOF
network:
  version: 2
  ethernets:
    enp0s8:
      dhcp4: no
      addresses:
        - 192.168.56.11/24
EOF
sudo netplan apply

The API Server Was Listening on the Wrong Interface

Even with both VMs able to reach each other, joining still wasn't going to work, because the API server had only ever been told to listen on 10.0.2.15 — the NAT interface — since that was the address I originally passed to kubeadm init back in Part 1. The worker needed to reach it over the internal network instead, on 192.168.56.10, and the API server simply wasn't listening there.

sudo sed -i 's/--advertise-address=10.0.2.15/--advertise-address=192.168.56.10/' /etc/kubernetes/manifests/kube-apiserver.yaml

sleep 30

sudo ss -tlnp | grep 6443
# LISTEN 0 4096 *:6443 *:*

Editing a file under /etc/kubernetes/manifests/ is enough on its own — kubelet watches that directory and automatically restarts the API server pod when it changes, no manual restart command needed.

Joining the Worker

kubeadm token create --print-join-command

Run that on the master, then on the worker with sudo:

sudo kubeadm join 192.168.56.10:6443 --token <TOKEN> --discovery-token-ca-cert-hash sha256:<HASH>

Quick note on what those two values actually do, since it's easy to treat them as just noise you copy-paste: the token is a short-lived credential that authenticates the worker to the cluster, and the discovery hash lets the worker verify it's actually talking to the real API server's certificate authority and not something impersonating it. Both matter — without the hash check, a join command intercepted on the network could point your worker at the wrong cluster entirely.

It joined successfully.

Fixing the Node IPs

kubectl get nodes -o wide

NAME           STATUS   ROLES           AGE   VERSION   INTERNAL-IP   OS-IMAGE
k8s-worker-1   Ready    <none>          25s   v1.29.6   10.0.2.15     Ubuntu 22.04.5 LTS
ubuntu         Ready    control-plane   4m    v1.29.6   10.0.2.15     Ubuntu 22.04.5 LTS

Both nodes reporting the exact same INTERNAL-IP was the giveaway that something was off — that's the NAT address again, not the internal network address I actually wanted the cluster using. Kubelet, left to its own defaults, picks whatever it decides is the primary interface, which on both VMs was still the original NAT adapter rather than the internal one I'd added afterward. The fix is to tell it explicitly which IP to register:

# On master
sudo nano /var/lib/kubelet/kubeadm-flags.env
# Add: --node-ip=192.168.56.10

# On worker
sudo nano /var/lib/kubelet/kubeadm-flags.env
# Add: --node-ip=192.168.56.11

sudo systemctl daemon-reload
sudo systemctl restart kubelet

NAME           STATUS   ROLES           INTERNAL-IP
ubuntu         Ready    control-plane   192.168.56.10
k8s-worker-1   Ready    <none>          192.168.56.11

That's a real two-node cluster, each one correctly identified on the network I actually built for them.

Proving It Actually Works

kubectl create deployment nginx --image=nginx --replicas=3
kubectl get pods -o wide

Pods landed on both nodes, not just the control plane. That was the moment this stopped being "a VM with kubectl installed" and started being an actual cluster doing what a cluster is supposed to do.

What This Whole Weekend Actually Taught Me

Multipass on Windows is a gamble. Great when it works, but the moment it breaks — ghost VMs, corrupted caches, auth loops — you're debugging the tool instead of learning Kubernetes. Driving VirtualBox directly took longer to set up, but at least every failure told me something true about what was actually happening.

Cloud images assume cloud-init will run. Check that it actually did. Missing SSH host keys, a network interface sitting down, a disk partition way smaller than the disk itself — all three of those trace back to the same root cause: cloud-init's first-boot routine never completed. Once I understood that, every one of those errors made sense instead of feeling random.

A CNI needs API server connectivity to bootstrap itself, so check that connectivity first. Both Flannel and Weave failed for the same underlying reason. If a CNI daemon can't reach 10.96.0.1:443, no amount of reinstalling it will help — the problem is upstream of the CNI entirely.

NAT gets you internet, not VM-to-VM communication. Any time you're building a multi-node setup locally, you need a second, internal-only network specifically for the nodes to reach each other, plus static IPs so kubeadm and kubelet know exactly where things live.

Kubelet and the API server don't automatically know which interface you meant. Both default to whatever they see first, which is often not the interface you actually care about. Pin --advertise-address and --node-ip explicitly instead of trusting the defaults.

The logs were right every time. Every single failure in this whole weekend had its answer sitting in the output, if I actually read it instead of just rerunning the command:

kubectl describe pod <name>
journalctl -u kubelet -f
sudo crictl logs <container-id>

Full Command Reference

Windows host:

& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" showvminfo k8s-master | Select-String "State"
& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" controlvm k8s-master poweroff
& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifyvm k8s-master --cpus 3 --memory 6144
& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" clonehd "source.vdi" "dest.vdi" --format VDI
& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifyhd "disk.vdi" --resize 20480
& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifyvm k8s-master --nic2 intnet --intnet2 "k8s-net"

ssh -p 2222 k8suser@localhost  # Master
ssh -p 2223 k8suser@localhost  # Worker

Linux VM, start to finish:

# System prep
sudo apt-get update && sudo apt-get upgrade -y
sudo apt-get install -y containerd containernetworking-plugins

sudo mkdir -p /etc/containerd
sudo containerd config default | sudo tee /etc/containerd/config.toml
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl restart containerd

sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

sudo modprobe overlay
sudo modprobe br_netfilter
sudo tee /etc/sysctl.d/kubernetes.conf <<EOF
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
sudo sysctl --system

# Kubernetes install
sudo apt-get install -y apt-transport-https ca-certificates curl gnupg
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet=1.29.6-1.1 kubeadm=1.29.6-1.1 kubectl=1.29.6-1.1 --allow-downgrades --allow-change-held-packages
sudo apt-mark hold kubelet kubeadm kubectl

# Master init
sudo kubeadm init --pod-network-cidr=192.168.0.0/16 --apiserver-advertise-address=192.168.56.10 --node-name k8s-master

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# CNI (plain bridge)
sudo mkdir -p /etc/cni/net.d
sudo tee /etc/cni/net.d/10-bridge.conf <<EOF
{
  "cniVersion": "1.0.0",
  "name": "bridge",
  "type": "bridge",
  "bridge": "cni0",
  "isGateway": true,
  "ipMasq": true,
  "ipam": {
    "type": "host-local",
    "subnet": "10.244.0.0/16",
    "routes": [{ "dst": "0.0.0.0/0" }]
  }
}
EOF
sudo systemctl restart containerd

# Worker join
sudo kubeadm join 192.168.56.10:6443 --token <TOKEN> --discovery-token-ca-cert-hash sha256:<HASH>

# Verification
kubectl get nodes -o wide
kubectl get pods -A
kubectl create deployment nginx --image=nginx --replicas=3
kubectl get pods -o wide

That's the whole weekend. One control plane, one worker, three nginx pods scheduled across both — and about eight hours of the kind of debugging that teaches you more than a tutorial ever would.

The Kubernetes Weekend That Humbled Me — Part 1: Just Getting a VM to Boot

Adeoye Malumi — Thu, 30 Jul 2026 13:31:20 +0000

I had a simple plan. Spin up a couple of VMs, run `kubeadm init`, join a worker, deploy nginx, post about it. That plan lasted about twenty minutes before Windows and VirtualBox decided I hadn't suffered enough that week.

This is the honest account of everything that went wrong before I even got to write a single Kubernetes command that mattered — the ghost VMs, the corrupted caches, the recovery-mode hacking, all of it. If you're trying this on a Windows laptop yourself, I want you to actually understand why each thing broke, not just copy my fix and move on. Because trust me, you'll hit your own version of these and you'll need to reason your way out.

Round 1: Multipass, or How to Meet a Ghost

Multipass looked like the easy way in. One command and you're supposed to have a VM:

multipass launch --name k8s-master --cpus 2 --memory 4G --disk 20G

It hung, then eventually gave me timed out waiting for response. Fine, I thought, maybe it's slow. I checked what Multipass thought existed versus what was actually sitting in VirtualBox:

multipass list
# Name        State    IPv4    Image
# k8s-master  Running  N/A     Ubuntu 22.04 LTS

& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" list vms
# "Ubuntu 2" {b03575b5-db86-4c38-9195-d689257821d4}

See the problem? Multipass is convinced it created something called k8s-master. VirtualBox, which is the thing actually doing the work underneath, has never heard that name in its life — it just has some auto-generated "Ubuntu 2". Multipass is basically a client talking to a background service that drives VirtualBox for you, and the two sides keep their own separate records of what exists. When the launch times out mid-handshake, those two records can fall out of sync, and you end up with a VM that's "running" on paper but doesn't actually correspond to anything reachable. No wonder SSH never connected — Multipass was trying to reach a machine that, as far as VirtualBox knows, doesn't exist under that name.

The only clean way out was to delete both records separately, since neither tool knows how to clean up after the other:

multipass delete k8s-master --purge

& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" unregistervm "b03575b5-db86-4c38-9195-d689257821d4" --delete

multipass list
& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" list vms

Lesson for anyone doing this: whenever Multipass acts up, don't just trust multipass list. Cross-check it against VirtualBox directly, because those two can disagree and Multipass won't tell you.

Then it got worse. On the next launch attempt:

multipass launch 22.04 --name k8s-master --cpus 2 --memory 4G --disk 20G
# launch failed: Hash of ... does not match
# launch failed: Remote "" is unknown or unreachable.

The cached base image itself was corrupted, so I wiped it:

Remove-Item -Recurse -Force "C:\ProgramData\Multipass\cache\*"
Restart-Service -Name "Multipass" -Force

And that's when I fell into the real trap — an authentication loop. Multipass suddenly wanted a passphrase it had never asked me to set:

multipass authenticate
# Please enter passphrase:
# authenticate failed: Passphrase is not set.

multipass set local.passphrase
# set failed: The client is not authenticated

You can see the chicken-and-egg there: it wants me authenticated to set the passphrase, but it wants the passphrase to authenticate me. I tried nuking every certificate I could find on both the client and daemon side:

Stop-Service -Name "Multipass" -Force
Remove-Item -Recurse -Force "C:\ProgramData\Multipass\data"
Remove-Item -Recurse -Force "$env:LOCALAPPDATA\Multipass"
Start-Service -Name "Multipass"

Still looped. At that point I stopped treating it as something I could fix and accepted it as a known Windows bug in how Multipass handles cert corruption. I abandoned Multipass entirely and went straight to driving VirtualBox myself — more manual work, but at least I could see exactly what was happening at every step instead of trusting a layer that had just proven it could quietly lie to me.

Round 2: Cloud Images and the ISO That Wouldn't Build

Without Multipass holding my hand, I needed to do what it was doing under the hood: get an Ubuntu cloud image into VirtualBox and feed it configuration on first boot.

cd C:\Users\dell\Downloads
Invoke-WebRequest -Uri "https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.vmdk" -OutFile "ubuntu-22.04-cloud.vmdk" -UseBasicParsing

Here's the thing about cloud images that nobody tells you upfront: they're not a normal installable Ubuntu. They ship stripped down and expect a cloud provider (or you, faking one) to hand them a small config package on first boot — a username, an SSH key, network settings. That package needs to arrive as a specific kind of ISO, called a cidata ISO, containing a user-data and meta-data file that cloud-init reads on startup. Without it, the image boots into basically nothing: no user account, no password, nothing to log in with.

So I went looking for mkisofs to build that ISO:

Invoke-WebRequest -Uri "https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/cdrtools/win32.zip" -OutFile "cdrtools.zip" -UseBasicParsing
# ERROR: AccessDenied

Dead link. Tried SourceForge instead:

Invoke-WebRequest -Uri "https://sourceforge.net/projects/cdrtools/files/3.01/cdrtools-3.01-win32-bin.zip/download" -OutFile "cdrtools.zip" -UseBasicParsing
Expand-Archive -Path "cdrtools.zip" -DestinationPath "C:\cdrtools"
# ERROR: End of Central Directory record could not be found

That error means the file I downloaded wasn't actually a zip — SourceForge's download link redirects through an intermediate page, and Invoke-WebRequest saved that redirect page instead of following it to the real file. Rather than keep fighting dead mirrors for a tool that just builds an ISO 9660 image, I wrote a small Python script to build the cidata.iso myself directly. Sometimes the fastest fix isn't finding the "proper" tool, it's just building the thirty lines of code that does the one thing you actually need.

Round 3: I Nearly Downloaded the Full ISO

At one point I considered giving up on cloud images completely and grabbing the full installable Ubuntu Server ISO instead:

curl.exe -L -o "ubuntu-22.04-server.iso" "https://releases.ubuntu.com/jammy/ubuntu-22.04.5-live-server-amd64.iso"

2.1 GB, and my connection at the time made that a non-starter. Stuck with the cloud image and pushed forward another way.

Round 4: Breaking In Through Recovery Mode

Even with the cidata.iso attached, VirtualBox wasn't picking it up properly, so cloud-init never ran, which meant — same problem as before — no user account existed to log in with. Since I couldn't get in through the front door, I went in through the recovery console instead.

In the VirtualBox window: start the VM, hold Shift immediately as it boots (this is what gets you into the GRUB menu instead of straight into Ubuntu), then pick "Advanced options for Ubuntu" → "Recovery mode" → "root". That drops you into a root shell with the filesystem mounted read-only, so the first thing to do is make it writable:

mount -o remount,rw /

adduser k8suser
# Password: k8s123

usermod -aG sudo k8suser

sed -i 's/PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config.d/60-cloudimg-settings.conf

ssh-keygen -A

service ssh restart

That ssh-keygen -A line matters more than it looks — cloud images ship without SSH host keys generated, because normally cloud-init generates them itself on first boot as part of its regular setup routine. Since cloud-init never got to run properly on this VM, those keys were just missing, and SSH can't serve connections without them.

Even after all that, SSH from Windows still failed:

ssh -p 2222 k8suser@localhost
# kex_exchange_identification: read: Connection reset

Checked the network interface and found the real problem:

ip addr show
# enp0s3: <BROADCAST,MULTICAST> mtu 1500 ... state DOWN

The interface was just sitting there down. Again, this is cloud-init's job normally — bring up networking, request a DHCP lease, the works. With it never having run, nothing had touched the NIC. Brought it up by hand:

sudo ip link set enp0s3 up
sudo dhclient enp0s3

SSH connected right after. The takeaway here: if you're working with a cloud image outside of an actual cloud, don't assume anything cloud-init normally does for you has happened. Check the interface state before you waste time debugging SSH itself.

Building the Master Node

With a working, reachable VM, I could finally get to the actual Kubernetes setup. Container runtime first:

sudo apt-get update && sudo apt-get install -y containerd
sudo mkdir -p /etc/containerd
sudo containerd config default | sudo tee /etc/containerd/config.toml
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl restart containerd
sudo systemctl enable containerd

That SystemdCgroup = true line isn't cosmetic — kubelet and containerd both need to agree on which cgroup driver manages resource limits. If they don't match, you get subtle scheduling and resource-accounting problems down the line, so it's worth setting correctly now rather than debugging it later.

Swap has to go — kubelet flatly refuses to run properly with swap enabled, because its whole resource-accounting model assumes memory limits mean something real:

sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

Then the kernel modules and sysctl settings pods actually depend on:

sudo modprobe overlay
sudo modprobe br_netfilter

sudo tee /etc/sysctl.d/kubernetes.conf <<EOF
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF

sudo sysctl --system

overlay supports the container image layering. br_netfilter makes sure bridged traffic between pods actually gets seen by iptables rules instead of silently bypassing them. ip_forward lets the box route traffic between interfaces at all — without it, nothing about pod networking works later, no matter how good your CNI choice is.

Installing Kubernetes itself:

sudo apt-get install -y apt-transport-https ca-certificates curl gnupg

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
sudo apt-get install -y kubelet=1.29.6-1.1 kubeadm=1.29.6-1.1 kubectl=1.29.6-1.1 --allow-downgrades --allow-change-held-packages
sudo apt-mark hold kubelet kubeadm kubectl

That last apt-mark hold is deliberate — a regular apt upgrade could otherwise silently bump these to a version that doesn't match across the cluster, which is exactly the kind of thing that ruins your week later for no visible reason right now.

The Disk That Was Way Too Small

First kubeadm init attempt:

sudo kubeadm init --pod-network-cidr=192.168.0.0/16 --apiserver-advertise-address=10.0.2.15 --node-name k8s-master

It died with cannot copy extracted data... failed to write (No space left on device). Checked the disk and the root partition was sitting at a laughable 2.1 GB. This traces straight back to the cloud-init issue from earlier — cloud images ship with small root partitions on purpose, because normally cloud-init's growpart module expands the partition to match whatever disk size the cloud provider actually gave you, as part of its first-boot routine. Since cloud-init never completed properly on this VM, that resize never happened, and I was stuck with a partition sized for a tiny default disk instead of the 20 GB I'd actually allocated.

Fixing it meant leaving the guest OS entirely and working from the Windows side. VMDK format can't be resized directly, so first it had to become a VDI:

& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" clonehd "ubuntu-22.04-cloud.vmdk" "disk.vdi" --format VDI
& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifyhd "disk.vdi" --resize 20480

Then, back in recovery mode, grow the partition and filesystem to actually use that new space:

apt-get install -y cloud-guest-utils

growpart /dev/sda 1
resize2fs /dev/sda1

df -h
# /dev/sda1  20G  1.7G  18G  9% /

Moral of this one: check df -h before you install anything on a cloud image locally, not after it fails halfway through.

Finally, a Working Control Plane

sudo kubeadm init --pod-network-cidr=192.168.0.0/16 --apiserver-advertise-address=10.0.2.15 --node-name k8s-master

It went through this time. CoreDNS did throw a timeout warning during the addon phase, but that's expected at this stage — CoreDNS pods stay stuck pending until there's an actual pod network for them to run on, and I hadn't installed one yet. The control plane itself was healthy, which was the real milestone.

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

kubectl was finally talking to something real.

That's where I'll cut Part 1 — I had one working node and a control plane that actually responded to me, after what felt like an unreasonable amount of suffering to get there. Part 2 covers the part I actually set out to do: picking a CNI that would work, adding a second node, and getting a real multi-node cluster scheduling pods across both.

Debugging a Broken Ingress Setup: A Step-by-Step Walkthrough

Adeoye Malumi — Fri, 24 Jul 2026 10:05:25 +0000

This post documents a real debugging session that started with a simple symptom — example.com wouldn't load — and ended up uncovering several separate, unrelated problems layered on top of each other. It's written to walk through the actual diagnostic process, not just the final fix, since the process is usually the more useful part to learn from.

Starting point: an Ingress setup (from a previous post) was working, then stopped working after some changes were made mid-session, including installing a different CNI (Calico) on top of the cluster's existing one. This is the story of tracking that down.

The Symptom

❯ curl example.com
curl: (7) Failed to connect to example.com port 80 after 21007 ms: Could not connect to server

A connection timeout like this can mean a lot of different things: DNS pointing to the wrong place, a firewall silently dropping packets, a service that's down, or a machine that simply can't reach the target network at all. The rest of this post is the process of ruling each of those out, one at a time.

Step 1: Confirm What the Hostname Actually Resolves To

Before assuming anything about networking, the first thing to check is whether example.com is even resolving to the IP it's supposed to:

❯ getent hosts example.com
10.96.101.127   example.com

This confirmed the local /etc/hosts entry was working correctly — example.com resolved to 10.96.101.127, matching the Ingress controller's IP from earlier setup. So the problem wasn't DNS/hosts resolution. The IP itself was the next thing to check.

Step 2: Test the IP Directly, Bypassing the Hostname

If the hostname resolves correctly but the connection still fails, the next step is to test the raw IP directly — this removes DNS/hosts from the equation entirely and asks: "is this address reachable at all?"

❯ curl -v http://10.96.101.127
*   Trying 10.96.101.127:80...
* connect to 10.96.101.127 port 80 from 10.0.2.15 port 57576 failed: Connection refused
* Failed to connect to 10.96.101.127 port 80 after 21003 ms: Could not connect to server

Two useful details came out of this:

Connection refused, not a timeout. This is a meaningful difference. A timeout usually means packets are going out and nothing is answering. "Refused" happens quickly and means something did respond — but actively said "no."
The source IP was 10.0.2.15. This is VirtualBox's default address for a guest machine in NAT networking mode, which confirmed this was being tested from inside a VM.

The actual explanation: 10.96.101.127 is a Kubernetes ClusterIP. A ClusterIP isn't a real address that exists anywhere on the network — it's a virtual IP that only works because of routing rules (iptables/IPVS) that Kubernetes' kube-proxy installs inside the specific machine running the cluster. No other machine — not a VM, not a laptop on the same network — can ever reach a ClusterIP directly, because that address doesn't exist anywhere outside that one machine's own routing table.

This meant the original approach (pointing /etc/hosts at the ClusterIP) was never going to work from a separate machine, regardless of any other fix.

Step 3: Find a Real, Reachable Address Instead

Since ClusterIPs don't work across machines, the next step was finding an address that does — specifically, the IP of one of the actual Docker containers backing the kind cluster's nodes, combined with the Ingress controller's NodePort (a real port opened on the node itself, unlike a ClusterIP).

❯ docker ps
CONTAINER ID   IMAGE                  COMMAND                  CREATED      STATUS          PORTS                       NAMES
691fe58f2715   kindest/node:v1.35.0   "/usr/local/bin/entr…"   2 days ago   Up 18 minutes   127.0.0.1:38079->6443/tcp   cka-new-control-plane
aae68ca53b7b   kindest/node:v1.35.0   "/usr/local/bin/entr…"   2 days ago   Up 18 minutes                               cka-new-worker2
b51bce23b84f   kindest/node:v1.35.0   "/usr/local/bin/entr…"   2 days ago   Up 18 minutes                               cka-new-worker
75634b82514e   kindest/node:v1.35.0   "/usr/local/bin/entr…"   2 days ago   Up 18 minutes                               cka-new-worker3

None of the worker nodes had any ports published to the host — only the control-plane's Kubernetes API port was mapped. So the node's actual IP address on Docker's internal network was needed instead:

❯ docker inspect cka-new-worker | grep -A 15 '"kind"'
...
"IPAddress": "172.18.0.5",
...

A discrepancy showed up here worth noting: this IP (172.18.0.5) didn't match what kubectl get nodes -o wide had shown earlier in the session for this same node (172.18.0.3). Docker assigns container IPs dynamically, and since these containers showed Up 18 minutes (meaning they had recently restarted), the IP had simply changed. This is a good general reminder that hardcoding container IPs is fragile — they aren't guaranteed to stay the same across restarts.

Testing the NodePort against this IP, however, still failed:

❯ curl -v http://172.18.0.5:31003 -H "Host: example.com"
*   Trying 172.18.0.5:31003...
* connect to 172.18.0.5 port 31003 from 172.18.0.1 port 53748 failed: Connection timed out

So even with a real, reachable-in-theory IP and the correct port, nothing answered. This meant the problem was something deeper than just "wrong IP" — worth investigating the cluster's internal health next.

Step 4: Ruling Out the Obvious Culprits

At this point, several standard checks were run to narrow down where the actual fault was.

Was the Ingress controller pod even healthy?

❯ kubectl get pods -n ingress-nginx -o wide
NAME                                        READY   STATUS    RESTARTS      AGE   IP           NODE
ingress-nginx-controller-7d65c586d6-kvn9s   1/1     Running   3 (20m ago)   16h   10.244.1.2   cka-new-worker2

It was Running, but had restarted 3 times recently — worth remembering, but not conclusive on its own.

Was kube-proxy (the component responsible for NodePort/ClusterIP routing) healthy on every node?

❯ kubectl get pods -n kube-system -l k8s-app=kube-proxy -o wide
NAME               READY   STATUS    RESTARTS      AGE     NODE
kube-proxy-2mlbd   1/1     Running   4 (25m ago)   2d18h   cka-new-worker
kube-proxy-jb2rt   1/1     Running   4             2d18h   cka-new-worker2
kube-proxy-m2vq9   1/1     Running   4             2d18h   cka-new-control-plane
kube-proxy-rnlwc   1/1     Running   4             2d18h   cka-new-worker3

All Running, though cka-new-worker had also restarted recently.

Was there a conflict between two CNI plugins (since Calico had been installed on top of the cluster's default kindnet networking earlier)?

❯ docker exec cka-new-worker2 ls /etc/cni/net.d/
10-kindnet.conflist

Only one CNI config was present — no conflict there, at least on this node.

Did basic pod-to-pod networking inside the cluster even work at all?

❯ kubectl run curl --rm -it --image=curlimages/curl -- curl http://hello-world
curl: (6) Could not resolve host: hello-world

This was the first real breakthrough. This same command had worked earlier in the setup process — now it couldn't even resolve the Service's name. This shifted the investigation toward DNS specifically.

Step 5: Investigating the DNS Failure

First, check whether restarting kube-proxy (to clear out any stale routing rules) fixes it:

❯ kubectl delete pods -n kube-system -l k8s-app=kube-proxy
pod "kube-proxy-2mlbd" deleted from kube-system namespace
pod "kube-proxy-jb2rt" deleted from kube-system namespace
pod "kube-proxy-m2vq9" deleted from kube-system namespace
pod "kube-proxy-rnlwc" deleted from kube-system namespace

After waiting for the new pods to come back Running:

❯ kubectl run curl --rm -it --image=curlimages/curl -- curl http://hello-world
curl: (6) Could not resolve host: hello-world

No change. This ruled out stale kube-proxy rules as the cause.

Check whether the cluster's internal DNS service (CoreDNS) has healthy endpoints behind it:

❯ kubectl get endpoints -n kube-system kube-dns
NAME       ENDPOINTS                                                   AGE
kube-dns   10.244.0.2:9153,10.244.0.3:9153,10.244.0.2:53 + 3 more...   2d18h

This looked healthy — CoreDNS pods were properly registered.

Check whether a real pod is even configured to use the right DNS server:

❯ kubectl exec -it hello-world-c8b6f8dbd-9t9n6 -- sh
# cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5

This was also correct — 10.96.0.10 is the standard internal DNS Service IP.

So far: correct DNS configuration, healthy-looking CoreDNS, no stale kube-proxy rules — yet DNS lookups still failed. The next step was testing whether this was actually a DNS-specific issue, or something more fundamental with networking between nodes.

Step 6: Isolating DNS vs. General Networking

Test a plain HTTP connection to a Service, bypassing DNS entirely (using the raw ClusterIP):

# python3 -c "import urllib.request; print(urllib.request.urlopen('http://10.96.74.127').read())"
b'Hello, World!'

This worked. So general Service routing (HTTP, TCP) was fine — the problem was specific to DNS.

Test raw connectivity to the DNS port itself (port 53), both against the Service IP and a CoreDNS pod's IP directly:

# python3 -c "
import socket
try:
    s = socket.create_connection(('10.96.0.10', 53), timeout=3)
    print('connected')
    s.close()
except Exception as e:
    print('failed:', e)
"
failed: timed out

# python3 -c "
import socket
try:
    s = socket.create_connection(('10.244.0.2', 53), timeout=3)
    print('connected')
    s.close()
except Exception as e:
    print('failed:', e)
"
failed: timed out

Both failed — even a direct connection attempt to a CoreDNS pod's own IP, with no DNS resolution or Service routing involved at all, still timed out.

The key detail that explained this: checking which node each pod involved was running on showed that the working test (10.96.74.127, resolving to the hello-world pod) happened to be a pod on the same node the test was run from. The failing tests (both CoreDNS IPs) were both on the control-plane node — a different node entirely. This pointed toward a cross-node networking problem, not a DNS-specific one.

Step 7: Confirming Cross-Node Networking Was Broken

Checked whether the Ingress controller's NodePort Service had a networking policy that might explain uneven behavior across nodes:

❯ kubectl get svc -n ingress-nginx ingress-nginx-controller -o jsonpath='{.spec.externalTrafficPolicy}'
Local

This turned out to be a real, separate finding: externalTrafficPolicy: Local means kube-proxy will only forward NodePort traffic on a node if that node happens to be running a copy of the target pod locally — every other node will actively drop the traffic rather than forward it elsewhere. This fully explained the earlier NodePort timeout, since the Ingress controller pod only ran on cka-new-worker2, and the test had hit a different node.

Confirmed by checking iptables on the failing node:

❯ docker exec cka-new-worker3 iptables-save | grep -i "53\|drop" | head -30
-A KUBE-EXTERNAL-SERVICES -p tcp -m comment --comment "ingress-nginx/ingress-nginx-controller:http has no local endpoints" -m addrtype --dst-type LOCAL -m tcp --dport 31003 -j DROP

This line confirms it directly: any NodePort traffic to port 31003 on this node is explicitly dropped, because this node has no local copy of the Ingress controller pod.

Fixing the NodePort routing issue was straightforward — hit the correct node directly:

❯ curl -v http://172.18.0.4:31003 -H "Host: example.com"
* Connected to 172.18.0.4 (172.18.0.4) port 31003
< HTTP/1.1 504 Gateway Time-out

The connection succeeded this time — but nginx itself returned a 504 Gateway Time-out. This meant the request reached the Ingress controller fine, but the Ingress controller couldn't reach its own backend (the hello-world Service). And, notably, the Ingress controller pod and the hello-world pod were running on two different nodes — another cross-node hop, timing out in exactly the same pattern as the DNS failures.

This was the real conclusion: same-node pod-to-pod traffic worked consistently throughout this session. Cross-node pod-to-pod traffic (DNS lookups to CoreDNS, and now the Ingress controller reaching its backend) consistently failed. Along the way, other explanations were ruled out — no competing CNI configs, no blocking NetworkPolicy, correctly configured DNS, correct iptables NAT rules, and Calico's own control-plane pods weren't even running:

❯ kubectl get pods -n calico-system -o wide
No resources found in calico-system namespace.

The most likely explanation: installing Calico mid-way through the cluster's life (on top of the existing kindnet CNI, without removing it) had left the routes between nodes in a broken, inconsistent state — even after Calico itself was no longer running. Untangling that kind of mixed-CNI networking state by hand is generally not worth the time; the standard, reliable fix is to rebuild the cluster from scratch.

Step 8: Rebuilding the Cluster

❯ kind delete cluster --name cka-new

To make future testing easier, a kind config file was created to forward the Ingress controller's expected ports straight from the container to localhost on the host machine — removing the need to look up container IPs at all going forward:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: cka-new
nodes:
  - role: control-plane
    extraPortMappings:
      - containerPort: 31003
        hostPort: 31003
        protocol: TCP
      - containerPort: 30437
        hostPort: 30437
        protocol: TCP
  - role: worker
  - role: worker
  - role: worker

❯ kind create cluster --name cka-new --config kind-config.yaml

This time, Calico was intentionally left out — the default kindnet CNI had been working fine before Calico was introduced mid-session, and nothing about the setup actually required Calico's features.

Reapplying the application and Ingress manifests:

❯ kubectl apply -f deployment.yaml -f service.yaml -f ingress.yaml
❯ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.15.1/deploy/static/provider/aws/deploy.yaml

Step 9: A Slow Pod Isn't Always a Stuck Pod

While waiting for the Ingress controller to come up, it sat in ContainerCreating for several minutes, which raised the question of whether something was actually wrong:

❯ kubectl get pods -n ingress-nginx -w
ingress-nginx-controller-7d65c586d6-fz54r   0/1   ContainerCreating   0   2m59s

Checking the pod's events showed the actual cause:

❯ kubectl describe pod -n ingress-nginx ingress-nginx-controller-7d65c586d6-fz54r
...
Events:
  Warning  FailedMount  4m41s (x8 over 5m45s)  kubelet  MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found
  Normal   Pulling      3m30s                  kubelet  Pulling image "registry.k8s.io/ingress-nginx/controller:v1.15.1@sha256:..."

This looked alarming at first, but both issues were self-resolving, expected steps of the install process rather than actual problems:

The FailedMount warnings happened because two setup Jobs (ingress-nginx-admission-create and ingress-nginx-admission-patch) needed to finish first — they're responsible for generating the certificate secret the controller pod was waiting to mount. Until those Jobs completed, that secret simply didn't exist yet.
The image pull itself was just slow — checking kubectl get events showed the two setup Jobs' images had each taken over a minute to pull, which suggested a slow registry pull rather than anything broken locally.

Waiting it out confirmed this:

❯ kubectl get pods -n ingress-nginx
ingress-nginx-controller-7d65c586d6-fz54r   1/1   Running   0   7m34s

Lesson here: not every ContainerCreating or warning event means something is broken — checking the actual Events section (via kubectl describe pod) before assuming failure saves a lot of unnecessary troubleshooting.

Step 10: Fixing the Service Configuration Correctly

With the controller running, the externalTrafficPolicy and Service type still needed to be set correctly, to avoid re-hitting the earlier Local-policy and LoadBalancer-pending issues.

The first attempt patched only externalTrafficPolicy:

❯ kubectl patch svc ingress-nginx-controller -n ingress-nginx -p '{"spec":{"externalTrafficPolicy":"Cluster"}}'
❯ kubectl get svc -n ingress-nginx ingress-nginx-controller
NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)
ingress-nginx-controller   LoadBalancer   10.96.61.252   <pending>     80:32087/TCP,443:32290/TCP

This revealed two remaining issues: the Service was still type: LoadBalancer (stuck <pending>, same problem as before), and the NodePorts had landed on different numbers (32087/32290) than what kind-config.yaml was forwarding (31003/30437).

Fixing both at once, by explicitly setting the type to NodePort and pinning the exact port numbers:

❯ kubectl patch svc ingress-nginx-controller -n ingress-nginx -p '{"spec":{"type":"NodePort","ports":[{"name":"http","port":80,"targetPort":"http","protocol":"TCP","nodePort":31003},{"name":"https","port":443,"targetPort":"https","protocol":"TCP","nodePort":30437}]}}'

❯ kubectl get svc -n ingress-nginx ingress-nginx-controller
NAME                       TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)
ingress-nginx-controller   NodePort   10.96.61.252   <none>        80:31003/TCP,443:30437/TCP

This matched the port forwarding configured in kind-config.yaml.

Step 11: Confirming Everything Works

❯ curl -v http://localhost:31003 -H "Host: example.com"
*   Trying [::1]:31003...
* connect to ::1 port 31003 from ::1 port 49216 failed: Connection refused
*   Trying 127.0.0.1:31003...
* Connected to localhost (127.0.0.1) port 31003
< HTTP/1.1 200 OK
Hello, World!

This confirmed everything was working correctly: the port forwarding in kind-config.yaml, the Cluster traffic policy correctly routing regardless of which node the request landed on, and the Ingress controller successfully reaching the hello-world backend (no more 504 Gateway Time-out).

The ::1 (IPv6) connection attempt failing before falling back to IPv4 is expected — kind-config.yaml only forwards the port over IPv4, so curl's dual-stack attempt simply skips it and retries over IPv4 instead. Not an error to fix.

Final step — update /etc/hosts, this time pointing at 127.0.0.1 instead of any container IP, since port forwarding now handles the routing:

127.0.0.1   example.com

❯ curl http://example.com:31003
Hello, World!

Why the Port Still Has to Be Specified

Running curl http://example.com without a port defaults to port 80 — but nothing is mapped to host port 80 in this setup, only 31003 and 30437. Without a port, the connection fails immediately:

curl: (7) Failed to connect to example.com port 80 after ...: Connection refused

This is expected: the Ingress controller is exposed via a NodePort Service, and NodePorts are, by design, high-numbered ports (typically in the 30000–32767 range) — they were never going to answer on plain port 80 without an explicit mapping.

To make curl http://example.com work without a port number, kind-config.yaml could instead map host port 80 directly to the NodePort:

extraPortMappings:
  - containerPort: 31003
    hostPort: 80
    protocol: TCP

This requires a cluster rebuild to take effect, and on Linux, binding to port 80 may require elevated (sudo) privileges since ports below 1024 are privileged. For local testing and practice environments, simply including the NodePort in the URL is generally the more common and simpler approach — production clusters solve the "no port number" problem differently anyway, using a real external load balancer or a bare-metal solution like MetalLB in front of the cluster.

Setting Up Ingress for a Flask App

Adeoye Malumi — Fri, 24 Jul 2026 08:53:20 +0000

Today's goal: take a simple Flask app running in Kubernetes and expose it to the outside world using Ingress, instead of relying on NodePort or LoadBalancer services. This post walks through the setup, the errors I hit along the way, and how each one was resolved.

Why Use Ingress Instead of NodePort or LoadBalancer?

Kubernetes gives you a few ways to expose a Service outside the cluster:

NodePort — opens a fixed port (usually in the 30000–32767 range) on every node. Works, but the port numbers are ugly and it doesn't scale well for multiple applications.
LoadBalancer — provisions a cloud load balancer per Service. This has real drawbacks:
- Cost — each LoadBalancer Service typically means a new cloud load balancer, billed separately.
- Cloud lock-in — it depends on your cloud provider's integration (AWS, GCP, Azure, etc.).
- No advanced routing — no path-based or host-based routing, and no built-in SSL termination.

Ingress solves this with two separate pieces:

Ingress Resource — a YAML manifest where you declare routing rules (e.g., "requests for example.com/ should go to the hello-world Service").
Ingress Controller — the component that actually watches the API server for Ingress Resources and configures a real proxy (in this case, NGINX) to implement those rules.

You write the rules; the controller enforces them. Without a running Ingress Controller, an Ingress Resource does nothing on its own.

Step 0: Building and Pushing the Docker Image

Before any of the Kubernetes manifests would work, the Flask app first had to be containerized and pushed to a registry, Kubernetes only pulls and runs images, not raw source code. This step happens before everything else, even though it isn't captured in the terminal history below.

app.py — a minimal Flask app:

from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello, World!'

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=80)

Dockerfile:

# Use the official image as a parent image
FROM python:3.7-slim

# Set the working directory in the container
WORKDIR /app

# Copy the dependencies file to the working directory
COPY requirements.txt .

# Install any needed packages specified in requirements.txt
RUN pip install -r requirements.txt

# Copy the content of the local src directory to the working directory
COPY . .

# Run app.py when the container launches
CMD ["python", "app.py"]

requirements.txt:

Flask==2.2.5

This image is built and pushed to Docker Hub as oyebobs/ingress-demo:v1 — the same tag referenced in deployment.yaml below. If this step is skipped, kubectl will still accept the Deployment manifest, but the Pod will get stuck in ImagePullBackOff because there's nothing to pull.

Step 1: The Kubernetes Manifests

Three manifests, each with a distinct job.

deployment.yaml — defines the Pod and keeps it running:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world
  labels:
    app: hello-world
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-world
  template:
    metadata:
      labels:
        app: hello-world
    spec:
      containers:
      - name: hello-world
        image: oyebobs/ingress-demo:v1
        ports:
        - containerPort: 80

service.yaml — gives the Pod(s) a stable internal address:

apiVersion: v1
kind: Service
metadata:
  name: hello-world
spec:
  selector:
    app: hello-world
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

ingress.yaml — defines the routing rule for external traffic:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-world
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
  - host: "example.com"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: hello-world
            port:
              number: 80

The ingressClassName: nginx field matters: in a cluster with more than one Ingress Controller installed, this tells Kubernetes which controller should handle this particular Ingress Resource.

After applying the Deployment and Service, both looked healthy:

❯ k get po
NAME                          READY   STATUS    RESTARTS      AGE
curl-test                     1/1     Running   1 (44s ago)   3m43s
hello-world-c8b6f8dbd-9t9n6   1/1     Running   1             33m

❯ k get svc
NAME          TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
hello-world   ClusterIP   10.96.74.127   <none>        80/TCP    26m

Step 2: Debugging Internal Connectivity

A curl-test Pod was already running to test connectivity to the hello-world Service. The first attempt failed:

❯ k exec -it curl-test -- curl 10.96.74.127
command terminated with exit code 137

Exit code 137 means the container was OOM-killed (killed for exceeding its memory limit) — this is not a networking error. Running kubectl describe pod curl-test confirmed repeated restarts with Last State: Terminated, Reason: Error.

To check whether the app itself was actually working, I shelled directly into the hello-world Pod:

❯ kubectl exec -it hello-world-c8b6f8dbd-9t9n6 -- curl localhost:80
error: ... exec: "curl": executable file not found in $PATH

The python:3.7-slim base image doesn't include curl — "slim" images intentionally leave out extras like this. As a workaround, I used Python's built-in urllib module instead:

# python3 -c "import urllib.request; print(urllib.request.urlopen('http://localhost:80').read())"
b'Hello, World!'

This confirmed the app itself was working correctly inside its own Pod. The issue was isolated to the curl-test Pod repeatedly getting OOM-killed, not the app or the network.

Fix: rather than continuing to debug a resource-constrained, long-running Pod, I switched to a disposable Pod using kubectl run --rm -it, which creates a temporary Pod, opens a shell, and deletes the Pod automatically on exit:

❯ kubectl run curl-test --image=curlimages/curl -it --rm -- curl -v 10.244.3.2:80
...
< HTTP/1.1 200 OK
< Server: Werkzeug/2.2.3 Python/3.7.17
...
Hello, World!

The same worked against the Service's ClusterIP directly:

❯ kubectl run curl-test --image=curlimages/curl -it --rm -- curl -v 10.96.74.127:80
< HTTP/1.1 200 OK
Hello, World!

Takeaway: internal networking and the app were fine the entire time. The failures were caused by one Pod being killed for memory usage, not by a cluster-wide issue. When only one Pod misbehaves and everything else works, the Pod itself is usually the place to look first.

Step 3: Applying the Ingress — Address Stays Empty

With internal connectivity confirmed, I applied the Ingress resource:

❯ k apply -f ingress.yaml
ingress.networking.k8s.io/hello-world created

❯ k get ingress
NAME          CLASS   HOSTS         ADDRESS   PORTS   AGE
hello-world   nginx   example.com             80      8s

The ADDRESS column stayed empty. This is expected at this stage: the Ingress Resource exists, but no Ingress Controller is running yet to act on it.

Step 4: Installing the NGINX Ingress Controller

❯ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.15.1/deploy/static/provider/aws/deploy.yaml
namespace/ingress-nginx created
...
deployment.apps/ingress-nginx-controller created
...
ingressclass.networking.k8s.io/nginx created

This manifest creates a dedicated namespace, RBAC rules, the controller Deployment, and admission webhooks. The controller Pod came up successfully:

❯ k get po -A | grep nginx
ingress-nginx   ingress-nginx-controller-7d65c586d6-kvn9s   1/1   Running   0   7m24s

Its logs confirmed it detected the Ingress resource and reloaded its configuration:

I0723 16:19:18.803001 ... "Found valid IngressClass" ingress="default/hello-world" ingressclass="nginx"
I0723 16:19:19.839235 ... "Backend successfully reloaded"

The controller was running and had picked up the routing rule. The next question was why the Ingress still had no address.

Step 5: LoadBalancer Stuck on `<pending>`

❯ k get svc -n ingress-nginx
NAME                        TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)
ingress-nginx-controller    LoadBalancer   10.96.101.127   <pending>     80:31003/TCP,443:30437/TCP

EXTERNAL-IP stayed on <pending> indefinitely. This happens because a type: LoadBalancer Service depends on a Cloud Controller Manager to provision an actual external load balancer and assign it an IP. On a self-managed kind/kubeadm cluster with no cloud provider integration, there's nothing available to fulfill that request, so it never resolves.

Fix: change the Service type to NodePort, which doesn't require any cloud integration:

❯ k edit svc ingress-nginx-controller -n ingress-nginx
service/ingress-nginx-controller edited

❯ k get svc -n ingress-nginx
NAME                        TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)
ingress-nginx-controller    NodePort   10.96.101.127   <none>        80:31003/TCP,443:30437/TCP

After deleting and re-applying ingress.yaml, the ADDRESS field populated:

❯ k get ing -w
NAME          CLASS   HOSTS         ADDRESS         PORTS   AGE
hello-world   nginx   example.com   10.96.101.127   80      44s

Testing directly from the local machine, however, still failed:

❯ curl 10.96.101.127
curl: (7) Failed to connect to 10.96.101.127 port 80 after 21024 ms: Could not connect to server

Step 6: Investigating the CNI (Calico Installation)

While troubleshooting the connection timeout, I also installed Calico as the cluster's CNI (Container Network Interface) plugin, replacing the default kindnet networking that kind clusters ship with. Calico is a common choice in real-world clusters because of its more detailed network policy support and easier debugging tools.

❯ kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.30.3/manifests/tigera-operator.yaml
namespace/tigera-operator created
...
deployment.apps/tigera-operator created

❯ kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.30.3/manifests/custom-resources.yaml
resource mapping not found for name: "default" ... no matches for kind "Installation" in version "operator.tigera.io/v1"
ensure CRDs are installed first

This error occurs when the Calico custom resources are applied before the tigera-operator has finished installing its Custom Resource Definitions (CRDs) — the CRDs need to exist before Kubernetes can accept objects of that kind. Fix: wait for the operator Pod to reach Running, then re-apply:

❯ kubectl get pods -n tigera-operator -w
tigera-operator-75c7c596d9-tsmbz   1/1   Running   0   2m4s

❯ kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.30.3/manifests/custom-resources.yaml
installation.operator.tigera.io/default created
apiserver.operator.tigera.io/default created
...

Calico's components came up correctly:

❯ k get po -A -n calico-system | grep calico
calico-apiserver   calico-apiserver-64cf87458d-qrwnb   1/1   Running   0   3m58s
calico-apiserver   calico-apiserver-64cf87458d-tqw44   1/1   Running   0   3m58s

Even with Calico running, curling the Ingress IP from the local machine still timed out:

❯ curl 10.96.101.127
curl: (7) Failed to connect to 10.96.101.127 port 80 after 21007 ms: Could not connect to server

Important clarification: 10.96.101.127 is a ClusterIP, which is only routable inside the cluster's network — it is not reachable from an external machine like a laptop, regardless of which CNI is installed. Swapping the CNI plugin did not (and could not) resolve this particular timeout, since the address itself is not meant to be reachable externally. The correct way to test was to send traffic from inside the cluster, or through the exposed NodePort.

Step 7: Testing From Inside the Cluster

To test correctly, I ran a temporary Pod inside the cluster and issued requests from there, which reflects how real traffic would actually reach the Service:

❯ kubectl run curl --rm -it --image=curlimages/curl -- sh
~ $ curl http://hello-world
Hello, World!

This confirmed internal DNS-based service discovery was working. Next, hitting the Ingress Service's ClusterIP directly by IP, without specifying a hostname:

~ $ curl http://10.96.101.127
<html>
<head><title>404 Not Found</title></head>
<body><center><h1>404 Not Found</h1></center>
<hr><center>nginx</center></body>
</html>

This 404 is expected behavior, not an error. The Ingress rule only matches requests where the Host header is example.com. Curling the raw IP sends no Host header, so NGINX has no matching rule and returns a 404.

To simulate a request with the correct hostname, I used curl's --resolve flag to manually map example.com to the Ingress IP. The first attempt used the wrong syntax:

~ $ curl example.com --resolve example.com:80 http://10.96.101.127
curl: (49) Could not parse CURLOPT_RESOLVE entry 'example.com:80'

--resolve expects the format host:port:ip, not a separate URL. Corrected:

~ $ curl example.com --resolve example.com:80:10.96.101.127
Hello, World!

~ $ curl http://example.com
Hello, World!

This confirmed host-based routing was working as configured: the same Ingress IP serves different responses depending on the requested hostname — the core capability that distinguishes Ingress from a plain NodePort or LoadBalancer setup.

Step 8: Mapping example.com in /etc/hosts

--resolve is convenient for one-off curl requests, but it only applies to that single command — it doesn't help if you want to open http://example.com in a browser or run multiple tools against the same hostname. For that, the standard approach is to edit /etc/hosts directly, which tells the local machine to resolve example.com to the Ingress IP for every request, not just one curl call.

❯ sudo nano /etc/hosts

Inside the file, a line was added mapping the hostname example.com to the Ingress controller's IP address:

10.96.101.127   example.com

After saving the file, any tool on the machine — curl, a browser, etc. — resolves example.com to that IP without needing --resolve or any other flag:

❯ curl http://example.com
Hello, World!

This simulates what a real DNS record would do in production: pointing a domain name at the Ingress controller's IP. In production, you'd create an actual DNS A record for example.com pointing at the Ingress controller's external IP; /etc/hosts is the local-machine equivalent for testing without needing a real, registered domain.

One thing worth noting: this entry only affects DNS resolution on the local machine — it doesn't change how anyone else resolves the domain. It's also worth removing once testing is done, so it doesn't cause confusion later if the Ingress IP changes.

Understanding PersistentVolumes (PV) and PersistentVolumeClaims (PVC)

Adeoye Malumi — Thu, 16 Jul 2026 16:23:52 +0000

Today I learned one of the most confusing Kubernetes concepts for beginners: PersistentVolumes (PV) and PersistentVolumeClaims (PVC).

At first, my PVC refused to bind and stayed in the Pending state. After some troubleshooting, I finally got everything working. And honestly? Breaking it first helped me understand it much better.

Why do we even need PVs?

Imagine you're living in a rented apartment.

Pod = You.
Container filesystem = Your temporary room.
PersistentVolume (PV) = A storage unit that exists independently of you.
PersistentVolumeClaim (PVC) = The rental agreement saying, "I'd like to use a storage unit."

If your Pod dies, everything inside the container normally disappears.

A PV lets your data survive even when the Pod is deleted.

The Relationship

Pod
 │
 ▼
PersistentVolumeClaim (PVC)
 │
 ▼
PersistentVolume (PV)
 │
 ▼
Actual storage (hostPath, NFS, cloud disk, etc.)

The Pod never talks directly to the PV.

Instead it says:

"Hey PVC, can I borrow some storage?"

The PVC then finds a suitable PV.

Step 1 — Create a PersistentVolume

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-demo
  labels:
    type: local
spec:
  storageClassName: ""
  capacity:
    storage: 512Mi
  accessModes:
    - ReadWriteMany
  hostPath:
    path: "/data/config"

This creates 512Mi of storage backed by a directory on the node.

Step 2 — Create a PersistentVolumeClaim

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-demo
spec:
  storageClassName: ""
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 256Mi

Notice that the claim only requests 256Mi.

Kubernetes looks for a PV that satisfies the request and binds them together.

My First Roadblock

After creating the PVC:

kubectl apply -f pvc.yaml
kubectl get pvc

I saw:

STATUS
Pending

Why Was My PVC Stuck in `Pending`?

This was the part that confused me the most.

I had already created my PersistentVolume (PV), so I expected my PersistentVolumeClaim (PVC) to bind to it immediately. Instead, every time I checked, the PVC was still in the Pending state.

kubectl get pvc

Output:

NAME       STATUS
pvc-demo   Pending

To investigate, I ran:

kubectl describe pvc pvc-demo

The Events section showed:

FailedBinding
no persistent volumes available for this claim and no storage class is set

At first, I thought Kubernetes couldn't see my PersistentVolume at all. That wasn't actually the problem.

The issue was that my PVC and PV didn't agree on the access mode.

My PVC requested:

accessModes:
  - ReadWriteOnce

while my PV was configured with:

accessModes:
  - ReadWriteMany

Because of this difference, Kubernetes couldn't match the claim to the volume, so the PVC stayed in the Pending state.

After deleting the old resources, correcting the configuration, and recreating both the PV and PVC, Kubernetes successfully matched them and their status changed to Bound.

The biggest lesson I learned is that whenever a PVC is stuck in Pending, the first command to run should be:

kubectl describe pvc pvc-demo

The Events section usually contains the clue you need to figure out why the claim couldn't be bound.

Step 3 — Mount the PVC inside a Pod

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: pv-container
      image: nginx:latest
      volumeMounts:
        - name: pv-storage
          mountPath: /var/app/config
      ports:
        - containerPort: 80
          name: http-server
  volumes:
    - name: pv-storage
      persistentVolumeClaim:
        claimName: pvc-demo

Think of this as plugging an external hard drive into your laptop.

The Pod can now read and write files through the PVC.

Step 4 — Test It

kubectl exec -it app -- bash

Inside the container:

cd /var/app/config

echo "This is a test" > test.html

cat test.html

Output:

This is a test

Success!

I accidentally tried:

curl test.html

which failed because curl fetches URLs, not local files.

Lesson learned.

The Big Idea

Many beginners confuse PVs and PVCs.

Remember this:

PersistentVolume (PV) = The actual storage.
PersistentVolumeClaim (PVC) = The request for storage.
Pod = The application that uses the claim.

A simple analogy:

🍕 PV = The pizza.
🧾 PVC = The order receipt.
😋 Pod = The hungry person.

You don't grab the pizza straight from the oven—you present your receipt first!

Commands I Used

kubectl apply -f pv.yaml
kubectl get pv

kubectl apply -f pvc.yaml
kubectl get pvc
kubectl describe pvc pvc-demo

kubectl apply -f pod.yaml
kubectl get pods

kubectl exec -it app -- bash

What I Learned

Persistent storage survives Pod restarts.
Pods use PVCs, not PVs, directly.
A PVC must successfully bind before a Pod can use it.
kubectl describe is one of the best debugging tools when something is stuck.

Kubernetes Network Policies Explained (and How I Actually Made Them Work)

Adeoye Malumi — Thu, 09 Jul 2026 16:50:22 +0000

If you've ever wondered, "How do I stop my pods from randomly talking to everything in my cluster?" – this is the post for you.

Today, I dove into Kubernetes Network Policies – the way you control who can talk to whom inside your cluster. I started with weavenet, then switched to Calico, and I'm excited to walk you through how it all came together.

What Even Is a Network Policy?

A NetworkPolicy in Kubernetes is like a firewall rule for your pods. It lets you:

Control inbound traffic (who can talk to a pod)
Control outbound traffic (where a pod can talk to)

For example:

You can create a deny-all policy that blocks all incoming traffic to a pod.
Or an allow-only policy that says: "Only the backend pods can talk to the mysql pod on port 3306."

Without network policies, by default, everything can talk to everything in your cluster. That's fine for demos, terrible for real security.

Why I Needed a Different CNI

Kubernetes itself doesn't implement network policies. It just defines them. The actual work is done by your CNI plugin (Container Network Interface).

I started with weavenet, which supports basic network policies. But as I dug deeper, I realized I wanted something more powerful and production-friendly. That's when I switched to Calico, which is:

Fast
Scalable
Great for complex network policies
Works well with kind clusters (as long as you configure it correctly)

If you're just experimenting, weavenet is fine. If you want to level up your cluster networking, Calico is a great next step.

My Test Cluster: A Kind Setup with Calico

To test network policies, I created a simple kind cluster with:

1 control-plane node
2 worker nodes
Disabled default CNI (disableDefaultCNI: true)
Custom pod subnet: 192.168.0.0/16

kind Cluster YAML

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    extraPortMappings:
      - containerPort: 30001
        hostPort: 30001
  - role: worker
  - role: worker
networking:
  disableDefaultCNI: true
  podSubnet: "192.168.0.0/16"

Creating the Cluster

With the YAML saved as kind-config.yaml, I created the cluster like this:

kind create cluster --name day26 --config kind-config.yaml

Then I checked the nodes:

kubectl get nodes

And here's the part that was a bit confusing at first:

Until I installed a CNI (like Calico), the worker nodes stayed in a NotReady state.

That's because Kubernetes needs a CNI to:

Assign IPs to pods
Set up networking between nodes
Enable traffic to flow across the cluster

Without a CNI, the control-plane node can be Ready, but the worker nodes will remain NotReady until networking is configured.

Only after installing Calico did all nodes show as Ready:

kubectl get nodes
# NAME           STATUS   ROLES           AGE   VERSION
# day26-control-plane   Ready    control-plane   ...
# day26-worker          Ready    worker          ...
# day26-worker2         Ready    worker          ...

Once Calico was running, my cluster was ready for real network policy experiments.

The App I Built: Frontend → Backend → MySQL

To make network policies meaningful, I created a tiny 3-tier app:

frontend pod + service
backend pod + service
mysql pod + service (db service)

Pods & Services

apiVersion: v1
kind: Pod
metadata:
  name: frontend
  labels:
    role: frontend
spec:
  containers:
    - name: nginx
      image: nginx
      ports:
        - name: http
          containerPort: 80
          protocol: TCP
***
apiVersion: v1
kind: Service
metadata:
  name: frontend
  labels:
    role: frontend
spec:
  selector:
    role: frontend
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
***
apiVersion: v1
kind: Pod
metadata:
  name: backend
  labels:
    role: backend
spec:
  containers:
    - name: nginx
      image: nginx
      ports:
        - name: http
          containerPort: 80
          protocol: TCP
***
apiVersion: v1
kind: Service
metadata:
  name: backend
  labels:
    role: backend
spec:
  selector:
    role: backend
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
***
apiVersion: v1
kind: Service
metadata:
  name: db
  labels:
    name: mysql
spec:
  selector:
    name: mysql
  ports:
    - protocol: TCP
      port: 3306
      targetPort: 3306
***
apiVersion: v1
kind: Pod
metadata:
  name: mysql
  labels:
    name: mysql
spec:
  containers:
    - name: mysql
      image: mysql:latest
      env:
        - name: "MYSQL_USER"
          value: "mysql"
        - name: "MYSQL_PASSWORD"
          value: "mysql"
        - name: "MYSQL_DATABASE"
          value: "testdb"
        - name: "MYSQL_ROOT_PASSWORD"
          value: "verysecure"
      ports:
        - name: http
          containerPort: 3306
          protocol: TCP

At this point, everything could talk to everything:

frontend could reach backend
backend could reach mysql
frontend could also reach mysql directly (which we probably don't want)

Now it was time to lock it down.

My First Real Network Policy: Only Backend → MySQL

The goal:

mysql should only accept traffic from pods with label role: backend
On port 3306
No one else (including frontend) should touch the database

NetworkPolicy Manifest

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: db-test
spec:
  podSelector:
    matchLabels:
      name: mysql
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              role: backend
      ports:
        - port: 3306

Key parts:

podSelector: targets the mysql pod (name: mysql)
policyTypes: [Ingress]: we're controlling inbound traffic
ingress.from.podSelector: only pods with role: backend are allowed
ports: only port 3306

After applying this:

backend → mysql works
frontend → mysql blocked
mysql → anywhere (outbound) still allowed (we didn't restrict Egress)

How I Tested It

I used a simple approach:

Run a temporary pod with curl or mysql client.
Try to reach services from different pods.
Observe which connections succeed or fail.

Example:

# From frontend pod – should fail
kubectl exec -it frontend -- curl -s http://db:3306

# From backend pod – should succeed
kubectl exec -it backend -- curl -s http://db:3306

Seeing the first one fail and the second one succeed was the "aha!" moment: the policy is actually working.

Wrapping Up

Network Policies are one of those things that feel abstract until you actually apply them and see connections blocked in real time. Once that happens, you suddenly understand:

Why security matters
Why "everything can talk to everything" is not a strategy
How Kubernetes gives you granular control over your cluster's traffic

Understanding Kubernetes Service Accounts (The Identity Every Pod Uses)

Adeoye Malumi — Tue, 07 Jul 2026 15:58:47 +0000

Most of us learn about Kubernetes users before we learn about Service Accounts. That's actually where the confusion starts Service Accounts end up looking like "just another kind of user," and you file them away without really getting what makes them different.

They're not just another user. They solve a completely different problem.

Today I learned that human users authenticate from outside the cluster — you, sitting at your laptop, running kubectl with your certificate or your login. But Pods run inside the cluster, and they often need to talk to the Kubernetes API too — to list other pods, read a ConfigMap, watch for changes, whatever. A Pod can't exactly type in a username and password. It needs its own identity.

That identity is called a Service Account.

User vs. Service Account

Before touching a single command, it's worth seeing the two side by side:

User	Service Account
Used by humans	Used by Pods
External to Kubernetes	Managed by Kubernetes
Usually authenticated using certificates	Authenticated using tokens
You create it outside the cluster	Kubernetes creates and manages it

Keep that table in your head — everything below is really just this table playing out in the terminal.

Step 1 — Checking Existing Service Accounts

❯ k get sa
NAME      AGE
default   38d

Notice I didn't create this. It was already there, 38 days old — as old as the namespace itself.

That's the first thing to internalize: every namespace automatically gets a Service Account called default. You don't ask for it. If you create a Pod and never say which Service Account it should use, Kubernetes quietly attaches this one for you. Keep that in mind — it comes back to bite (in a good way) later in this post.

Step 2 — Creating My Own Service Account

❯ k create sa build-sa
serviceaccount/build-sa created

❯ k describe sa build-sa
Name:                build-sa
Namespace:           default
Labels:              <none>
Annotations:         <none>
Image pull secrets:  <none>
Events:              <none>

Simple enough — I now have a second identity, build-sa, sitting in the default namespace, ready to be attached to Pods later.

But here's the question that trips a lot of beginners up: why doesn't it have a token?

If you learned Kubernetes a few years ago, or you're reading an older tutorial, you'll expect describe sa to show a Tokens: field pointing at an auto-generated Secret. Not anymore. Newer versions of Kubernetes stopped automatically minting long-lived Secret-based tokens for every Service Account, mainly for security reasons — a token that never expires and just sits around as a Secret is a standing risk if it leaks. Instead, Kubernetes now prefers short-lived, automatically-rotated tokens issued on demand.

Since I specifically wanted a durable token I could inspect and use for --as testing, I had to create it myself.

Step 3 — Creating a ServiceAccount Token Secret

I wrote a secret.yaml that, at minimum, needs this annotation:

metadata:
  annotations:
    kubernetes.io/service-account.name: build-sa

This one line is doing all the work. It tells the Kubernetes controller: "This Secret isn't just a random Opaque blob — it's a token for the build-sa Service Account." Because of that annotation, a built-in controller notices the Secret, generates a signed token for build-sa, and populates the Secret with it automatically.

❯ k apply -f secret.yaml
secret/build-robot-secret created

❯ k get secret
NAME                 TYPE                                  DATA   AGE
backend-user         Opaque                                1      5d3h
build-robot-secret   kubernetes.io/service-account-token   3      14s

See the TYPE column — kubernetes.io/service-account-token, not Opaque. That type is exactly what tells Kubernetes "treat this Secret specially and fill it with SA credentials."

Inspecting the Secret

❯ k describe secret build-robot-secret
Name:         build-robot-secret
Namespace:    default
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: build-sa
              kubernetes.io/service-account.uid: d065d358-58fa-44a6-9656-8263e0fa05eb

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1107 bytes
namespace:  7 bytes
token:      eyJhbGciOiJSUzI1NiIs... (truncated)

Three fields, each with a job:

ca.crt — the cluster's certificate authority. Whoever holds this Secret can verify they're really talking to the real API server, not an impersonator.
namespace — just the namespace this identity belongs to (default), so any client using this Secret knows where it "lives."
token — the important one. This is a JWT (JSON Web Token), signed by the cluster. This is what a Pod, script, or client hands to the API server to say "I am build-sa, and here's proof." I've truncated the token above — treat these things like passwords, not like log output, even in a lab cluster.

If you decode a Service Account JWT (there's nothing secret about the structure, just don't paste your actual token online), you'll find claims like the issuer, the namespace, and a sub field that looks like system:serviceaccount:default:build-sa. That string is the Service Account's real, canonical identity inside Kubernetes — remember it, it matters in a minute.

Testing Permissions

Now the interesting part. I wanted to see what build-sa could actually do, without creating a Pod yet:

❯ k get pods --as build-sa
Error from server (Forbidden): pods is forbidden: User "build-sa" cannot list resource "pods" in API group "" in the namespace "default"

Two things to unpack here.

First — what does --as actually do? It's easy to assume you're "logging in" as build-sa. You're not. --as is impersonation. You're still using your own admin credentials to talk to the API server, but you're telling it: "Evaluate this request as if it came from build-sa instead of me." The API server checks whether your account is allowed to impersonate build-sa (admin usually is), and if so, it re-runs the authorization check as that identity. It's a testing and debugging superpower — you get to try on someone else's permissions without actually needing their credentials.

Second — why Forbidden? This is the single most important lesson in this whole exercise: the Service Account existed. The token existed. Authentication succeeded. Authorization failed.

Those are two separate gates in Kubernetes:

Authentication — "Who are you?" build-sa proved this fine, via impersonation.
Authorization — "What are you allowed to do?" Nothing had granted build-sa any permissions at all, so RBAC said no.

A lot of beginners collapse these into one concept. They're not the same thing, and separating them in your head will save you hours of confused debugging later.

Creating the Role

❯ k create role build-role \
--verb=list,get,watch \
--resource=pod
role.rbac.authorization.k8s.io/build-role created

Breaking down the flags:

role build-role — I'm creating a Role, which is a namespaced set of permissions (as opposed to a ClusterRole, which applies cluster-wide).
--verb=list,get,watch — the specific actions this Role allows. Not create, not delete — just read-style operations.
--resource=pod — the Kubernetes resource type these verbs apply to.

On its own, a Role does nothing. It's a permission template sitting unused until something binds an identity to it.

Creating the RoleBinding

❯ k create rolebinding rb \
--role=build-role \
--user=build-sa
rolebinding.rbac.authorization.k8s.io/rb created

This is the glue: it binds the build-role permissions to the identity build-sa.

One honest caveat worth flagging, since I want this post to be technically accurate and not just "it worked so it's fine": I used --user=build-sa here, and it worked in this exercise because I was testing purely through impersonation with --as build-sa, and --as treats whatever string you give it as a plain username.

But that's not actually how a real Service Account identifies itself to the cluster. Its canonical identity is system:serviceaccount:<namespace>:<name> — in my case, system:serviceaccount:default:build-sa. In production, the standard, correct way to bind a RoleBinding to a Service Account is:

--serviceaccount=default:build-sa

If I'd actually mounted this token inside a real client and tried to use it (rather than testing through admin impersonation), a binding made with --user=build-sa would not match that Service Account's real identity, and the request would still be Forbidden. So: my command worked for this specific impersonation test, but if you're setting this up for a real workload, use --serviceaccount=<namespace>:<name> instead. Good to know the difference exists before it costs you a debugging session.

Testing Again

❯ k get pods --as build-sa
No resources found in default namespace.

No error this time — just an empty result, because there genuinely were no pods yet. Authentication and authorization both passed. That's the fix landing.

Creating a Deployment

❯ k create deploy test --image=nginx
deployment.apps/test created

Worth pointing out deliberately: I created this Deployment as myself (admin), not as build-sa. build-sa only has list, get, and watch on pods — nowhere near enough permission to create a Deployment. If I'd tried k create deploy test --image=nginx --as build-sa, it would have been Forbidden too.

Watching the Pod

❯ k get po -w --as build-sa
NAME                    READY   STATUS    RESTARTS   AGE
test-56848fd9dc-fhdjv   1/1     Running   0          89s

The -w flag means watch — instead of printing the pod state once and exiting, kubectl keeps the connection open and streams updates as the pod's status changes. Combined with --as build-sa, this also confirms build-sa's watch verb from the Role is working, not just list and get.

Going Inside the Pod

This is where it got genuinely interesting.

❯ k exec -it test-56848fd9dc-fhdjv bash
error: exec [POD] [COMMAND] is not supported anymore. Use exec [POD] -- [COMMAND] instead

I hit this error a few times before it clicked. Older kubectl versions let you run kubectl exec pod bash and it would figure out where your flags ended and the container's command began. Modern kubectl refuses to guess — it wants an explicit -- to separate "arguments meant for kubectl" from "the command to run inside the container." So:

❯ k exec -it test-56848fd9dc-fhdjv -- bash
root@test-56848fd9dc-fhdjv:/#

That extra -- is the whole fix. Everything after it belongs to the container, not to kubectl.

The Most Interesting Discovery

Once inside, I went looking for exactly the kind of thing this Pod would need to talk to the API server — and there it was:

root@test-56848fd9dc-fhdjv:/var/run/secrets/kubernetes.io/serviceaccount# ls -lrt
total 0
lrwxrwxrwx 1 root root 12 Jul  7 14:38 token -> ..data/token
lrwxrwxrwx 1 root root 16 Jul  7 14:38 namespace -> ..data/namespace
lrwxrwxrwx 1 root root 13 Jul  7 14:38 ca.crt -> ..data/ca.crt

Same three files as the Secret I inspected earlier — token, namespace, ca.crt — automatically mounted at /var/run/secrets/kubernetes.io/serviceaccount, with no configuration from me at all. Kubernetes does this for every Pod by default, via a projected volume, so that any process running inside the container can immediately authenticate to the API server if it needs to (this is how things like controllers and operators, running as Pods themselves, talk back to Kubernetes).

Now the obvious beginner question, and the one I asked myself immediately: "Wait — I created build-sa. Why does the token in here say default?"

Sure enough, decoding what's inside shows a sub claim of system:serviceaccount:default:default — the default Service Account, not build-sa.

The answer is almost anticlimactic once you see it: my Deployment spec never included serviceAccountName: build-sa. I never told Kubernetes which Service Account this Pod should use — so it fell back to the one thing every namespace already has, the one from Step 1. The default Service Account.

This is, genuinely, the most valuable lesson from the whole exercise: creating a Service Account doesn't automatically attach it to anything. You have to explicitly wire it into a Pod (or Deployment, or Job, etc.) with serviceAccountName — or Kubernetes will silently keep using default on your behalf, and you won't get an error telling you. You'll just quietly have the wrong permissions, or, worse in production, the default Service Account might have more access than you meant to grant that workload.

Key Takeaways

Service Accounts are identities for Pods — not humans.
Every namespace automatically gets a default Service Account.
Service Accounts authenticate using tokens (JWTs), not certificates.
Newer Kubernetes versions don't auto-create long-lived token Secrets anymore — you create them explicitly when you need one.
RBAC still governs what a Service Account is allowed to do, exactly like it does for users.
--as impersonates another identity for testing — it doesn't log you in as them.
Authentication and authorization are two separate checks: proving who you are is not the same as being allowed to do something.
Bind RoleBindings to Service Accounts with --serviceaccount=<namespace>:<name> in real setups, not --user=<name>.
Pods automatically get their Service Account's credentials mounted at /var/run/secrets/kubernetes.io/serviceaccount — no extra config needed.
If you don't set serviceAccountName on a Pod, Kubernetes silently uses default. Always check.

Understanding Kubernetes ClusterRoles and ClusterRoleBindings

Adeoye Malumi — Tue, 07 Jul 2026 13:13:42 +0000

Yesterday I learned about Roles and RoleBindings. Today I explored ClusterRoles and ClusterRoleBindings for cluster-scoped resources like Nodes.

Why a Role Wasn't Enough

kubectl auth can-i get nodes

Output:

Warning: resource 'nodes' is not namespace scoped
no

Nodes are cluster-scoped resources, so a namespace Role cannot grant access.

Creating a ClusterRole

kubectl create clusterrole node-reader --verb=get,list,watch --resource=nodes

Verify:

kubectl get clusterrole | grep node-reader

Creating a ClusterRoleBinding

I first made a mistake:

kubectl create clusterrolebinding reader-binding --clusterrole=cluster-reader --user=adeoye

Checking permissions:

kubectl auth can-i get nodes --as adeoye

returned:

no - RBAC: clusterrole.rbac.authorization.k8s.io "cluster-reader" not found

The binding referenced a ClusterRole that didn't exist.

I fixed it by deleting and recreating the binding:

kubectl delete clusterrolebinding reader-binding
kubectl create clusterrolebinding reader-binding --clusterrole=node-reader --user=adeoye

Now:

kubectl auth can-i get nodes --as adeoye

returned:

yes

Testing as adeoye

kubectl config use-context adeoye
kubectl get nodes

Listing nodes worked.

Describing a node:

kubectl describe node cka-cluster3-worker

showed most information, but RBAC blocked access to leases and pods because the ClusterRole only granted access to nodes.

Trying to delete a node:

kubectl delete node cka-cluster3-worker

returned Forbidden because the ClusterRole only included get, list, and watch.

Key Takeaways

Roles are namespace-scoped.
ClusterRoles are cluster-scoped.
ClusterRoleBindings grant cluster-wide permissions.
RBAC permissions are resource-specific.
Kubernetes follows the principle of least privilege.

Understanding Kubernetes RBAC: Roles, RoleBindings, and and Client Certificates

Adeoye Malumi — Mon, 06 Jul 2026 14:14:46 +0000

One thing I discovered today is that before Kubernetes can decide what you're allowed to do, it first needs to know who you are.

That means there are two separate steps:

Authentication – Proving your identity using a client certificate.
Authorization – Deciding what you're allowed to do using RBAC.

In this lab, I created my own Kubernetes user named adeoye, authenticated the user with a certificate, and then used RBAC to control what that user could access.

Step 1 – Generate a Private Key

openssl genrsa -out adeoye.key 2048

This generates a 2048-bit RSA private key.

Think of this as your digital identity. It should never be shared.

Step 2 – Generate a Certificate Signing Request (CSR)

openssl req -new -key adeoye.key -out adeoye.csr -subj "/CN=adeoye"

A Certificate Signing Request asks Kubernetes to issue a client certificate.

The important part is:

/CN=adeoye

The Common Name (CN) becomes the Kubernetes username.

Step 3 – Base64 Encode the CSR

cat adeoye.csr | base64 | tr -d '\n'

Kubernetes expects the CSR to be embedded inside a YAML manifest as Base64-encoded text. The tr -d '\n' removes line breaks so it becomes a single continuous string.

Step 4 – Submit the CSR

After inserting the encoded CSR into csr.yaml, I submitted it.

kubectl apply -f csr.yaml

Initially, the request was in the Pending state so I had to approve the request.

Step 5 – Approve the Certificate

kubectl certificate approve adeoye

Approving the request tells Kubernetes to trust the user and issue a signed client certificate.

Step 6 – Extract the Signed Certificate

kubectl get csr adeoye -o jsonpath='{.status.certificate}' | base64 -d > adeoye.crt

Now I had:

adeoye.key
adeoye.crt

The key proves my identity, while the certificate proves Kubernetes trusts that identity.

Step 7 – Configure kubectl

kubectl config set-credentials adeoye \
--client-key=adeoye.key \
--client-certificate=adeoye.crt

Then I created a context.

kubectl config set-context adeoye \
--cluster=kind-cka-cluster3 \
--user=adeoye

A mistake I made: I initially typed lind-cka-cluster3 instead of kind-cka-cluster3, which caused API errors until I corrected the cluster name.

Finally, I switched to my new user.

kubectl config use-context adeoye

At this point I was authenticated, but I still didn't have permissions to perform actions.

RBAC Time!

Authentication tells Kubernetes who I am.

RBAC tells Kubernetes what I'm allowed to do.

Creating the Role

role.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: pod-reader

rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs:
    - get
    - watch
    - list

This Role only allows reading Pods.

Apply it:

kubectl apply -f role.yaml

Creating the RoleBinding

binding.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: default

subjects:
- kind: User
  name: adeoye
  apiGroup: rbac.authorization.k8s.io

roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

This binds the adeoye user to the pod-reader Role.

kubectl apply -f binding.yaml

Testing the Permissions

Verify permission:

kubectl auth can-i get pods

Output:

yes

List Pods:

kubectl get pods

Output:

No resources found in default namespace.

That means the command succeeded, but there were simply no Pods.

Next, I tried listing Deployments.

kubectl get deploy

Output:

Error from server (Forbidden): deployments.apps is forbidden

Why Could I Read Pods but Not Deployments?

The answer is in the Role.

It only grants access to:

resources:
- pods

It does not include Deployments.

Deployments belong to the apps API group, while Pods belong to the core API group.

Since my RoleBinding connects adeoye to the pod-reader Role, Kubernetes only allows the permissions defined there.

No Deployment permissions were granted, so Kubernetes correctly denied access.

This is the Principle of Least Privilege in action.

What I Learned

Kubernetes first authenticates users using client certificates.
The certificate's Common Name becomes the Kubernetes username.
A Role defines permissions.
A RoleBinding assigns those permissions to users.
Authentication and authorization are separate processes.
RBAC only grants explicitly defined permissions.
Pods and Deployments are different resources, so access to one doesn't imply access to the other.

Final Thoughts

Today's lab tied together authentication and authorization. Creating my own client certificate, configuring a new kubectl context, and then restricting that user with RBAC made the concepts much easier to understand.

Security in Kubernetes isn't about giving users full access,but about giving them exactly the permissions they need, nothing more.

Day 20&21/40 - Manage TLS Certificates In a Kubernetes Cluster

Adeoye Malumi — Fri, 03 Jul 2026 11:57:11 +0000

Today's challenge was about understanding how Kubernetes uses TLS certificates to secure communication between its components, and getting hands-on by onboarding a "new user" into the cluster using a Certificate Signing Request (CSR).

If you've ever wondered how kubectl actually proves to the API server that you're allowed to run commands, this is the exercise that answers it.

A quick rule of thumb before diving in: if a filename or extension has the word "key" in it, it's a private key. Otherwise, it's a public certificate. Keep that in mind as you read through the commands below.

Step 1: Generate a private key and a CSR

openssl genrsa -out adeoye.key 2048
openssl req -new -key adeoye.key -out adeoye.csr -subj "/CN=adeoye"

genrsa creates the private key, and req -new uses that key to generate a Certificate Signing Request. The -subj "/CN=adeoye" flag sets the "Common Name" on the certificate — this becomes the username Kubernetes will recognize once the cert is issued.

Gotcha: I typed adam.key in my req command before actually creating a file with that name, and got:
Could not open file or uri for loading private key from adam.key: No such file or directory
Lesson learned — the key file has to exist before you reference it when generating the CSR. Double-check your filenames match across commands.

Step 2: Base64-encode the CSR

The CSR object in Kubernetes expects the request in base64 form inside a YAML manifest. This is where things got interesting:

cat adeoye.csr | base64 tr -d "/"

That's wrong — it treats tr as a filename argument for base64, not a separate piped command:

base64: extra operand '/'

The correct version needs a pipe (|) before tr, and the actual newline character (\n), not the literal string "/n":

cat adeoye.csr | base64 | tr -d "\n"

This gives you a single-line base64 blob with no line breaks — exactly what the CSR YAML needs.

Step 3: Create the CSR manifest and apply it

Create a csr.yaml file with the base64-encoded request embedded in it, following the structure from the Kubernetes docs, then:

kubectl apply -f csr.yaml
kubectl get csr

NAME     AGE   SIGNERNAME                            REQUESTOR          REQUESTEDDURATION   CONDITION
adeoye   25s   kubernetes.io/kube-apiserver-client   kubernetes-admin   24h                 Pending

The CSR shows up as Pending — Kubernetes has received the request but it hasn't been approved yet.

Step 4: Approve the CSR

kubectl certificate approve adeoye

certificatesigningrequest.certificates.k8s.io/adeoye approved

Checking again with kubectl get csr now shows Approved,Issued — the cluster has signed the certificate.

Gotcha: kubectl approve certificate adeoye doesn't work — the order of the words matters:
error: unknown command "approve" for "kubectl"
It's kubectl certificate approve <name>, not kubectl approve certificate <name>.

Step 5: Retrieve and decode the issued certificate

kubectl get csr adeoye -o yaml > issuecert.yaml

The issued certificate lives (base64-encoded) inside that YAML file under status.certificate. To read the actual PEM certificate, decode it:

echo "<base64-blob>" | base64 -d

This prints out the human-readable certificate:

-----BEGIN CERTIFICATE REQUEST-----
MIICVjCCAT4CAQAwETEPMA0GA1UEAww...
-----END CERTIFICATE REQUEST-----

Gotcha: Pasting a long base64 string directly into the terminal can trigger "bracketed paste mode," which shows up as weird escape codes and leaves your shell hanging on a dquote> prompt. If that happens, press Ctrl+C to cancel and try again — wrapping the string in quotes (echo "...") helps avoid it.

To save the decoded certificate directly to a file instead of just printing it, redirect the output:

echo "<base64-blob>" | base64 -d > learner.crt

Gotcha: Writing -o learner.crt instead of > learner.crt doesn't work here — -o isn't a flag for base64, and the shell will try to run it as its own command:
zsh: command not found: -o
Use the > redirect operator to send output to a file.

Repeating the process for a second user

To make sure the workflow actually sank in, I repeated the whole thing for a second user, learner, this time being more careful with the pipe syntax:

openssl genrsa -out learner.key 2048
openssl req -new -key learner.key -out learner.csr -subj "/CN=learner"
kubectl apply -f learner.yaml
kubectl certificate approve learner
kubectl get csr learner -o yaml > issuecert-learner.yaml

NAME      AGE     SIGNERNAME                            REQUESTOR          REQUESTEDDURATION   CONDITION
learner   4m37s   kubernetes.io/kube-apiserver-client   kubernetes-admin   7d                  Approved,Issued

Once the certificate was decoded and saved to learner.crt, it would be ready to plug into a kubeconfig for that user, giving them client-cert-based access to the cluster.

Key commands from today

Generate a private key:

openssl genrsa -out adeoye.key 2048

Generate a CSR:

openssl req -new -key adeoye.key -out adeoye.csr -subj "/CN=adeoye"

Approve a CSR:

kubectl certificate approve <certificate-signing-request-name>

Deny a CSR:

kubectl certificate deny <certificate-signing-request-name>

Reference: Kubernetes docs — Create CertificateSigningRequest

What I took away from today

A CSR workflow has four clean stages: generate a key → generate a CSR → submit it to the API server as a CertificateSigningRequest object → get it approved and retrieve the signed cert.
base64 | tr -d "\n" (with the pipe, and a real newline character) is the correct way to get a clean, single-line encoded string for a YAML manifest.
kubectl certificate approve — command, then subcommand, then name — not the other way around.
Almost all my mistakes today were syntax slips (missing pipes, wrong flags, mismatched filenames), not conceptual ones. That's usually a good sign you understand what you're doing, even when the terminal disagrees with you for a minute.

Day 19/40 - Kubernetes ConfigMaps and Secrets

Adeoye Malumi — Thu, 02 Jul 2026 11:22:28 +0000

Today's mission: stop hardcoding values into pod manifests and learn how to inject them properly using ConfigMaps and Secrets. I also managed to break a few things along the way, which honestly taught me more than if everything had just worked first try. Let's get into it.

What is a ConfigMap?

As your manifest grows, it becomes harder to manage a pile of environment variables directly inside it. A ConfigMap lets you:

Pull those key-value pairs out of the manifest and store them as their own Kubernetes object
Inject that ConfigMap into a pod as environment variables (or as files, but that's for another day)
Reuse the same ConfigMap across multiple pods, so you're not repeating yourself everywhere

Basically, it's a clean way to separate configuration from your application definition.

Creating a ConfigMap

kubectl create cm app-cm --from-literal=firstname=adeoye --from-literal=lastname=malumi

Here, firstname and lastname are the keys, and adeoye / malumi are the values.

My first "gotcha" of the day

I actually tried to split this command across two lines like this:

kubectl create cm app-cm --from-literal=firstname=adeoye \
--from-literal=lastname=malumi

First attempt gave me:

error: exactly one NAME is required, got 2

Turns out I had a trailing space after the backslash on the first line. In bash, the \ has to be the very last character on the line for line continuation to work. A space after it breaks the continuation, and the shell just treats the next line as a separate (garbled) argument. When I finally got the syntax right, kubectl describe cm app-cm showed the value for firstname as adeoye--from-literal=lastname=malumi — my second flag had literally been glued onto the first value as text!

Lesson learned: either keep the command on one line, or make sure there's truly nothing (not even a space) after your \.

Once fixed, describe showed exactly what I wanted:

Data
====
firstname:
----
adeoye

lastname:
----
malumi

Injecting a ConfigMap into a Pod

Here's the pod manifest I used to consume the ConfigMap as environment variables:

apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
  labels:
    app.kubernetes.io/name: MyApp
spec:
  containers:
  - name: myapp-container
    image: busybox:1.28
    env:
    - name: FIRSTNAME
      valueFrom:
        configMapKeyRef:
          name: app-cm
          key: firstname
    command: ['sh', '-c', 'echo The app is running! && sleep 3600']

The valueFrom.configMapKeyRef block tells Kubernetes: "don't hardcode this value — go fetch it from the app-cm ConfigMap, using the firstname key."

Gotcha #2: Pods are mostly immutable

After creating my ConfigMap with both firstname and lastname, I went back and added a LASTNAME env var to my already-running pod's manifest, then ran:

kubectl apply -f pod.yaml

And got slapped with:

The Pod "myapp-pod" is invalid: spec: Forbidden: pod updates may not change fields other than 
`spec.containers[*].image`,`spec.initContainers[*].image`,`spec.activeDeadlineSeconds`,
`spec.tolerations` (only additions to existing tolerations),
`spec.terminationGracePeriodSeconds` (allow it to be set to 1 if it was previously negative)

This was a great reminder: a bare Pod's spec is mostly frozen once it's created. Kubernetes only lets you patch a small allowlist of fields in-place — image, a couple of grace period settings, and tolerations. Environment variables are not on that list.

Two ways around this:

Option 1 — Delete and recreate:

kubectl delete pod myapp-pod
kubectl apply -f pod.yaml

Option 2 — Force replace in one shot:

kubectl replace --force -f pod.yaml

I went with replace --force, and it worked perfectly:

❯ kubectl replace --force -f pod.yaml
pod "myapp-pod" deleted from default namespace
pod/myapp-pod replaced

Verified inside the container:

❯ kubectl exec -it myapp-pod -- sh
/ # echo $FIRSTNAME
adeoye
/ # echo $LASTNAME
malumi

Side note: this is actually one of the reasons Deployments exist and are preferred over bare Pods in real-world use. A Deployment lets you update the pod template (image, env vars, etc.) and kubectl apply will handle the rolling replacement of pods for you — no manual delete/recreate dance required.

Secrets: same idea, but for sensitive data

Secrets work almost identically to ConfigMaps, except they're meant for sensitive values like passwords, tokens, or usernames you don't want sitting in plain text in your manifests.

I followed the official Kubernetes docs on defining container environment variables using Secret data.

Creating a Secret

kubectl create secret generic backend-user --from-literal=backend-username='adeoye-admin'

Using the sample pod from the docs

kubectl create -f https://k8s.io/examples/pods/inject/pod-single-secret-env-variable.yaml

This pod injects the secret as an environment variable called SECRET_USERNAME.

One more small hiccup

Right after creating the pod, I tried to exec into it immediately:

❯ kubectl exec -i -t env-single-secret -- /bin/sh -c 'echo $SECRET_USERNAME'
error: Internal error occurred: unable to upgrade connection: container not found ("envars-test-container")

Checking kubectl get po showed the pod was still ContainerCreating. Classic case of being too impatient — I just needed to wait a few seconds for the container to actually spin up. Once it hit Running:

❯ kubectl exec -i -t env-single-secret -- /bin/sh -c 'echo $SECRET_USERNAME'
adeoye-admin

And there it is — the secret successfully injected as an env var.

Key Takeaways

ConfigMaps decouple non-sensitive configuration from your pod manifests, and can be reused across multiple pods.
Secrets work the same way but are the right home for sensitive values.
Both can be injected into a pod's containers via env.valueFrom.configMapKeyRef or secretKeyRef.
Watch your shell syntax — a stray space after a line-continuation \ will silently mangle your command.
Pods are mostly immutable once created. Only image, tolerations, and a couple of grace-period fields can be patched live. Everything else means delete + recreate, or kubectl replace --force. This is a big part of why Deployments exist.
Give newly created pods a moment to leave ContainerCreating before you exec into them.

That's Day 19 done — small errors, but each one taught me something I'll remember longer than if it had just worked on the first try.

Day 17/40 — Kubernetes Autoscaling: HPA vs VPA Explained With Hands-On Practice

Adeoye Malumi — Wed, 01 Jul 2026 12:09:21 +0000

If you've ever wondered how Kubernetes knows when to spin up more pods or give a pod more memory, that's autoscaling — and it's one of those concepts that sounds intimidating until you actually do it yourself. Day 17 of the #40DaysOfKubernetes challenge is where it clicked for me.

What is Autoscaling in Kubernetes?

At its core, autoscaling means Kubernetes adjusts resources automatically based on demand. You don't manually intervene every time traffic spikes. There are two main types:

HPA (Horizontal Pod Autoscaler) — adds or removes pods based on CPU/memory usage
VPA (Vertical Pod Autoscaler) — adjusts the resources (CPU/memory) of existing pods

Think of HPA as hiring more staff when the shop gets busy. VPA is more like giving one staff member more tools to handle the workload alone.

What I Did — Setting Up HPA

First I deployed the sample php-apache app with defined CPU requests and limits:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: php-apache
spec:
  selector:
    matchLabels:
      run: php-apache
  template:
    metadata:
      labels:
        run: php-apache
    spec:
      containers:
      - name: php-apache
        image: registry.k8s.io/hpa-example
        ports:
        - containerPort: 80
        resources:
          limits:
            cpu: 500m
          requests:
            cpu: 200m

apiVersion: v1
kind: Service
metadata:
  name: php-apache
  labels:
    run: php-apache
spec:
  ports:
  - port: 80
  selector:
    run: php-apache

The key part is setting resources.requests.cpu — HPA needs this to calculate utilization. Without it, the autoscaler has nothing to measure against.

Then I created the HPA object targeting 50% average CPU utilization, with a minimum of 1 pod and maximum of 10. This is the declarative method:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: php-apache
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: php-apache
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50

While this is the imperative method:

kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10

Generating Load to Watch It Scale

This is the fun part. I ran a load generator in a separate pod — basically a loop hammering the apache service with requests:

kubectl run -i --tty load-generator --rm --image=busybox:1.28 --restart=Never -- \
  /bin/sh -c "while sleep 0.01; do wget -q -O- http://php-apache; done"

Then watched the HPA respond in real time:

kubectl get hpa php-apache --watch

Watching the replica count climb from 1 to several pods as CPU utilization crossed 50% made the whole concept land in a way that reading documentation never does.

HPA vs VPA — When Do You Use Which?

	HPA	VPA
Scales	Number of pods	Pod resource limits
Best for	Stateless apps with variable traffic	Apps where sizing is hard to predict upfront
Works with	CPU, memory, custom metrics	CPU and memory

In practice, most production workloads use HPA. VPA is useful during early deployment when you're still figuring out the right resource requests for an app.

Key Takeaway

Don't skip setting resources.requests in your deployment spec. HPA is blind without it. That one line is what connects your workload to the autoscaler.

DEV Community: Adeoye Malumi

The Kubernetes Weekend That Humbled Me — Part 2: From One VM to a Real Cluster

The CNI Wars

Adding a Worker Node

The Network Both VMs Actually Needed

The API Server Was Listening on the Wrong Interface

Joining the Worker

Fixing the Node IPs

Proving It Actually Works

What This Whole Weekend Actually Taught Me

Full Command Reference

The Kubernetes Weekend That Humbled Me — Part 1: Just Getting a VM to Boot

I had a simple plan. Spin up a couple of VMs, run kubeadm init, join a worker, deploy nginx, post about it. That plan lasted about twenty minutes before Windows and VirtualBox decided I hadn't suffered enough that week.

Round 1: Multipass, or How to Meet a Ghost

Round 2: Cloud Images and the ISO That Wouldn't Build

Round 3: I Nearly Downloaded the Full ISO

Round 4: Breaking In Through Recovery Mode

Building the Master Node

The Disk That Was Way Too Small

Finally, a Working Control Plane

Debugging a Broken Ingress Setup: A Step-by-Step Walkthrough

The Symptom

Step 1: Confirm What the Hostname Actually Resolves To

Step 2: Test the IP Directly, Bypassing the Hostname

Step 3: Find a Real, Reachable Address Instead

Step 4: Ruling Out the Obvious Culprits

Step 5: Investigating the DNS Failure

Step 6: Isolating DNS vs. General Networking

Step 7: Confirming Cross-Node Networking Was Broken

Step 8: Rebuilding the Cluster

Step 9: A Slow Pod Isn't Always a Stuck Pod

Step 10: Fixing the Service Configuration Correctly

Step 11: Confirming Everything Works

Why the Port Still Has to Be Specified

Setting Up Ingress for a Flask App

Why Use Ingress Instead of NodePort or LoadBalancer?

Step 0: Building and Pushing the Docker Image

Step 1: The Kubernetes Manifests

Step 2: Debugging Internal Connectivity

Step 3: Applying the Ingress — Address Stays Empty

Step 4: Installing the NGINX Ingress Controller

Step 5: LoadBalancer Stuck on <pending>

Step 6: Investigating the CNI (Calico Installation)

Step 7: Testing From Inside the Cluster

Step 8: Mapping example.com in /etc/hosts

Understanding PersistentVolumes (PV) and PersistentVolumeClaims (PVC)

Why do we even need PVs?

The Relationship

Step 1 — Create a PersistentVolume

Step 2 — Create a PersistentVolumeClaim

My First Roadblock

Why Was My PVC Stuck in Pending?

Step 3 — Mount the PVC inside a Pod

Step 4 — Test It

The Big Idea

Commands I Used

What I Learned

Kubernetes Network Policies Explained (and How I Actually Made Them Work)

What Even Is a Network Policy?

Why I Needed a Different CNI

My Test Cluster: A Kind Setup with Calico

kind Cluster YAML

Creating the Cluster

The App I Built: Frontend → Backend → MySQL

Pods & Services

My First Real Network Policy: Only Backend → MySQL

NetworkPolicy Manifest

How I Tested It

Wrapping Up

Understanding Kubernetes Service Accounts (The Identity Every Pod Uses)

User vs. Service Account

Step 1 — Checking Existing Service Accounts

Step 2 — Creating My Own Service Account

Step 3 — Creating a ServiceAccount Token Secret

Inspecting the Secret

Testing Permissions

Creating the Role

Creating the RoleBinding

Testing Again

Creating a Deployment

I had a simple plan. Spin up a couple of VMs, run `kubeadm init`, join a worker, deploy nginx, post about it. That plan lasted about twenty minutes before Windows and VirtualBox decided I hadn't suffered enough that week.

Step 5: LoadBalancer Stuck on `<pending>`

Why Was My PVC Stuck in `Pending`?