DEV Community

Cover image for The Kubernetes Weekend That Humbled Me — Part 2: From One VM to a Real Cluster
Adeoye Malumi
Adeoye Malumi

Posted on

The Kubernetes Weekend That Humbled Me — Part 2: From One VM to a Real Cluster

#learning #kubernetes #cloudnative #devops

Quick recap in case you're jumping in here: in Part 1 I fought Multipass, broken cloud-init, a 2.1 GB disk, and a bunch of other nonsense just to get one Ubuntu VM with a healthy Kubernetes control plane on it. That was the "get anything to boot" fight. This part is the fight I actually signed up for — giving pods a real network and turning one lonely VM into an actual cluster.

The CNI Wars

Quick context for anyone newer to this: a CNI plugin is the thing responsible for giving every pod its own IP and making sure traffic can actually get routed between pods, including pods on different nodes. Without one, kubectl works, your control plane is healthy, but nothing you deploy can actually talk to anything else. CoreDNS staying pending at the end of Part 1 was a symptom of exactly this — no CNI, no pod network, no CoreDNS.

First attempt was Flannel, since it's the "default" answer most tutorials give you:

kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml
Enter fullscreen mode Exit fullscreen mode

The flannel pod went straight into CrashLoopBackOff with:

failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory
Enter fullscreen mode Exit fullscreen mode

That file gets written by flanneld itself, after it successfully reaches the API server and learns what pod CIDR the cluster is using. In my setup, flannel's own pod couldn't reach the API server at 10.96.0.1:443 — the cluster-internal service IP for the API — so it never got far enough to write that file in the first place. Chicken and egg: the CNI needs API connectivity to bootstrap the network, but with a single-NIC NAT setup like mine, that internal routing wasn't reliably working yet.

Tried Weave next, thinking a different implementation might sidestep it:

kubectl apply -f https://github.com/weaveworks/weave/releases/download/v2.8.1/weave-daemonset-k8s.yaml
Enter fullscreen mode Exit fullscreen mode

Images pulled fine this time, but the readiness probe failed:

Readiness probe failed: Get "http://127.0.0.1:6784/status": dial tcp 127.0.0.1:6784: connect: connection refused
Enter fullscreen mode Exit fullscreen mode

Same root cause wearing a different outfit — Weave's own daemon also needs to reach the API server to establish its network bridge, and it was hitting the same wall Flannel did.

At that point I stopped trying "real" CNIs and just wanted pods to schedule so I could keep moving. I hand-wrote a plain bridge network config that doesn't depend on talking to the API server at all:

sudo apt-get install -y containernetworking-plugins

sudo mkdir -p /etc/cni/net.d
sudo tee /etc/cni/net.d/10-bridge.conf <<EOF
{
  "cniVersion": "1.0.0",
  "name": "bridge",
  "type": "bridge",
  "bridge": "cni0",
  "isGateway": true,
  "ipMasq": true,
  "ipam": {
    "type": "host-local",
    "subnet": "10.244.0.0/16",
    "routes": [{ "dst": "0.0.0.0/0" }]
  }
}
EOF

sudo systemctl restart containerd
Enter fullscreen mode Exit fullscreen mode

The node flipped to Ready and nginx deployed. Worth being straight with you about what this actually is, though: it's a local bridge, not an overlay network. It's fine for unblocking a single node and proving pod scheduling works, but it doesn't give you real cross-node pod routing the way Flannel or Calico would once properly configured. Good enough to keep learning on, not something I'd trust for anything beyond that.

Adding a Worker Node

Rather than repeating the whole cloud-image nightmare from Part 1, I cloned the master's disk — it already had containerd, kubeadm, kubelet, and kubectl installed, which saved me from doing that setup twice:

& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" clonehd "k8s-master\disk.vdi" "k8s-worker-1\disk.vdi" --format VDI

& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" createvm --name k8s-worker-1 --ostype Ubuntu_64 --register
& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifyvm k8s-worker-1 --cpus 2 --memory 4096
& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" storagectl k8s-worker-1 --name "SATA" --add sata
& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" storageattach k8s-worker-1 --storagectl "SATA" --port 0 --device 0 --type hdd --medium "k8s-worker-1\disk.vdi"
& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifyvm k8s-worker-1 --nic1 nat
& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifyvm k8s-worker-1 --natpf1 "ssh,tcp,,2223,,22"
Enter fullscreen mode Exit fullscreen mode

But because I cloned the disk, this new VM also thinks it's still the master. Had to strip that identity out before it could join as anything:

sudo hostnamectl set-hostname k8s-worker-1

sudo sed -i 's/k8s-master/k8s-worker-1/g' /etc/hosts

sudo kubeadm reset -f
sudo rm -rf /etc/kubernetes/ /var/lib/kubelet/ /var/lib/dockershim/ /var/run/kubernetes/ ~/.kube/
Enter fullscreen mode Exit fullscreen mode

kubeadm reset tears down the control-plane state and cluster certificates the clone inherited, so the node starts clean instead of trying to act as a second control plane.

The Network Both VMs Actually Needed

Here's something that trips people up: each VM's default NAT adapter can reach the internet, but NAT networking isolates VMs from each other by design — that's the whole point of NAT, it's not meant for VM-to-VM traffic. Master and worker literally couldn't see each other over NAT alone. Both needed a second network adapter, set to VirtualBox's internal network mode, purely for talking to one another:

& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifyvm k8s-master --nic2 intnet --intnet2 "k8s-net"
& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifyvm k8s-worker-1 --nic2 intnet --intnet2 "k8s-net"
Enter fullscreen mode Exit fullscreen mode

Then static IPs on that internal network so the addresses wouldn't shift around:

Master:

sudo tee /etc/netplan/02-internal.yaml <<EOF
network:
  version: 2
  ethernets:
    enp0s8:
      dhcp4: no
      addresses:
        - 192.168.56.10/24
EOF
sudo netplan apply
Enter fullscreen mode Exit fullscreen mode

Worker:

sudo tee /etc/netplan/02-internal.yaml <<EOF
network:
  version: 2
  ethernets:
    enp0s8:
      dhcp4: no
      addresses:
        - 192.168.56.11/24
EOF
sudo netplan apply
Enter fullscreen mode Exit fullscreen mode

The API Server Was Listening on the Wrong Interface

Even with both VMs able to reach each other, joining still wasn't going to work, because the API server had only ever been told to listen on 10.0.2.15 — the NAT interface — since that was the address I originally passed to kubeadm init back in Part 1. The worker needed to reach it over the internal network instead, on 192.168.56.10, and the API server simply wasn't listening there.

sudo sed -i 's/--advertise-address=10.0.2.15/--advertise-address=192.168.56.10/' /etc/kubernetes/manifests/kube-apiserver.yaml

sleep 30

sudo ss -tlnp | grep 6443
# LISTEN 0 4096 *:6443 *:*
Enter fullscreen mode Exit fullscreen mode

Editing a file under /etc/kubernetes/manifests/ is enough on its own — kubelet watches that directory and automatically restarts the API server pod when it changes, no manual restart command needed.

Joining the Worker 

kubeadm token create --print-join-command
Enter fullscreen mode Exit fullscreen mode

Run that on the master, then on the worker with sudo:

sudo kubeadm join 192.168.56.10:6443 --token <TOKEN> --discovery-token-ca-cert-hash sha256:<HASH>
Enter fullscreen mode Exit fullscreen mode

Quick note on what those two values actually do, since it's easy to treat them as just noise you copy-paste: the token is a short-lived credential that authenticates the worker to the cluster, and the discovery hash lets the worker verify it's actually talking to the real API server's certificate authority and not something impersonating it. Both matter — without the hash check, a join command intercepted on the network could point your worker at the wrong cluster entirely.

It joined successfully.

Fixing the Node IPs 

kubectl get nodes -o wide
Enter fullscreen mode Exit fullscreen mode 
NAME           STATUS   ROLES           AGE   VERSION   INTERNAL-IP   OS-IMAGE
k8s-worker-1   Ready    <none>          25s   v1.29.6   10.0.2.15     Ubuntu 22.04.5 LTS
ubuntu         Ready    control-plane   4m    v1.29.6   10.0.2.15     Ubuntu 22.04.5 LTS
Enter fullscreen mode Exit fullscreen mode

Both nodes reporting the exact same INTERNAL-IP was the giveaway that something was off — that's the NAT address again, not the internal network address I actually wanted the cluster using. Kubelet, left to its own defaults, picks whatever it decides is the primary interface, which on both VMs was still the original NAT adapter rather than the internal one I'd added afterward. The fix is to tell it explicitly which IP to register:

# On master
sudo nano /var/lib/kubelet/kubeadm-flags.env
# Add: --node-ip=192.168.56.10

# On worker
sudo nano /var/lib/kubelet/kubeadm-flags.env
# Add: --node-ip=192.168.56.11

sudo systemctl daemon-reload
sudo systemctl restart kubelet
Enter fullscreen mode Exit fullscreen mode 
NAME           STATUS   ROLES           INTERNAL-IP
ubuntu         Ready    control-plane   192.168.56.10
k8s-worker-1   Ready    <none>          192.168.56.11
Enter fullscreen mode Exit fullscreen mode

That's a real two-node cluster, each one correctly identified on the network I actually built for them.

Proving It Actually Works 

kubectl create deployment nginx --image=nginx --replicas=3
kubectl get pods -o wide
Enter fullscreen mode Exit fullscreen mode

Pods landed on both nodes, not just the control plane. That was the moment this stopped being "a VM with kubectl installed" and started being an actual cluster doing what a cluster is supposed to do.

What This Whole Weekend Actually Taught Me

Multipass on Windows is a gamble. Great when it works, but the moment it breaks — ghost VMs, corrupted caches, auth loops — you're debugging the tool instead of learning Kubernetes. Driving VirtualBox directly took longer to set up, but at least every failure told me something true about what was actually happening.

Cloud images assume cloud-init will run. Check that it actually did. Missing SSH host keys, a network interface sitting down, a disk partition way smaller than the disk itself — all three of those trace back to the same root cause: cloud-init's first-boot routine never completed. Once I understood that, every one of those errors made sense instead of feeling random.

A CNI needs API server connectivity to bootstrap itself, so check that connectivity first. Both Flannel and Weave failed for the same underlying reason. If a CNI daemon can't reach 10.96.0.1:443, no amount of reinstalling it will help — the problem is upstream of the CNI entirely.

NAT gets you internet, not VM-to-VM communication. Any time you're building a multi-node setup locally, you need a second, internal-only network specifically for the nodes to reach each other, plus static IPs so kubeadm and kubelet know exactly where things live.

Kubelet and the API server don't automatically know which interface you meant. Both default to whatever they see first, which is often not the interface you actually care about. Pin --advertise-address and --node-ip explicitly instead of trusting the defaults.

The logs were right every time. Every single failure in this whole weekend had its answer sitting in the output, if I actually read it instead of just rerunning the command:

kubectl describe pod <name>
journalctl -u kubelet -f
sudo crictl logs <container-id>
Enter fullscreen mode Exit fullscreen mode

Full Command Reference

Windows host:

& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" showvminfo k8s-master | Select-String "State"
& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" controlvm k8s-master poweroff
& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifyvm k8s-master --cpus 3 --memory 6144
& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" clonehd "source.vdi" "dest.vdi" --format VDI
& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifyhd "disk.vdi" --resize 20480
& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifyvm k8s-master --nic2 intnet --intnet2 "k8s-net"

ssh -p 2222 k8suser@localhost  # Master
ssh -p 2223 k8suser@localhost  # Worker
Enter fullscreen mode Exit fullscreen mode

Linux VM, start to finish:

# System prep
sudo apt-get update && sudo apt-get upgrade -y
sudo apt-get install -y containerd containernetworking-plugins

sudo mkdir -p /etc/containerd
sudo containerd config default | sudo tee /etc/containerd/config.toml
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl restart containerd

sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

sudo modprobe overlay
sudo modprobe br_netfilter
sudo tee /etc/sysctl.d/kubernetes.conf <<EOF
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
sudo sysctl --system

# Kubernetes install
sudo apt-get install -y apt-transport-https ca-certificates curl gnupg
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet=1.29.6-1.1 kubeadm=1.29.6-1.1 kubectl=1.29.6-1.1 --allow-downgrades --allow-change-held-packages
sudo apt-mark hold kubelet kubeadm kubectl

# Master init
sudo kubeadm init --pod-network-cidr=192.168.0.0/16 --apiserver-advertise-address=192.168.56.10 --node-name k8s-master

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# CNI (plain bridge)
sudo mkdir -p /etc/cni/net.d
sudo tee /etc/cni/net.d/10-bridge.conf <<EOF
{
  "cniVersion": "1.0.0",
  "name": "bridge",
  "type": "bridge",
  "bridge": "cni0",
  "isGateway": true,
  "ipMasq": true,
  "ipam": {
    "type": "host-local",
    "subnet": "10.244.0.0/16",
    "routes": [{ "dst": "0.0.0.0/0" }]
  }
}
EOF
sudo systemctl restart containerd

# Worker join
sudo kubeadm join 192.168.56.10:6443 --token <TOKEN> --discovery-token-ca-cert-hash sha256:<HASH>

# Verification
kubectl get nodes -o wide
kubectl get pods -A
kubectl create deployment nginx --image=nginx --replicas=3
kubectl get pods -o wide
Enter fullscreen mode Exit fullscreen mode

That's the whole weekend. One control plane, one worker, three nginx pods scheduled across both — and about eight hours of the kind of debugging that teaches you more than a tutorial ever would.

Top comments (0)