DEV Community

Cover image for [Lab Notes] Kubernetes the Hard Way, For Real This Time (Step 09)
Luger Lex Pit-og
Luger Lex Pit-og

Posted on

[Lab Notes] Kubernetes the Hard Way, For Real This Time (Step 09)

#kubernetes #devops #beginners #sre

Continuing my Kubernetes the Hard Way homelab build. Steps 01-08 are already done, this covers step 09.

Original guide: 09-bootstrapping-kubernetes-workers.md

Thoughts I had while doing this

Same pattern as the last two steps I worked on, just on the worker nodes this time: copy files over, install and configure them, then start them as services. Quick TLDR:

  1. From the jumpbox, make a few directories and copy the worker binaries, configs, and service files to each worker node
  2. On each worker, install the OS dependencies, disable swap, and create the directories the components need
  3. Set up pod networking (CNI), containerd, the kubelet, and kube-proxy
  4. Start the worker services

The worker steps below were done on node-0 first, then repeated on node-1.

Prerequisites (on the jumpbox)

From what I can tell, this part is mostly making directories and copying files over to the worker nodes. First, the per-node configs, with each node's pod subnet filled in from machines.txt:

root@jumpbox:~/kubernetes-the-hard-way# for HOST in node-0 node-1; do
  SUBNET=$(grep ${HOST} machines.txt | cut -d " " -f 4)
  sed "s|SUBNET|$SUBNET|g" \
    configs/10-bridge.conf > 10-bridge.conf

  sed "s|SUBNET|$SUBNET|g" \
    configs/kubelet-config.yaml > kubelet-config.yaml

  scp 10-bridge.conf kubelet-config.yaml \
  root@${HOST}:~/
done
10-bridge.conf         100%  265   525.3KB/s   00:00
kubelet-config.yaml    100%  610     1.7MB/s   00:00
10-bridge.conf         100%  265   538.1KB/s   00:00
kubelet-config.yaml    100%  610     1.6MB/s   00:00
Enter fullscreen mode Exit fullscreen mode

Then the binaries, configs, and service files:

root@jumpbox:~/kubernetes-the-hard-way# for HOST in node-0 node-1; do
  scp \
    downloads/worker/* \
    downloads/client/kubectl \
    configs/99-loopback.conf \
    configs/containerd-config.toml \
    configs/kube-proxy-config.yaml \
    units/containerd.service \
    units/kubelet.service \
    units/kube-proxy.service \
    root@${HOST}:~/
done
containerd                 100%   56MB 138.3MB/s   00:00
containerd-shim-runc-v2    100% 8060KB 125.4MB/s   00:00
containerd-stress          100%   22MB 117.8MB/s   00:00
crictl                     100%   38MB 129.3MB/s   00:00
ctr                        100%   23MB 103.9MB/s   00:00
kubelet                    100%   74MB  87.5MB/s   00:00
kube-proxy                 100%   64MB  79.6MB/s   00:00
runc                       100%   11MB 131.8MB/s   00:00
kubectl                    100%   55MB 142.0MB/s   00:00
...
(same again for node-1)
Enter fullscreen mode Exit fullscreen mode

A gotcha here: the CNI plugins copy failed 

root@jumpbox:~/kubernetes-the-hard-way# for HOST in node-0 node-1; do
  scp \
    downloads/cni-plugins/* \
    root@${HOST}:~/cni-plugins/
done
scp: /root/cni-plugins/: No such file or directory
scp: /root/cni-plugins/: No such file or directory
Enter fullscreen mode Exit fullscreen mode

The cni-plugins directory didn't exist on the workers yet. Fix was to create it on each worker node, then re-run the same command from the jumpbox:

root@jumpbox:~/kubernetes-the-hard-way# for HOST in node-0 node-1; do
  scp \
    downloads/cni-plugins/* \
    root@${HOST}:~/cni-plugins/
done
bandwidth     100% 4546KB 137.5MB/s   00:00
bridge        100% 5163KB 133.9MB/s   00:00
dhcp          100%   12MB 153.0MB/s   00:00
host-local    100% 3965KB 124.6MB/s   00:00
loopback      100% 4018KB 131.4MB/s   00:00
portmap       100% 4603KB 139.1MB/s   00:00
...
(same again for node-1)
Enter fullscreen mode Exit fullscreen mode

Provision a Kubernetes worker node

This part is just installing OS dependencies, disabling swap, and creating directories for the binaries. Everything from here on runs on the worker node itself.

Install the dependencies:

root@node-0:~# {
  apt-get update
  apt-get -y install socat conntrack ipset kmod
}
...
The following NEW packages will be installed:
  conntrack ipset libipset13 socat
0 upgraded, 4 newly installed, 0 to remove and 72 not upgraded.
...
Enter fullscreen mode Exit fullscreen mode

Check that swap is off (no output means it's already disabled):

root@node-0:~# swapon --show
Enter fullscreen mode Exit fullscreen mode

Create the directories and move the binaries into place:

root@node-0:~# mkdir -p \
  /etc/cni/net.d \
  /opt/cni/bin \
  /var/lib/kubelet \
  /var/lib/kube-proxy \
  /var/lib/kubernetes \
  /var/run/kubernetes
root@node-0:~# {
  mv crictl kube-proxy kubelet runc \
    /usr/local/bin/
  mv containerd containerd-shim-runc-v2 containerd-stress /bin/
  mv cni-plugins/* /opt/cni/bin/
}
Enter fullscreen mode Exit fullscreen mode

Configure CNI networking

This part sets up the networking so pods can communicate with each other:

root@node-0:~# mv 10-bridge.conf 99-loopback.conf /etc/cni/net.d/
root@node-0:~# {
  modprobe br-netfilter
  echo "br-netfilter" >> /etc/modules-load.d/modules.conf
}
root@node-0:~# {
  echo "net.bridge.bridge-nf-call-iptables = 1" \
    >> /etc/sysctl.d/kubernetes.conf
  echo "net.bridge.bridge-nf-call-ip6tables = 1" \
    >> /etc/sysctl.d/kubernetes.conf
  sysctl -p /etc/sysctl.d/kubernetes.conf
}
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
Enter fullscreen mode Exit fullscreen mode

What's CNI actually doing here?

The CNI part was more interesting than the rest of this step. It wasn't just copying files and binaries around, so I dug into what it's actually doing.

From what I understand, CNI (Container Network Interface) is the part that configures networking for pods so they can communicate with each other. The plugin binaries in /opt/cni/bin do the actual work, and the config files in /etc/cni/net.d tell the runtime which plugins to call and how. Breaking down what each piece does:

  • The bridge: 10-bridge.conf sets up a "bridge", basically a virtual switch on the node that pods plug into so they can talk to each other. Each node has its own pod subnet (the SUBNET value filled in from machines.txt in the prerequisites), and pods get their IPs from it.
  • The loopback: 99-loopback.conf gives each pod its own loopback interface.
  • The firewall part: the br-netfilter module and the two sysctl settings make traffic crossing the bridge pass through the host's iptables rules. This matters because kube-proxy relies on iptables to route traffic to Services.

Also, it seems that each bridge only connects the pods on its own node.

Configure containerd 

root@node-0:~# {
  mkdir -p /etc/containerd/
  mv containerd-config.toml /etc/containerd/config.toml
  mv containerd.service /etc/systemd/system/
}
Enter fullscreen mode Exit fullscreen mode

Configure the kubelet 

root@node-0:~# {
  mv kubelet-config.yaml /var/lib/kubelet/
  mv kubelet.service /etc/systemd/system/
}
Enter fullscreen mode Exit fullscreen mode

Configure the Kubernetes proxy 

root@node-0:~# {
  mv kube-proxy-config.yaml /var/lib/kube-proxy/
  mv kube-proxy.service /etc/systemd/system/
}
Enter fullscreen mode Exit fullscreen mode

Start the worker services 

root@node-0:~# {
  systemctl daemon-reload
  systemctl enable containerd kubelet kube-proxy
  systemctl start containerd kubelet kube-proxy
}
Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /etc/systemd/system/containerd.service.
Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service → /etc/systemd/system/kubelet.service.
Created symlink /etc/systemd/system/multi-user.target.wants/kube-proxy.service → /etc/systemd/system/kube-proxy.service.
root@node-0:~# systemctl is-active kubelet
active
Enter fullscreen mode Exit fullscreen mode

Another gotcha: the nodes never showed up

The kubelet was running, but when I checked from the controller, the cluster had no nodes:

root@jumpbox:~/kubernetes-the-hard-way# ssh root@server "kubectl get nodes \
  --kubeconfig admin.kubeconfig"
No resources found
Enter fullscreen mode Exit fullscreen mode

The kubelet logs on the worker showed why. The API server kept rejecting it with "forbidden" errors:

Oct 25 19:41:52 k8s-solo-w1 kubelet[706]: E1025 19:41:52.168304  706 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"k8s-solo-w1\" is forbidden: User >
Oct 25 19:41:52 k8s-solo-w1 kubelet[706]: I1025 19:41:52.378476  706 kubelet_node_status.go:75] "Attempting to register node" node="k8s-solo-w1"
Oct 25 19:41:52 k8s-solo-w1 kubelet[706]: E1025 19:41:52.381530  706 kubelet_node_status.go:113] "Unable to register node with API server, error getting existing node" err="nodes \"k8s-solo-w1\" is f>
Enter fullscreen mode Exit fullscreen mode

The worker was still calling itself by its old VM name (k8s-solo-w1), but its kubelet certificate was issued for node-0 back in step 04. The fix was to set the hostname to the name registered in the certificate, then restart the kubelet:

hostnamectl set-hostname node-0
systemctl restart kubelet
Enter fullscreen mode Exit fullscreen mode

Honestly, I thought this wasn't needed since I'd already set up /etc/hosts in step 03 with entries for node-0 and node-1. But as I understand it now, /etc/hosts only maps names to IPs. The kubelet picks its node name from the machine's own hostname, and the API server only lets a certificate for system:node:node-0 manage the node object named node-0. So the hostname itself has to match the certificate, not just resolve.

Repeated the same fix on the second worker (node-1), and both nodes came up:

root@jumpbox:~/kubernetes-the-hard-way# ssh root@server \
  "kubectl get nodes \
  --kubeconfig admin.kubeconfig"
NAME     STATUS   ROLES    AGE     VERSION
node-0   Ready    <none>   8d      v1.32.3
node-1   Ready    <none>   4m54s   v1.32.3
Enter fullscreen mode Exit fullscreen mode

Summary

Step 09 got both worker nodes running containerd, the kubelet, and kube-proxy, with CNI networking configured for pod communication. Both nodes now show Ready.

Top comments (0)