[Lab Notes] Kubernetes the Hard Way, For Real This Time (Step 05)

Continuing my Kubernetes the Hard Way homelab build. Steps 1-4 are already done, this is for step 05.

Original guide: 05-kubernetes-configuration-files.md

Thoughts I had while doing this

This step is about creating kubeconfigs, basically the .conf files each Kubernetes service uses to know how to connect. Each service (scheduler, proxy, etc.) gets its own kubeconfig, and each one bundles in the cert files from step 04 so the connection to the API server (the control node) is authenticated and encrypted.

A gotcha along the way

Since I'm on Ubuntu instead of the guide's recommended Debian, I keep running into small differences. This step's version: my /etc/hosts changes from step 03 weren't surviving a reboot. Turned out Ubuntu's cloud-init manages /etc/hosts by default and quietly overwrites it:

# Your system has configured 'manage_etc_hosts' as True.
# As a result, if you wish for changes to this file to persist
# then you will need to either
# a.) make changes to the master file in /etc/cloud/templates/hosts.debian.tmpl
# b.) change or remove the value of 'manage_etc_hosts' in
#     /etc/cloud/cloud.cfg or cloud-config from user-data

Fixed it by updating the cloud-init template with the Kubernetes hostname entries instead of /etc/hosts directly:

# edited /etc/cloud/templates/hosts.debian.tmpl to include the Kubernetes host entries
sudo cloud-init clean --logs && sudo reboot

That made the entries persist across reboots.

Now here's the actual step.

Quick TLDR though: I just followed the guide to generate the kubeconfigs, and they worked as expected. Let me take a shot at explaining what these commands are actually doing.

Essentially, there are 4 commands run for each kubeconfig file, and all four are just pointed at that one file. Think of the kubeconfig as a plain text file, these commands just append to or edit pieces of it. That's really all that's happening:

set-cluster: adds the API server's URL and the CA cert, so the client knows who it's talking to and can verify it's really the API server.
set-credentials: adds the client's own cert and key, so the client can prove who it is to the API server.
set-context: bundles a cluster and a set of credentials together under a named context, so "use this identity to talk to this cluster" becomes one reusable label instead of two separate things. Basically, it just takes the outputs of set-cluster and set-credentials and bundles them together, that's really all set-context is doing.
use-context: sets that context as the active/default one, so any command using this kubeconfig automatically uses that cluster + credentials pair without you specifying them every time. Basically, you're just telling the kubeconfig "use the context I created in set-context as the default whenever a client loads this file."

The kubelet kubeconfig

root@luger-VirtualBox:~/kubernetes-the-hard-way# for host in node-0 node-1; do
  kubectl config set-cluster kubernetes-the-hard-way \
    --certificate-authority=ca.crt \
    --embed-certs=true \
    --server=https://server.kubernetes.local:6443 \
    --kubeconfig=${host}.kubeconfig

  kubectl config set-credentials system:node:${host} \
    --client-certificate=${host}.crt \
    --client-key=${host}.key \
    --embed-certs=true \
    --kubeconfig=${host}.kubeconfig

  kubectl config set-context default \
    --cluster=kubernetes-the-hard-way \
    --user=system:node:${host} \
    --kubeconfig=${host}.kubeconfig

  kubectl config use-context default \
    --kubeconfig=${host}.kubeconfig
done
Cluster "kubernetes-the-hard-way" set.
User "system:node:node-0" set.
Context "default" created.
Switched to context "default".
Cluster "kubernetes-the-hard-way" set.
User "system:node:node-1" set.
Context "default" created.
Switched to context "default".

The kube-proxy kubeconfig

root@luger-VirtualBox:~/kubernetes-the-hard-way# {
  kubectl config set-cluster kubernetes-the-hard-way \
    --certificate-authority=ca.crt \
    --embed-certs=true \
    --server=https://server.kubernetes.local:6443 \
    --kubeconfig=kube-proxy.kubeconfig

  kubectl config set-credentials system:kube-proxy \
    --client-certificate=kube-proxy.crt \
    --client-key=kube-proxy.key \
    --embed-certs=true \
    --kubeconfig=kube-proxy.kubeconfig

  kubectl config set-context default \
    --cluster=kubernetes-the-hard-way \
    --user=system:kube-proxy \
    --kubeconfig=kube-proxy.kubeconfig

  kubectl config use-context default \
    --kubeconfig=kube-proxy.kubeconfig
}
Cluster "kubernetes-the-hard-way" set.
User "system:kube-proxy" set.
Context "default" created.
Switched to context "default".

The kube-controller-manager kubeconfig

root@luger-VirtualBox:~/kubernetes-the-hard-way# {
  kubectl config set-cluster kubernetes-the-hard-way \
    --certificate-authority=ca.crt \
    --embed-certs=true \
    --server=https://server.kubernetes.local:6443 \
    --kubeconfig=kube-controller-manager.kubeconfig

  kubectl config set-credentials system:kube-controller-manager \
    --client-certificate=kube-controller-manager.crt \
    --client-key=kube-controller-manager.key \
    --embed-certs=true \
    --kubeconfig=kube-controller-manager.kubeconfig

  kubectl config set-context default \
    --cluster=kubernetes-the-hard-way \
    --user=system:kube-controller-manager \
    --kubeconfig=kube-controller-manager.kubeconfig

  kubectl config use-context default \
    --kubeconfig=kube-controller-manager.kubeconfig
}
Cluster "kubernetes-the-hard-way" set.
User "system:kube-controller-manager" set.
Context "default" created.
Switched to context "default".

The kube-scheduler kubeconfig

root@luger-VirtualBox:~/kubernetes-the-hard-way# {
  kubectl config set-cluster kubernetes-the-hard-way \
    --certificate-authority=ca.crt \
    --embed-certs=true \
    --server=https://server.kubernetes.local:6443 \
    --kubeconfig=kube-scheduler.kubeconfig

  kubectl config set-credentials system:kube-scheduler \
    --client-certificate=kube-scheduler.crt \
    --client-key=kube-scheduler.key \
    --embed-certs=true \
    --kubeconfig=kube-scheduler.kubeconfig

  kubectl config set-context default \
    --cluster=kubernetes-the-hard-way \
    --user=system:kube-scheduler \
    --kubeconfig=kube-scheduler.kubeconfig

  kubectl config use-context default \
    --kubeconfig=kube-scheduler.kubeconfig
}
Cluster "kubernetes-the-hard-way" set.
User "system:kube-scheduler" set.
Context "default" created.
Switched to context "default".

The admin kubeconfig

root@luger-VirtualBox:~/kubernetes-the-hard-way# {
  kubectl config set-cluster kubernetes-the-hard-way \
    --certificate-authority=ca.crt \
    --embed-certs=true \
    --server=https://127.0.0.1:6443 \
    --kubeconfig=admin.kubeconfig

  kubectl config set-credentials admin \
    --client-certificate=admin.crt \
    --client-key=admin.key \
    --embed-certs=true \
    --kubeconfig=admin.kubeconfig

  kubectl config set-context default \
    --cluster=kubernetes-the-hard-way \
    --user=admin \
    --kubeconfig=admin.kubeconfig

  kubectl config use-context default \
    --kubeconfig=admin.kubeconfig
}
Cluster "kubernetes-the-hard-way" set.
User "admin" set.
Context "default" created.
Switched to context "default".

Kubeconfig files generated

root@luger-VirtualBox:~/kubernetes-the-hard-way# ls -ltr *.kubeconfig
-rw------- 1 root root 10157 Sep 16 20:59 node-0.kubeconfig
-rw------- 1 root root 10161 Sep 16 20:59 node-1.kubeconfig
-rw------- 1 root root 10187 Sep 16 21:14 kube-proxy.kubeconfig
-rw------- 1 root root 10305 Sep 16 21:14 kube-controller-manager.kubeconfig
-rw------- 1 root root 10231 Sep 16 21:14 kube-scheduler.kubeconfig
-rw------- 1 root root  9953 Sep 16 21:14 admin.kubeconfig

Distribute the Kubernetes configuration files

root@luger-VirtualBox:~/kubernetes-the-hard-way# for host in node-0 node-1; do
  ssh root@${host} "mkdir -p /var/lib/{kube-proxy,kubelet}"

  scp kube-proxy.kubeconfig \
    root@${host}:/var/lib/kube-proxy/kubeconfig

  scp ${host}.kubeconfig \
    root@${host}:/var/lib/kubelet/kubeconfig
done

Summary

Step 05 built a kubeconfig for every cluster component that needs to talk to the API server, kubelet on each worker, kube-proxy, kube-controller-manager, kube-scheduler, and admin. Each one embedding the right cert/key from step 04 so the connection is authenticated. Also patched a Ubuntu-specific gotcha where cloud-init was silently reverting /etc/hosts changes on reboot.