DEV Community

Cover image for [Lab Notes] Kubernetes the Hard Way, For Real This Time (Step 08)
Luger Lex Pit-og
Luger Lex Pit-og

Posted on

[Lab Notes] Kubernetes the Hard Way, For Real This Time (Step 08)

#kubernetes #proxmox #beginners #devops

Continuing my Kubernetes the Hard Way homelab build. Steps 01-07 are already done, this covers step 08.

Original guide: 08-bootstrapping-kubernetes-controllers.md

Thoughts I had while doing this

Much like the previous step with etcd, this one is about installing more components. Mainly the Kubernetes API Server, Scheduler, and Controller Manager.

Same pattern as before: copy the binaries over to the controller node, configure them, and each one becomes a running Linux service.

What I was trying to figure out was what each of these components is for. I already get etcd, it stores the state. Here's my attempt at explaining the other three:

  • API Server - this is where every API call from the other Kubernetes components ends up. If you want to do anything in the cluster, it has to go through here, since it's the only component that reads/writes directly to etcd. That's also why it's loaded with security config (ca.crt, ca.key, etc.). It's the gatekeeper in front of the database.
  • Scheduler - decides which node a pod should run on. It just checks whether a pod already has an assigned node; if not, it calls the API server to assign one. The API server is the one that actually writes that update to etcd, the scheduler just makes the decision and reports it.
  • Controller Manager - as far as I understand it, this is really just a bundle of other controllers. If the API server is the one directly reading/writing etcd, and the scheduler just decides pod placement, the controller manager is just "everything else." A couple examples of what that covers:
    • "One of the physical servers running our pods went down, let me do something about it."
    • "A job failed inside a pod, let me retry it."

A gotcha along the way: apiserver wouldn't start

Hit a wall trying to get kube-apiserver up:

root@server:~# systemctl is-active kube-apiserver
activating
root@server:~# systemctl status kube-apiserver
● kube-apiserver.service - Kubernetes API Server
     Loaded: loaded (/etc/systemd/system/kube-apiserver.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Thu 2026-09-17 13:13:42 UTC; 89ms ago
       Docs: https://github.com/kubernetes/kubernetes
    Process: 56537 ExecStart=/usr/local/bin/kube-apiserver --allow-privileged=true ... (code=exited, status=1/FAILURE)
   Main PID: 56537 (code=exited, status=1/FAILURE)
        CPU: 48ms

Sep 17 13:13:42 server systemd[1]: kube-apiserver.service: Main process exited, code=exited, status=1/FAILURE
Sep 17 13:13:42 server systemd[1]: kube-apiserver.service: Failed with result 'exit-code'.
Enter fullscreen mode Exit fullscreen mode

Checked the logs for the actual error:

root@server:~# journalctl -u kube-apiserver
...
Sep 17 13:08:07 server kube-apiserver[55147]: I0917 13:08:07.493634   55147 server.go:147] Version: v1.32.3
Sep 17 13:08:07 server kube-apiserver[55147]: E0917 13:08:07.493805   55147 run.go:72] "command failed" err="failed to create listener: failed to listen on 0.0.0.0:6443: listen tcp 0.0.0.0:6443: bind: address already in use"
Sep 17 13:08:07 server systemd[1]: kube-apiserver.service: Main process exited, code=exited, status=1/FAILURE
Sep 17 13:08:07 server systemd[1]: kube-apiserver.service: Failed with result 'exit-code'.
Sep 17 13:08:12 server systemd[1]: kube-apiserver.service: Scheduled restart job, restart counter is at 1.
...
Enter fullscreen mode Exit fullscreen mode

Something else was already bound to port 6443:

root@server:~# sudo ss -tlnp | grep 6443
LISTEN 0      4096               *:6443             *:*    users:(("k3s-server",pid=623,fd=13))
Enter fullscreen mode Exit fullscreen mode

Turned out I'd tried setting up k3s on this same machine a few months back and forgot about it. Stopped and disabled the leftover service:

root@server:~# sudo systemctl stop k3s
sudo systemctl disable k3s
Removed /etc/systemd/system/multi-user.target.wants/k3s.service.
Enter fullscreen mode Exit fullscreen mode

After that, kube-apiserver started cleanly:

root@server:~# systemctl is-active kube-apiserver
active
root@server:~# kubectl cluster-info \
  --kubeconfig admin.kubeconfig
Kubernetes control plane is running at https://127.0.0.1:6443
Enter fullscreen mode Exit fullscreen mode

Now here's the actual step.

Prerequisites 

root@luger-VirtualBox:~/kubernetes-the-hard-way# scp \
  downloads/controller/kube-apiserver \
  downloads/controller/kube-controller-manager \
  downloads/controller/kube-scheduler \
  downloads/client/kubectl \
  units/kube-apiserver.service \
  units/kube-controller-manager.service \
  units/kube-scheduler.service \
  configs/kube-scheduler.yaml \
  configs/kube-apiserver-to-kubelet.yaml \
  root@server:~/
kube-apiserver                     100%   89MB  90.4MB/s   00:00
kube-controller-manager            100%   82MB  93.6MB/s   00:00
kube-scheduler                     100%   63MB 157.4MB/s   00:00
kubectl                            100%   55MB 153.5MB/s   00:00
kube-apiserver.service             100% 1374     1.6MB/s   00:00
kube-controller-manager.service    100%  735   936.9KB/s   00:00
kube-scheduler.service             100%  281   349.7KB/s   00:00
kube-scheduler.yaml                100%  191   231.0KB/s   00:00
kube-apiserver-to-kubelet.yaml     100%  727   952.4KB/s   00:00
Enter fullscreen mode Exit fullscreen mode

Provision the Kubernetes control plane 

root@server:~# mkdir -p /etc/kubernetes/config
Enter fullscreen mode Exit fullscreen mode

Install the Kubernetes controller binaries 

root@server:~# {
  mv kube-apiserver \
    kube-controller-manager \
    kube-scheduler kubectl \
    /usr/local/bin/
}
Enter fullscreen mode Exit fullscreen mode

Configure the Kubernetes API Server 

root@server:~# {
  mkdir -p /var/lib/kubernetes/

  mv ca.crt ca.key \
    kube-api-server.key kube-api-server.crt \
    service-accounts.key service-accounts.crt \
    encryption-config.yaml \
    /var/lib/kubernetes/
}

root@server:~# mv kube-apiserver.service \
  /etc/systemd/system/kube-apiserver.service
Enter fullscreen mode Exit fullscreen mode

Configure the Kubernetes Controller Manager 

root@server:~# mv kube-controller-manager.kubeconfig /var/lib/kubernetes/
root@server:~# mv kube-controller-manager.service /etc/systemd/system/
Enter fullscreen mode Exit fullscreen mode

Configure the Kubernetes Scheduler 

root@server:~# mv kube-scheduler.kubeconfig /var/lib/kubernetes/
root@server:~# mv kube-scheduler.yaml /etc/kubernetes/config/
root@server:~# mv kube-scheduler.service /etc/systemd/system/
Enter fullscreen mode Exit fullscreen mode

Start the controller services 

root@server:~# {
  systemctl daemon-reload

  systemctl enable kube-apiserver \
    kube-controller-manager kube-scheduler

  systemctl start kube-apiserver \
    kube-controller-manager kube-scheduler
}
Created symlink /etc/systemd/system/multi-user.target.wants/kube-apiserver.service → /etc/systemd/system/kube-apiserver.service.
Created symlink /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service → /etc/systemd/system/kube-controller-manager.service.
Created symlink /etc/systemd/system/multi-user.target.wants/kube-scheduler.service → /etc/systemd/system/kube-scheduler.service.
Enter fullscreen mode Exit fullscreen mode

API server verification 

root@server:~# kubectl cluster-info \
  --kubeconfig admin.kubeconfig
Kubernetes control plane is running at https://127.0.0.1:6443

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Enter fullscreen mode Exit fullscreen mode

RBAC for kubelet authorization

This part just makes sure the API server is allowed to talk to the kubelet APIs on the worker nodes. As I understand it, the command here applies whatever's defined in kube-apiserver-to-kubelet.yaml to the cluster:

root@server:~# kubectl apply -f kube-apiserver-to-kubelet.yaml \
  --kubeconfig admin.kubeconfig
clusterrole.rbac.authorization.k8s.io/system:kube-apiserver-to-kubelet created
clusterrolebinding.rbac.authorization.k8s.io/system:kube-apiserver created
Enter fullscreen mode Exit fullscreen mode

Verification 

root@luger-VirtualBox:~/kubernetes-the-hard-way# curl --cacert ca.crt \
  https://server.kubernetes.local:6443/version
{
  "major": "1",
  "minor": "32",
  "gitVersion": "v1.32.3",
  "gitCommit": "32cc146f75aad04beaaa245a7157eb35063a9f99",
  "gitTreeState": "clean",
  "buildDate": "2025-03-11T19:52:21Z",
  "goVersion": "go1.23.6",
  "compiler": "gc",
  "platform": "linux/amd64"
}
Enter fullscreen mode Exit fullscreen mode

TLS is working end-to-end. The jumpbox hit the API server over HTTPS using only the CA cert.

Summary

Setup the three control plane components: API server, controller manager, and scheduler as systemd services on the controller node, and placed RBAC so the API server can talk to kubelets on the workers.

Top comments (0)