Deploying-a-Kubernetes-Cluster-with-kubeadm-on-RHEL-9-CentOS-10
The guide is based on a hands-on implementation and is intended to serve as both a learning resource and a reusable deployment reference for students, DevOps engineers, and anyone building a Kubernetes lab environment.
Deploying a Kubernetes Cluster
with kubeadm on RHEL 9
A Corrected & Expanded Reference Build
1 Control Plane · 2 Worker Nodes · VMware
| AUTHOR | Muhammad Mursleen Mamaoon |
|---|---|
| PLATFORM | RHEL 9 |
| ORCHESTRATION | kubeadm |
| CONTAINER RUNTIME | containerd |
| CLUSTER NETWORKING | Calico CNI |
| DOCUMENT TYPE | Deployment Runbook & Reference Architecture |
| EDITION | Corrected Edition - networking, firewall, and lifecycle fixes applied |
Contents
3.1 VMware-Specific Gotcha: Clone Hygiene 4
4. Step 1 - Hostnames and Name Resolution [ALL] 4
5. Step 2 - Time Synchronization [ALL] 5
6. Step 3 - Disable Swap [ALL] 5
8. Step 5 - Kernel Modules [ALL] 5
9. Step 6 - Kernel Network Parameters [ALL] 6
10. Step 7 - Firewall Configuration [ALL] 6
11. Step 8 - Install and Configure containerd [ALL] 7
12. Step 9 - Install Kubernetes Components [ALL] 8
13. Step 10 - Initialize the Control Plane [MASTER] 8
14. Step 11 - Join Worker Nodes [WORKER] 9
15. Step 12 - Install Calico CNI [MASTER] 9
16. Step 13 - Validate the Cluster [MASTER] 9
18. What Changes for Production 10
Right-click the table above and choose "Update Field" after opening in Word if page numbers do not populate automatically.
1. Purpose and Scope
This document is a complete, validated procedure for bootstrapping a Kubernetes cluster using kubeadm, containerd, and Calico on RHEL 9, deployed across three VMware virtual machines. It is written as both an execution runbook and a design reference - each step includes not just the command, but the reasoning behind it, so it can be adapted safely rather than copy-pasted blind.
This edition corrects several issues identified in the original draft, most notably a pod-network / host-network CIDR collision and a firewall rule that blocked Calico's actual encapsulation mode, and adds operational detail commonly missing from kubeadm walkthroughs: VMware clone hygiene, time synchronization, version pinning, and a fuller validation pass.
SCOPE NOTE
As written, this procedure produces a functional lab / reference cluster - suitable for learning, testing, and demonstrating the mechanics of kubeadm. It is not, as-is, a hardened production topology: a single control-plane node is a single point of failure, SELinux is relaxed rather than properly policed, and there is no external etcd or highly-available API server. Section 18 outlines what changes for production use.
2. Lab Topology
| Role | Hostname | IP Address | vCPU | RAM |
|---|---|---|---|---|
| Control Plane | k8s-master | 192.168.1.10 | 2+ | 2 GB+ |
| Worker Node 1 | k8s-worker1 | 192.168.1.11 | 2+ | 2 GB+ |
| Worker Node 2 | k8s-worker2 | 192.168.1.12 | 2+ | 2 GB+ |
Stack: RHEL 9 · VMware Workstation/ESXi · kubeadm · containerd · Calico CNI
Notation Used Throughout
- [ALL] - run on every node
- [MASTER] - control plane only
- [WORKER] - worker nodes only
3. Prerequisites
- Three RHEL 9 VMs with routable connectivity to each other and to the internet, for package and container image pulls.
- SSH access with sudo privileges on all nodes.
- A hard minimum of 2 vCPUs on the control-plane node. This is not a soft recommendation - kubeadm init runs a preflight check and refuses to proceed on a single-vCPU control plane.
- A pod network CIDR that does not overlap the physical/VM network. The original draft used 192.168.0.0/16 for pods on a 192.168.1.0/24 host network - those ranges overlap. This guide uses 10.244.0.0/16 instead (see Section 10).
3.1 VMware-Specific Gotcha: Clone Hygiene
If any of these three VMs were created by cloning a template - extremely common in VMware labs - they can inherit an identical /etc/machine-id and product UUID. Kubernetes uses these values to distinguish nodes, and duplicates cause node registration conflicts, DHCP lease collisions, and kubelet identity confusion that are painful to debug after the fact. Check uniqueness before proceeding:
# [ALL] Every value below must differ across all three nodes
cat /etc/machine-id
cat /sys/class/dmi/id/product_uuid
ip link show | grep ether
If any values match across nodes, regenerate the machine ID on the affected VM:
# [ALL, only if duplicated]
sudo rm -f /etc/machine-id /var/lib/dbus/machine-id
sudo systemd-machine-id-setup
sudo ln -s /etc/machine-id /var/lib/dbus/machine-id
Duplicate product_uuid values require a fix at the hypervisor level - regenerate the BIOS UUID in the VM's .vmx settings, or re-clone from the template with "new identity" options enabled.
4. Step 1 - Hostnames and Name Resolution [ALL]
# Run the appropriate line on each respective node
sudo hostnamectl set-hostname k8s-master # on the control plane
sudo hostnamectl set-hostname k8s-worker1 # on worker 1
sudo hostnamectl set-hostname k8s-worker2 # on worker 2
Add all three nodes to /etc/hosts on every node:
| sudo tee -a /etc/hosts <
192.168.1.10 k8s-master
192.168.1.11 k8s-worker1
192.168.1.12 k8s-worker2
EOF | |
| --- | |
| IDEMPOTENCY NOTE
tee -a appends unconditionally. Re-running this step produces duplicate lines - harmless for resolution, but if scripting this for repeatable builds, guard it with a grep -qxF check first. | |
5. Step 2 - Time Synchronization [ALL]
ADDED IN THIS EDITION
The original draft did not include this step, and it should have. Kubernetes leans heavily on TLS certificate validity windows and etcd's Raft consensus, both sensitive to clock drift between nodes.
RHEL 9 ships chrony by default - confirm it is active:
sudo systemctl enable --now chronyd
chronyc tracking
Confirm System time shows a sub-second offset before continuing.
6. Step 3 - Disable Swap [ALL]
The kubelet refuses to start with swap enabled, unless explicitly configured via NodeSwap / MemorySwap, which is out of scope for this build.
sudo swapoff -a
sudo sed -i '/\sswap\s/s/^/#/' /etc/fstab
7. Step 4 - SELinux [ALL]
| sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config | |
| --- | |
| PRODUCTION CAUTION
Permissive mode disables SELinux enforcement cluster-wide, which is acceptable for a lab but removes a real security boundary in production. If extending this design, install the container-selinux package (ships proper policies for containerd/CRI-O) and keep SELinux in enforcing mode - most modern kubeadm/containerd combinations on RHEL run fine enforced, provided the runtime's SELinux policy is installed correctly. | |
8. Step 5 - Kernel Modules [ALL]
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter
9. Step 6 - Kernel Network Parameters [ALL]
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
sudo sysctl --system
10. Step 7 - Firewall Configuration [ALL]
CORRECTED IN THIS EDITION
The original port list included two deprecated ports and opened the wrong protocol for Calico's default overlay mode.
- Dropped 10251/tcp and 10252/tcp - pre-1.19 insecure HTTP endpoints for kube-scheduler and kube-controller-manager, replaced by the authenticated endpoints on 10259 and 10257. Modern kubeadm does not expose the old ones at all.
- Replaced 4789/udp (VXLAN) with the IP-in-IP protocol - Calico's default manifest (calico.yaml) uses IP-in-IP encapsulation, not VXLAN, unless explicitly reconfigured. Opening only 4789/udp, as the original draft did, leaves the actual overlay traffic blocked by the firewall.
Control Plane
sudo firewall-cmd --permanent --add-port={6443,2379,2380,10250,10257,10259,179}/tcp
sudo firewall-cmd --permanent --add-protocol=ipip
sudo firewall-cmd --reload
Worker Nodes
sudo firewall-cmd --permanent --add-port={10250,179,30000-32767}/tcp
sudo firewall-cmd --permanent --add-protocol=ipip
sudo firewall-cmd --reload
Port Reference
| Port | Protocol | Purpose | Nodes |
|---|---|---|---|
| 6443 | TCP | kube-apiserver | Control plane |
| 2379-2380 | TCP | etcd client/peer | Control plane |
| 10250 | TCP | kubelet API | All |
| 10257 | TCP | kube-controller-manager (secure) | Control plane |
| 10259 | TCP | kube-scheduler (secure) | Control plane |
| 179 | TCP | Calico BGP | All |
| - | IP-in-IP (proto 4) | Calico overlay (default mode) | All |
| 30000-32767 | TCP | NodePort service range | All |
IF YOU SWITCH TO VXLAN
If Calico is later switched to VXLAN mode instead of IP-in-IP (common on networks that block IP-in-IP, e.g. some cloud VPCs), swap the --add-protocol=ipip rule for --add-port=4789/udp on every node. Don't open both unless intentionally running mixed mode.
11. Step 8 - Install and Configure containerd [ALL]
sudo dnf install -y dnf-plugins-core
sudo dnf config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo
sudo dnf install -y containerd.io
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml
# Kubernetes requires the systemd cgroup driver (RHEL 9 uses cgroup v2 by default)
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl enable --now containerd
sudo systemctl restart containerd
12. Step 9 - Install Kubernetes Components [ALL]
| VERSION UPDATED IN THIS EDITION
The original draft pinned v1.31, which has already reached end-of-life. Kubernetes follows an N-2 support policy across roughly four-month release cycles; the currently supported branches are 1.34, 1.35, and 1.36. Check kubernetes.io/releases for the current set before building, and substitute below. | |
| --- | |
| cat <
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.34/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.34/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF | |
Pin exact versions instead of floating on "latest." Without pinning, three separate dnf install runs across three nodes can silently pull three different patch versions if a new patch lands mid-build - tedious to debug later. Check what's available first:
dnf list --showduplicates kubelet --disableexcludes=kubernetes
Then install a matched version explicitly:
| sudo dnf install -y kubelet-1.34.* kubeadm-1.34.* kubectl-1.34.* --disableexcludes=kubernetes
sudo systemctl enable --now kubelet | |
| --- | |
| EXPECTED BEHAVIOR
It is normal for kubelet to sit in a crash-restart loop at this point (systemctl status kubelet will look unhappy) - it is waiting for kubeadm init / kubeadm join to supply a configuration. Not a bug. | |
13. Step 10 - Initialize the Control Plane [MASTER]
| CIDR CORRECTED IN THIS EDITION
The original draft used 192.168.0.0/16 for the pod network, which fully contains the 192.168.1.0/24 host network the VMs sit on. Overlapping ranges break routing between pods and real hosts - the kernel cannot disambiguate them. This guide uses a distinct, non-overlapping range. | |
| --- | |
| sudo kubeadm init \
--apiserver-advertise-address=192.168.1.10 \
--pod-network-cidr=10.244.0.0/16 | |
Save the kubeadm join … command printed at the end of initialization - it is required on both worker nodes, and expires after 24 hours by default.
Configure kubectl for your user:
mkdir -p \$HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf \$HOME/.kube/config
sudo chown \$(id -u):\$(id -g) \$HOME/.kube/config
14. Step 11 - Join Worker Nodes [WORKER]
sudo kubeadm join 192.168.1.10:6443 \
--token <token> \
--discovery-token-ca-cert-hash sha256:<hash>
If the token has expired (default TTL: 24 hours), generate a fresh one from the control plane:
kubeadm token create --print-join-command
15. Step 12 - Install Calico CNI [MASTER]
| CIDR ALIGNMENT REQUIRED
Calico's default manifest ships with its own IP pool CIDR (192.168.0.0/16), which must match the --pod-network-cidr given to kubeadm. Since that value changed to 10.244.0.0/16, the manifest needs the same edit before applying. | |
| --- | |
| curl -O https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml
sed -i 's/192.168.0.0\/16/10.244.0.0\/16/' calico.yaml
kubectl apply -f calico.yaml | |
Double-check the Calico version tag (v3.28.0) against what's current before building - Calico ships new minor releases on its own cadence, independent of Kubernetes.
16. Step 13 - Validate the Cluster [MASTER]
Nodes should transition to Ready once Calico's pods come up - this can take a minute or two:
kubectl get nodes -o wide
kubectl get pods -n kube-system -o wide
All calico-node, calico-kube-controllers, and coredns pods should be Running. If coredns is stuck Pending, it is almost always waiting on CNI - check the calico-node pods first.
Sanity-check core cluster health:
kubectl get componentstatuses # deprecated but still useful for a quick glance
kubectl cluster-info
Deploy and expose a smoke-test workload:
kubectl create deployment nginx-demo --image=nginx
kubectl expose deployment nginx-demo --type=NodePort --port=80
kubectl get svc nginx-demo
curl http://<any-worker-ip>:<node-port>
A default NGINX welcome page confirms the API server, scheduler, kubelet, CNI, and kube-proxy are all functioning end-to-end.
17. Troubleshooting
| Symptom | Cause / Fix |
|---|---|
| kubelet won't start / crashlooping before join | Expected pre-kubeadm init/join. If it persists after joining, confirm SystemdCgroup = true is actually set in /etc/containerd/config.toml and that containerd was restarted after the edit. |
| Worker stuck NotReady | Expected until Calico is applied. If it persists afterward, check kubectl get pods -n kube-system for calico-node crash loops - usually a firewall or IP-in-IP protocol issue (Section 10). |
| kubeadm join fails / times out | Verify firewall ports (Section 10), hostname resolution (Section 4), and that the join token/hash haven't expired. |
| Pods reach each other on the same node but not across nodes | Almost always the overlay protocol being blocked - confirm firewall-cmd --add-protocol=ipip was applied on every node, not just the control plane. |
| Two nodes seem to "merge" identity, or one registers then vanishes | Cloned-VM machine-id / UUID collision - see Section 3.1. |
| kubeadm init preflight fails on CPU count | Control-plane node has fewer than 2 vCPUs - a hard requirement, not advisory. |
| Cluster works but certs/etcd behave oddly under load | Check chronyc tracking on all nodes - clock drift across etcd peers causes subtle, hard-to-diagnose failures. |
| Lost the join command | kubeadm token create --print-join-command from the control plane, any time. |
18. What Changes for Production
This build is correct and internally consistent for a lab or reference deployment, but a few things are deliberately out of scope and worth naming explicitly if the design is extended:
- Single control-plane node = single point of failure. Production topologies run 3 (or 5) control-plane nodes behind a load balancer, with --control-plane-endpoint set to the LB's DNS name from the first kubeadm init, and --upload-certs for subsequent control-plane joins.
- etcd here is stacked on the control-plane node. Production clusters commonly run etcd on dedicated nodes for isolation and easier backup/restore.
- SELinux permissive should become enforcing with proper container SELinux policy.
- No backup strategy for etcd snapshots or cluster state is defined here - add one before treating this as anything beyond a lab.
- No admission control / network policy beyond Calico's defaults - production clusters typically layer in NetworkPolicies and often an admission controller (OPA/Gatekeeper, Kyverno, etc.).
This guide reflects Kubernetes' supported release lifecycle and Calico's default networking behavior as of the time of writing. Both move on independent release cadences - re-verify version numbers and encapsulation defaults before using this as a template for a new build.
Contact info:
Gmail: mursleenmamoon@gmail.comLinkedIn: https://www.linkedin.com/in/mursleen-mamoon-097837350/Github: https://github.com/Mursleen666
Top comments (1)
I found the section on troubleshooting to be particularly helpful, as it's often the most frustrating part of deploying a Kubernetes cluster. The step-by-step guide on deploying a Kubernetes cluster with kubeadm on RHEL 9 is well-structured and easy to follow. I've had similar experiences with pod-network and host-network CIDR collisions in the past, and it's great that this edition of the guide addresses that issue. I'm curious, have you considered adding any additional monitoring or logging tools to the cluster for improved visibility and debugging capabilities?