Deploying a Kubernetes Cluster with kubeadm on RHEL 9 & CentOS 10

Mursleen Mamoon — Tue, 14 Jul 2026 02:44:28 +0000

Deploying-a-Kubernetes-Cluster-with-kubeadm-on-RHEL-9-CentOS-10

The guide is based on a hands-on implementation and is intended to serve as both a learning resource and a reusable deployment reference for students, DevOps engineers, and anyone building a Kubernetes lab environment.

Deploying a Kubernetes Cluster

with kubeadm on RHEL 9

A Corrected & Expanded Reference Build

1 Control Plane · 2 Worker Nodes · VMware

AUTHOR	Muhammad Mursleen Mamaoon
PLATFORM	RHEL 9
ORCHESTRATION	kubeadm
CONTAINER RUNTIME	containerd
CLUSTER NETWORKING	Calico CNI
DOCUMENT TYPE	Deployment Runbook & Reference Architecture
EDITION	Corrected Edition - networking, firewall, and lifecycle fixes applied

Contents

1. Purpose and Scope 3

2. Lab Topology 3

3. Prerequisites 3

3.1 VMware-Specific Gotcha: Clone Hygiene 4

4. Step 1 - Hostnames and Name Resolution [ALL] 4

5. Step 2 - Time Synchronization [ALL] 5

6. Step 3 - Disable Swap [ALL] 5

7. Step 4 - SELinux [ALL] 5

8. Step 5 - Kernel Modules [ALL] 5

9. Step 6 - Kernel Network Parameters [ALL] 6

10. Step 7 - Firewall Configuration [ALL] 6

11. Step 8 - Install and Configure containerd [ALL] 7

12. Step 9 - Install Kubernetes Components [ALL] 8

13. Step 10 - Initialize the Control Plane [MASTER] 8

14. Step 11 - Join Worker Nodes [WORKER] 9

15. Step 12 - Install Calico CNI [MASTER] 9

16. Step 13 - Validate the Cluster [MASTER] 9

17. Troubleshooting 10

18. What Changes for Production 10

Right-click the table above and choose "Update Field" after opening in Word if page numbers do not populate automatically.

1. Purpose and Scope

This document is a complete, validated procedure for bootstrapping a Kubernetes cluster using kubeadm, containerd, and Calico on RHEL 9, deployed across three VMware virtual machines. It is written as both an execution runbook and a design reference - each step includes not just the command, but the reasoning behind it, so it can be adapted safely rather than copy-pasted blind.

This edition corrects several issues identified in the original draft, most notably a pod-network / host-network CIDR collision and a firewall rule that blocked Calico's actual encapsulation mode, and adds operational detail commonly missing from kubeadm walkthroughs: VMware clone hygiene, time synchronization, version pinning, and a fuller validation pass.

SCOPE NOTE

As written, this procedure produces a functional lab / reference cluster - suitable for learning, testing, and demonstrating the mechanics of kubeadm. It is not, as-is, a hardened production topology: a single control-plane node is a single point of failure, SELinux is relaxed rather than properly policed, and there is no external etcd or highly-available API server. Section 18 outlines what changes for production use.

2. Lab Topology

Role	Hostname	IP Address	vCPU	RAM
Control Plane	k8s-master	192.168.1.10	2+	2 GB+
Worker Node 1	k8s-worker1	192.168.1.11	2+	2 GB+
Worker Node 2	k8s-worker2	192.168.1.12	2+	2 GB+

Stack: RHEL 9 · VMware Workstation/ESXi · kubeadm · containerd · Calico CNI

Notation Used Throughout

[ALL] - run on every node
[MASTER] - control plane only
[WORKER] - worker nodes only

3. Prerequisites

Three RHEL 9 VMs with routable connectivity to each other and to the internet, for package and container image pulls.
SSH access with sudo privileges on all nodes.
A hard minimum of 2 vCPUs on the control-plane node. This is not a soft recommendation - kubeadm init runs a preflight check and refuses to proceed on a single-vCPU control plane.
A pod network CIDR that does not overlap the physical/VM network. The original draft used 192.168.0.0/16 for pods on a 192.168.1.0/24 host network - those ranges overlap. This guide uses 10.244.0.0/16 instead (see Section 10).

3.1 VMware-Specific Gotcha: Clone Hygiene

If any of these three VMs were created by cloning a template - extremely common in VMware labs - they can inherit an identical /etc/machine-id and product UUID. Kubernetes uses these values to distinguish nodes, and duplicates cause node registration conflicts, DHCP lease collisions, and kubelet identity confusion that are painful to debug after the fact. Check uniqueness before proceeding:

# [ALL] Every value below must differ across all three nodes

cat /etc/machine-id

cat /sys/class/dmi/id/product_uuid

ip link show | grep ether

If any values match across nodes, regenerate the machine ID on the affected VM:

# [ALL, only if duplicated]

sudo rm -f /etc/machine-id /var/lib/dbus/machine-id

sudo systemd-machine-id-setup

sudo ln -s /etc/machine-id /var/lib/dbus/machine-id

Duplicate product_uuid values require a fix at the hypervisor level - regenerate the BIOS UUID in the VM's .vmx settings, or re-clone from the template with "new identity" options enabled.

4. Step 1 - Hostnames and Name Resolution [ALL]

# Run the appropriate line on each respective node

sudo hostnamectl set-hostname k8s-master # on the control plane

sudo hostnamectl set-hostname k8s-worker1 # on worker 1

sudo hostnamectl set-hostname k8s-worker2 # on worker 2

Add all three nodes to /etc/hosts on every node:

| sudo tee -a /etc/hosts <
192.168.1.10 k8s-master

192.168.1.11 k8s-worker1

192.168.1.12 k8s-worker2

EOF | |
| --- | |
| IDEMPOTENCY NOTE

tee -a appends unconditionally. Re-running this step produces duplicate lines - harmless for resolution, but if scripting this for repeatable builds, guard it with a grep -qxF check first. | |

5. Step 2 - Time Synchronization [ALL]

ADDED IN THIS EDITION

The original draft did not include this step, and it should have. Kubernetes leans heavily on TLS certificate validity windows and etcd's Raft consensus, both sensitive to clock drift between nodes.

RHEL 9 ships chrony by default - confirm it is active:

sudo systemctl enable --now chronyd

chronyc tracking

Confirm System time shows a sub-second offset before continuing.

6. Step 3 - Disable Swap [ALL]

The kubelet refuses to start with swap enabled, unless explicitly configured via NodeSwap / MemorySwap, which is out of scope for this build.

sudo swapoff -a

sudo sed -i '/\sswap\s/s/^/#/' /etc/fstab

7. Step 4 - SELinux [ALL]

| sudo setenforce 0

sudo sed -i 's/^SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config | |
| --- | |
| PRODUCTION CAUTION

Permissive mode disables SELinux enforcement cluster-wide, which is acceptable for a lab but removes a real security boundary in production. If extending this design, install the container-selinux package (ships proper policies for containerd/CRI-O) and keep SELinux in enforcing mode - most modern kubeadm/containerd combinations on RHEL run fine enforced, provided the runtime's SELinux policy is installed correctly. | |

8. Step 5 - Kernel Modules [ALL]

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf

overlay

br_netfilter

EOF

sudo modprobe overlay

sudo modprobe br_netfilter

9. Step 6 - Kernel Network Parameters [ALL]

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf

net.bridge.bridge-nf-call-iptables = 1

net.bridge.bridge-nf-call-ip6tables = 1

net.ipv4.ip_forward = 1

EOF

sudo sysctl --system

10. Step 7 - Firewall Configuration [ALL]

CORRECTED IN THIS EDITION

The original port list included two deprecated ports and opened the wrong protocol for Calico's default overlay mode.

Dropped 10251/tcp and 10252/tcp - pre-1.19 insecure HTTP endpoints for kube-scheduler and kube-controller-manager, replaced by the authenticated endpoints on 10259 and 10257. Modern kubeadm does not expose the old ones at all.
Replaced 4789/udp (VXLAN) with the IP-in-IP protocol - Calico's default manifest (calico.yaml) uses IP-in-IP encapsulation, not VXLAN, unless explicitly reconfigured. Opening only 4789/udp, as the original draft did, leaves the actual overlay traffic blocked by the firewall.

Control Plane

sudo firewall-cmd --permanent --add-port={6443,2379,2380,10250,10257,10259,179}/tcp

sudo firewall-cmd --permanent --add-protocol=ipip

sudo firewall-cmd --reload

Worker Nodes

sudo firewall-cmd --permanent --add-port={10250,179,30000-32767}/tcp

sudo firewall-cmd --permanent --add-protocol=ipip

sudo firewall-cmd --reload

Port Reference

Port	Protocol	Purpose	Nodes
6443	TCP	kube-apiserver	Control plane
2379-2380	TCP	etcd client/peer	Control plane
10250	TCP	kubelet API	All
10257	TCP	kube-controller-manager (secure)	Control plane
10259	TCP	kube-scheduler (secure)	Control plane
179	TCP	Calico BGP	All
-	IP-in-IP (proto 4)	Calico overlay (default mode)	All
30000-32767	TCP	NodePort service range	All

IF YOU SWITCH TO VXLAN

If Calico is later switched to VXLAN mode instead of IP-in-IP (common on networks that block IP-in-IP, e.g. some cloud VPCs), swap the --add-protocol=ipip rule for --add-port=4789/udp on every node. Don't open both unless intentionally running mixed mode.

11. Step 8 - Install and Configure containerd [ALL]

sudo dnf install -y dnf-plugins-core

sudo dnf config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo

sudo dnf install -y containerd.io

sudo mkdir -p /etc/containerd

containerd config default | sudo tee /etc/containerd/config.toml

# Kubernetes requires the systemd cgroup driver (RHEL 9 uses cgroup v2 by default)

sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml

sudo systemctl enable --now containerd

sudo systemctl restart containerd

12. Step 9 - Install Kubernetes Components [ALL]

| VERSION UPDATED IN THIS EDITION

The original draft pinned v1.31, which has already reached end-of-life. Kubernetes follows an N-2 support policy across roughly four-month release cycles; the currently supported branches are 1.34, 1.35, and 1.36. Check kubernetes.io/releases for the current set before building, and substitute below. | |
| --- | |
| cat <
[kubernetes]

name=Kubernetes

baseurl=https://pkgs.k8s.io/core:/stable:/v1.34/rpm/

enabled=1

gpgcheck=1

gpgkey=https://pkgs.k8s.io/core:/stable:/v1.34/rpm/repodata/repomd.xml.key

exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni

EOF | |

Pin exact versions instead of floating on "latest." Without pinning, three separate dnf install runs across three nodes can silently pull three different patch versions if a new patch lands mid-build - tedious to debug later. Check what's available first:

dnf list --showduplicates kubelet --disableexcludes=kubernetes

Then install a matched version explicitly:

| sudo dnf install -y kubelet-1.34.* kubeadm-1.34.* kubectl-1.34.* --disableexcludes=kubernetes

sudo systemctl enable --now kubelet | |
| --- | |
| EXPECTED BEHAVIOR

It is normal for kubelet to sit in a crash-restart loop at this point (systemctl status kubelet will look unhappy) - it is waiting for kubeadm init / kubeadm join to supply a configuration. Not a bug. | |

13. Step 10 - Initialize the Control Plane [MASTER]

| CIDR CORRECTED IN THIS EDITION

The original draft used 192.168.0.0/16 for the pod network, which fully contains the 192.168.1.0/24 host network the VMs sit on. Overlapping ranges break routing between pods and real hosts - the kernel cannot disambiguate them. This guide uses a distinct, non-overlapping range. | |
| --- | |
| sudo kubeadm init \

--apiserver-advertise-address=192.168.1.10 \

--pod-network-cidr=10.244.0.0/16 | |

Save the kubeadm join … command printed at the end of initialization - it is required on both worker nodes, and expires after 24 hours by default.

Configure kubectl for your user:

mkdir -p \$HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf \$HOME/.kube/config

sudo chown \$(id -u):\$(id -g) \$HOME/.kube/config

14. Step 11 - Join Worker Nodes [WORKER]

sudo kubeadm join 192.168.1.10:6443 \

--token <token> \

--discovery-token-ca-cert-hash sha256:<hash>

If the token has expired (default TTL: 24 hours), generate a fresh one from the control plane:

kubeadm token create --print-join-command

15. Step 12 - Install Calico CNI [MASTER]

| CIDR ALIGNMENT REQUIRED

Calico's default manifest ships with its own IP pool CIDR (192.168.0.0/16), which must match the --pod-network-cidr given to kubeadm. Since that value changed to 10.244.0.0/16, the manifest needs the same edit before applying. | |
| --- | |
| curl -O https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml

sed -i 's/192.168.0.0\/16/10.244.0.0\/16/' calico.yaml

kubectl apply -f calico.yaml | |

Double-check the Calico version tag (v3.28.0) against what's current before building - Calico ships new minor releases on its own cadence, independent of Kubernetes.

16. Step 13 - Validate the Cluster [MASTER]

Nodes should transition to Ready once Calico's pods come up - this can take a minute or two:

kubectl get nodes -o wide

kubectl get pods -n kube-system -o wide

All calico-node, calico-kube-controllers, and coredns pods should be Running. If coredns is stuck Pending, it is almost always waiting on CNI - check the calico-node pods first.

Sanity-check core cluster health:

kubectl get componentstatuses # deprecated but still useful for a quick glance

kubectl cluster-info

Deploy and expose a smoke-test workload:

kubectl create deployment nginx-demo --image=nginx

kubectl expose deployment nginx-demo --type=NodePort --port=80

kubectl get svc nginx-demo

curl http://<any-worker-ip>:<node-port>

A default NGINX welcome page confirms the API server, scheduler, kubelet, CNI, and kube-proxy are all functioning end-to-end.

17. Troubleshooting

Symptom	Cause / Fix
kubelet won't start / crashlooping before join	Expected pre-kubeadm init/join. If it persists after joining, confirm SystemdCgroup = true is actually set in /etc/containerd/config.toml and that containerd was restarted after the edit.
Worker stuck NotReady	Expected until Calico is applied. If it persists afterward, check kubectl get pods -n kube-system for calico-node crash loops - usually a firewall or IP-in-IP protocol issue (Section 10).
kubeadm join fails / times out	Verify firewall ports (Section 10), hostname resolution (Section 4), and that the join token/hash haven't expired.
Pods reach each other on the same node but not across nodes	Almost always the overlay protocol being blocked - confirm firewall-cmd --add-protocol=ipip was applied on every node, not just the control plane.
Two nodes seem to "merge" identity, or one registers then vanishes	Cloned-VM machine-id / UUID collision - see Section 3.1.
kubeadm init preflight fails on CPU count	Control-plane node has fewer than 2 vCPUs - a hard requirement, not advisory.
Cluster works but certs/etcd behave oddly under load	Check chronyc tracking on all nodes - clock drift across etcd peers causes subtle, hard-to-diagnose failures.
Lost the join command	kubeadm token create --print-join-command from the control plane, any time.

18. What Changes for Production

This build is correct and internally consistent for a lab or reference deployment, but a few things are deliberately out of scope and worth naming explicitly if the design is extended:

Single control-plane node = single point of failure. Production topologies run 3 (or 5) control-plane nodes behind a load balancer, with --control-plane-endpoint set to the LB's DNS name from the first kubeadm init, and --upload-certs for subsequent control-plane joins.
etcd here is stacked on the control-plane node. Production clusters commonly run etcd on dedicated nodes for isolation and easier backup/restore.
SELinux permissive should become enforcing with proper container SELinux policy.
No backup strategy for etcd snapshots or cluster state is defined here - add one before treating this as anything beyond a lab.
No admission control / network policy beyond Calico's defaults - production clusters typically layer in NetworkPolicies and often an admission controller (OPA/Gatekeeper, Kyverno, etc.).

This guide reflects Kubernetes' supported release lifecycle and Calico's default networking behavior as of the time of writing. Both move on independent release cadences - re-verify version numbers and encapsulation defaults before using this as a template for a new build.

Contact info:

Gmail: mursleenmamoon@gmail.comLinkedIn: https://www.linkedin.com/in/mursleen-mamoon-097837350/Github: https://github.com/Mursleen666

DEV Community: Mursleen Mamoon