Mastering On-Premise Kubernetes: A Comprehensive Guide to Setting Up Your Own Cluster

Sai Simha Reddy — Mon, 11 Mar 2024 12:55:38 +0000

Welcome to our guide on setting up a Kubernetes cluster on-premises! In this blog, we'll walk you through the process of implementing a Kubernetes cluster using one master node and three worker nodes, all running on Rocky Linux virtual servers. By following along, you'll gain the skills and knowledge needed to deploy and manage your own Kubernetes infrastructure in an on-premises environment. Let's dive in!

Before getting started:

What is Kubernetes?

Before we dive into the nitty-gritty of setting up our Kubernetes cluster on-premises, let's take a moment to understand what Kubernetes is and why it's such a powerful tool for container orchestration.

Kubernetes, often abbreviated as K8s, is an open-source platform designed to automate the deployment, scaling, and management of containerized applications. Originally developed by Google, Kubernetes has quickly become the de facto standard for container orchestration, offering features such as automated scaling, load balancing, and self-healing capabilities.

With Kubernetes, you can abstract away the underlying infrastructure and focus on deploying your applications in a consistent and efficient manner, regardless of whether they're running on-premises, in the cloud, or in a hybrid environment.
Now that we have a basic understanding of Kubernetes, let's roll up our sleeves and get started with setting up our on-premises cluster using Rocky Linux virtual servers.

Let's get started:

Pre-requisites:

4 Rocky Linux Servers.(I am using Rocky Linux 9 flavour in this blog).

Let's Start:

1. Set up the hostnames of all 4 servers in "/etc/hosts" file

10.168.100.21 master.kubernetes.com
10.168.100.22 worker1.kubernetes.com
10.168.100.23 worker2.kubernetes.com
10.168.100.24 worker3.kubernetes.com

(Note: Replace hostname with your original hostnames, i have used kubernetes.com just for demo purpose)

2. Disable firewall service in all four nodes

if you are using firewalld service:

  systemctl stop firewalld
  systemctl disable firewalld

if you are using iptables service:

  systemctl stop iptables
  systemctl disable iptables

3. Add container modules to the list of kernel modules

 sudo tee /etc/modules-load.d/containerd.conf <<EOF
 overlay
 br_netfilter
 EOF

4. Manually load kernel modules into the Linux kernel at
runtime, without needing to reboot the system

 sudo modprobe overlay
 sudo modprobe br_netfilter

5. Essential Network Configuration for Kubernetes: Enabling Advanced Networking Features

 sudo tee /etc/sysctl.d/kubernetes.conf <<EOF
 net.bridge.bridge-nf-call-ip6tables = 1
 net.bridge.bridge-nf-call-iptables = 1
 net.ipv4.ip_forward = 1
 EOF

 sudo sysctl --system

6. Deactivate all swap partitions or swap files that are currently in use

 swapoff -a
 sudo sed -i '/swap/ s/^/#/' /etc/fstab

7. Disable SELINUX

 sudo sed -i 's/^SELINUX=.*/SELINUX=disabled/' 
 /etc/selinux/config.bkp

*8. Reboot the all nodes once to apply the above changes *

 reboot

9. Adding Docker Repository to CentOS Yum Configuration in all the nodes

 yum-config-manager --add-repo
 https://download.docker.com/linux/centos/docker-ce.repo

10. Download and install the container run time in all the nodes

  yum install -y docker-ce docker-ce-cli containerd.io

11. Configuring Containerd to Use systemd Cgroups

  containerd config default | sudo tee 
  /etc/containerd/config.toml >/dev/null 2>&1

 sudo sed -i 's/SystemdCgroup \= false/SystemdCgroup \= 
 true/g' /etc/containerd/config.toml

12. Start and enable the container run time service

  systemctl start containerd
  systemctl enable containerd

  systemctl start docker
  systemctl enable docker

13. Adding Kubernetes Repository to CentOS Yum Configuration in all the nodes

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.29/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.29/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

14. Download and install kubelet in all the nodes

 yum install -y kubelet kubeadm kubectl -- 
 disableexcludes=kubernetes

15. Start and enable the kubelet service

 systemctl start kubelet
 systemctl eanble kubelet

16. Initializing a Kubernetes Cluster with Custom Configurations(Run the below command in master node)

kubeadm init --apiserver-advertise-address=10.168.100.21 --
pod-network-cidr=172.100.0.0/16

Note: apiserver-advertise-address should be the ip address of master node in our K8S cluster. You can assign your custom cidr for pod-network-cidr.

The cluster will be initialized, and all control plane components, such as the kube-api server, etcd, kube-scheduler, and kube-controller-manager, will be set up.

As soon as the cluster is initialized, it displays the command for worker nodes to join the cluster

kubeadm join 10.168.100.21:6443 --token koyi1v.hoq6ssx0dxy6k5ji --discovery-token-ca-cert-hash sha256:fc5066a9e04652e4e6f67fa9e49749b76c6f9873df382e9b9ac47d747877bd7d(in my case).

17. Setting Up Kubernetes Configuration for User Access(in
master node)

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

*18. Join the worker nodes to the cluster. *

  kubeadm join 10.168.100.21:6443 --token koyi1v.hoq6ssx0dxy6k5ji         --discovery-token-ca-cert-hash sha256:fc5066a9e04652e4e6f67fa9e49749b76c6f9873df382e9b9ac47d747877bd7d(run it on all worker nodes)

check the status in master node using "kubectl get nodes"
It will show the all the worker nodes info but the status is in NotReady state since we have not set up the cluster network yet.

19. Installing a Pod network add-on using calico network(in master node)

  curl https://raw.githubusercontent.com/projectcalico/calico/v3.27.2/manifests/calico.yaml -O

If you are using pod CIDR other than 192.168.0.0/16, then uncomment the CALICO_IPV4POOL_CIDR variable in the manifest(calico.yaml) and set it to the same value as your chosen pod CIDR.(in my case i set it to 172.100.0.0/16)

Apply the manifest using the following command.

   kubectl apply -f calico.yaml

Now the pod network and network policy are deployed in our cluster. The nodes use this pod network to communicate with each other.

20. check the pods and nodes are running(in master node)

kubectl get pods -n kube-system
kubectl get nodes

calico-kube-controllers-68cdf756d9-fjfjp   1/1     Running   2 (11d ago)   12d
calico-node-2frrp                          1/1     Running   2 (11d ago)   12d
calico-node-4hnfk                          1/1     Running   1 (11d ago)   12d
calico-node-kf7kf                          1/1     Running   1 (11d ago)   12d
calico-node-svvlh                          1/1     Running   1 (11d ago)   12d
coredns-5dd5756b68-8cqq8                   1/1     Running   1 (11d ago)   12d
coredns-5dd5756b68-x8hrw                   1/1     Running   1 (11d ago)   12d
etcd-aekmlpcik8sm212                       1/1     Running   2 (11d ago)   12d
kube-apiserver-aekmlpcik8sm212             1/1     Running   2 (11d ago)   12d
kube-controller-manager-aekmlpcik8sm212    1/1     Running   4 (11d ago)   12d
kube-proxy-2ffxk                           1/1     Running   1 (11d ago)   12d
kube-proxy-kbfj2                           1/1     Running   1 (11d ago)   12d
kube-proxy-m42v5                           1/1     Running   2 (11d ago)   12d
kube-proxy-v5dsq                           1/1     Running   1 (11d ago)   12d
kube-scheduler-aekmlpcik8sm212             1/1     Running   5 (11d ago)   12d


aekmlpcik8sm212   Ready    control-plane   12d   v1.28.7
aekmlpcik8sw213   Ready    <none>          12d   v1.28.7
aekmlpcik8sw214   Ready    <none>          12d   v1.28.7
aekmlpcik8sw215   Ready    <none>          12d   v1.28.7

The above output depicts that everything is good and working fine in our cluster.

Bravo, Our Kubernetes cluster has been set up successfully

DEV Community: Sai Simha Reddy

Mastering On-Premise Kubernetes: A Comprehensive Guide to Setting Up Your Own Cluster