DEV Community: Codewired

PART 1: Deploy modern applications on a production grade, local K8s Cluster, layered with Istio Service Mesh and Observability.

Codewired — Tue, 28 May 2024 21:09:34 +0000

This first part of the three part series will guide you through, how to setup a Hackathon Starter ready, production grade, local development k8s cluster and a service mesh using Rancher's k3s light weight clusters and Istio service mesh. We will then deploy a sample application, add observability and ramp up traffic to see the service mesh in action. The next two parts of this series which will be released next month, will focus on full stack application development with Next.js and FastAPI, effectively showing intermediate and advanced developers how to scaffold production grade dashboard applications and powerful, scalable Fast REST APIs for all purposes, and finally deploying them to the infrastructure that we will setup in this part.

INSTALL DOCKER AND k3d ON LINUX (Debian) & MAC

Install Docker Engine and k3d on MAC

brew update
brew install --cask docker (Recommended if you don't have docker desktop installed)
brew install k3d (Install k3d tool)

On Linux, you will need to uninstall older versions of docker engine on your Linux box if you installed an earlier version. There are two ways to do it.

The first option is to: Uninstall Docker Engine, CLI, containerd, and compose packages
sudo apt-get purge docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin docker-ce-rootless-extras

And delete all images, containers and volumes run:

sudo rm -rf /var/lib/docker
sudo rm -rf /var/lib/containerd

The second option is to run the command below to install all conflicting packages
for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do sudo apt-get remove $pkg; done

Install latest docker engine using the apt repository
Add Docker's official GPG key:

sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

Add the repository to Apt sources:

echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

If you use an Ubuntu derivative distro, such as Linux Mint, you may need to use UBUNTU_CODENAME instead of VERSION_CODENAME.

To install the latest version run:
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

To install a specific version based on your Linux distro version, do this:
List the available versions:
apt-cache madison docker-ce | awk '{ print $3 }'
Select and install the desired version

VERSION_STRING=5:26.1.0-1~ubuntu.24.04~noble
sudo apt-get install docker-ce=$VERSION_STRING docker-ce-cli=$VERSION_STRING containerd.io docker-buildx-plugin docker-compose-plugin

Create a docker group and add the current user

sudo groupadd -f docker
sudo usermod -aG docker $USER

Verify that the Docker Engine installation is successful by running the hello-world image.
sudo docker run hello-world

Install K3d, a light weight wrapper to run Rancher's K3s light weight clusters.
Latest Release curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash
Specific Release curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | TAG=v5.0.0 bash

Show docker version
docker version

Show k3d version
k3d version

INSTALL ISTIO (As of the time of writing this doc, the latest was 1.22.0 and it works with Kubernetes 1.30.1)

Install from Istio Download site

curl -L https://istio.io/downloadIstio | sh - (Install the Latest Release Version)
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.22.0 TARGET_ARCH=x86_64 sh - (Install a specific version OR override processor architecture)

Install from Github Repo

ISTIO_VERSION=1.22.0
ISTIO_URL=https://github.com/istio/istio/releases/download/$ISTIO_VERSION/istio-$ISTIO_VERSION-linux-amd64.tar.gz (For Linux processor ARCH, change as needed)
curl -L $ISTIO_URL | tar xz

Move to the Istio folder and set your PATH env variables for Istio bin directory so you can run istioctl from anywhere

cd istio-1.22.0
export PATH=$PWD/bin:$PATH (You should add this line to your shell config file, .zshrc or .bashrc. Replace the $PWD value with the actual value in your shell config)

Show Istio CTL version
istioctl version

Inspect profiles:

istioctl profile list (Will list available profiles)
istioctl profile dump default (Will dump the default profile config)

Install Istio with the demo profile
istioctl install --set profile=demo -y

Deploy Multi-Node K3s Kubernetes Cluster (v1.30.1) with a local registry, disabling treafik for istio instead (3 nodes, including control plane)
The incantation below creates a 3 node Kubernetes cluster (1 control plane and 2 workers) and uses a load balancer port to expose the internal application via the nginx load balancer, also setting up an internal repository for pushing local images

k3d cluster create svc-mesh-poc --agents 2 --port 7443:443@loadbalancer --port 3070:80@loadbalancer --api-port 6443 --registry-create svc-mesh-registry --image rancher/k3s:v1.30.1-k3s1 --k3s-arg '--disable=traefik@server:*'

Probe local k3s docker images and the newly installed cluster

docker ps --format 'table {{.ID}}\t{{.Image}}\t{{.Names}}\t{{.Ports}}'
kubectl get nodes
kubectl get ns
kubectl get pods -A
kubectl get services -A

Create a Namespace for Demo Application that will be deployed so we can see Istio in action
kubectl create namespace istio-demo-app

To enable the automatic injection of Envoy sidecar proxies on the demo app namespace, run the following: (Otherwise you will need to do this manually when you deploy your applications)
kubectl label namespace istio-demo-app istio-injection=enabled

Deploy Istio's demo application using images from the manifests in the Istio installation samples folder which points to their public registries (Please examine the manifest before applying and make sure you are in the istio version folder)

cd istio-1.22.0
kubectl -n istio-demo-app apply -f samples/bookinfo/platform/kube/bookinfo.yaml

When installation is complete verify pods and services:

kubectl -n istio-demo-app get services
kubectl -n istio-demo-app get pods

Open Outside Traffic to pods so we can browse it locally and on the internal network via the browser:
kubectl -n istio-demo-app apply -f samples/bookinfo/networking/bookinfo-gateway.yaml

Analyze Namespace for Errors
istioctl -n istio-demo-app analyze

Open in browser
http://localhost:3070/productpage

Install Metrics and Tracing Utilities

kubectl apply -f samples/addons
kubectl rollout status deployment/kiali -n istio-system

If there are errors trying to install the addons, try running the command again. There may be some timing issues which will be resolved when the command is run again.

Access the Kiali dashboard
istioctl dashboard kiali

Ramp up hits on the demo application to see Istio MESH in action on Kiali. Run the command
for i in $(seq 1 100); do curl -so /dev/null http://localhost:3070/productpage; done

You many need to clean up all installation in the cluster at some point:

kubectl delete -f samples/addons
kubectl -n istio-demo-app delete -f samples/bookinfo/networking/bookinfo-gateway.yaml
kubectl -n istio-demo-app delete -f samples/bookinfo/platform/kube/bookinfo.yaml
istioctl x uninstall --purge
kubectl delete namespace istio-system
kubectl delete namespace istio-demo-app
kubectl label namespace istio-demo-app istio-injection-

You my need to delete the k3d/k3s cluster:
k3d cluster delete svc-mesh-poc

Yo may want to have more Graphical User Interface to see your cluster in full swing.To do that, deploy Kubernete Dashboard by running the following commands:

Add kubernetes-dashboard repository
helm repo add kubernetes-dashboard https://kubernetes.github.io/dashboard/

Deploy a Helm Release named "kubernetes-dashboard" using the kubernetes-dashboard chart
helm upgrade --install kubernetes-dashboard kubernetes-dashboard/kubernetes-dashboard --create-namespace --namespace kubernetes-dashboard

Verify that Dashboard is deployed and running.
kubectl get pod -n kubernetes-dashboard

Create a ServiceAccount and ClusterRoleBinding to provide admin access to the newly created cluster.

kubectl create serviceaccount -n kubernetes-dashboard admin-user
kubectl create clusterrolebinding -n kubernetes-dashboard admin-user --clusterrole cluster-admin --serviceaccount=kubernetes-dashboard:admin-user

To log in to your Dashboard, you need a Bearer Token. Use the following command to store the token in a variable.
token=$(kubectl -n kubernetes-dashboard create token admin-user)

Display the token using the echo command and copy it to use for logging into your Dashboard.
echo $token

You can access your Dashboard using the kubectl command-line tool and port forwarding by running the following commands and pasting the Bearer token on the text box:
kubectl -n kubernetes-dashboard port-forward svc/kubernetes-dashboard-kong-proxy 8443:443

Browse to https://localhost:8443

Clean Up Admin Service Account and Cluster Role Binding for Kubernetes Dashboard user.

kubectl -n kubernetes-dashboard delete serviceaccount admin-user
kubectl -n kubernetes-dashboard delete clusterrolebinding admin-user

Now that you have A lightweight multi-node cluster running locally with Istio configured, you can now try out all of these Istio features:

Request Routing
Fault Injection
Traffic Shifting
Querying metrics
Visualizing metrics
Accessing external services
Visualizing your mesh.

Stay tuned for more!

Build a multi node Kubernetes Cluster on Google Cloud VMs using Kubeadm, from the ground up!

Codewired — Sat, 25 May 2024 19:00:42 +0000

Bringing in and updating an initial post written two years ago on my other profile here. These are the defined and authentic steps you can use to run a Three, Five, Seven...etc node cluster on GCP using Linux VMs.

Choose your Linux Flavor, recommended is Ubuntu 22.04 (Jammy) (This will work on local dev boxes and on Cloud Compute VMs with Jammy)

On a Google Cloud Web Console, pick your desired project that has billing enabled and setup your cli tool (Google Cloud CLI) and create a VPC, Subnet and Firewall to allow traffik.

(Replace resource names in square brackets without the brackets):
Create a Virtual Private Cloud Network
gcloud compute networks create [vpc name] --subnet-mode custom
Create a Subnet with a specific range (10.0.96.0/24)
gcloud compute networks subnets create [subnet name] --network [vpc name] --range 10.0.96.0/24
Create Firewall Rule that allows internal communication accros all protocols (10.0.96.0/24, 10.0.92.0/22)
gcloud compute firewall-rules create [internal network name] --allow tcp,udp,icmp --network [vpc name] --source-ranges 110.0.96.0/24, 10.0.92.0/22
Create a firewall rule that allows external SSH, ICMP, and HTTPS:
gcloud compute firewall-rules create [external network name] --allow tcp,icmp --network [vpc name] --source-ranges 0.0.0.0/0
List the firewall rules in the VPC network:
gcloud compute firewall-rules list --filter="network:[vpc name]"

Provision Nodes:
Create 3, 5 or 7 compute instances which will host the Kubernetes Proxy, control plane and worker nodes respectively (A proxy is recommended if you are creating 5 nodes or more):

Proxy Plane Node (Optional):
gcloud compute instances create proxynode --async --boot-disk-size 50GB --can-ip-forward --image-family ubuntu-2204-lts --image-project ubuntu-os-cloud --machine-type n2-standard-2 --private-network-ip 10.0.96.10 --scopes compute-rw,storage-ro,service-management,service-control,logging-write,monitoring --subnet [subnet name] --tags kubevms-node,proxy

Master Control Plane Node:
gcloud compute instances create masternode --async --boot-disk-size 200GB --can-ip-forward --image-family ubuntu-2204-lts --image-project ubuntu-os-cloud --machine-type n2-standard-2 --private-network-ip 10.0.96.11 --scopes compute-rw,storage-ro,service-management,service-control,logging-write,monitoring --subnet [subnet name] --tags kubeadm-node,controller

Worker Nodes: (10.0.96.21+ for the other worker nodes)
gcloud compute instances create workernode1 --async --boot-disk-size 100GB --can-ip-forward --image-family ubuntu-2204-lts --image-project ubuntu-os-cloud --machine-type n2-standard-2 --private-network-ip 10.0.96.20 --scopes compute-rw,storage-ro,service-management,service-control,logging-write,monitoring --subnet [subnet name] --tags kubeadm-node,worker

Print the Internal IP address and Pod CIDR range for each worker node
gcloud compute instances describe workernode1 --format 'value[separator=" "](networkInterfaces[0].networkIP,metadata.items[0].value)'

List the compute instances in your default compute zone:
gcloud compute instances list --filter="tags.items=kubeadm-node"

Test SSH Into Google Cloud VM Instance (You will need to SSH into all the VMs/Nodes to install software)
gcloud compute ssh [compute instance name]

RUN THESE INSTALLATIONS ON ALL NODES
a. sudo -i
b. apt-get update && apt-get upgrade -y
c. apt install curl apt-transport-https vim git wget gnupg2 software-properties-common apt-transport-https ca-certificates uidmap lsb-release -y
d. swapoff -a

INSTALL AND CONFIGURE CONTAINER RUNTIME PREREQUISITES ON ALL NODES
Verify that the br_netfilter module is loaded by running
lsmod | grep br_netfilter
In order for a Linux node's iptables to correctly view bridged traffic, verify that net.bridge.bridge-nf-call-iptables is set to 1 in your sysctl by running the following commands:

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf overlay br_netfilter EOF

sudo modprobe overlay
sudo modprobe br_netfilter

sysctl params required by setup, params persist across reboots
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.ipv4.ip_forward = 1 EOF

Apply sysctl params without reboot
sudo sysctl --system

INSTALL CONTAINER RUNTIME ON ALL NODES
a. mkdir -p /etc/apt/keyrings
b. curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
c. echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
d. apt-get update
e. apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin

CONFIGURE CGROUP DRIVER FOR CONTAINERD ON ALL NODES (We will use the more advanced SystemD that comes with Ubuntu 2204)
a. stat -fc %T /sys/fs/cgroup/ (Check to see if you are using the supported cgroupV2)
b. sudo containerd config default | sudo tee /etc/containerd/config.toml (Make sure that the config.toml is present with defaults)
c. Set the SystemDGroup = true to use the CGroup driver in the config.toml [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
...
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true
d. sudo systemctl restart containerd (Restart ContainerD)

INSTALL KUBEADM, KUBELET & KUBECTL ON ALL NODES
Download the Google Cloud public signing key:
a. sudo curl -fsSLo /etc/apt/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

Add the Kubernetes apt repository:
b. echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
c. sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

CONFIGURE CGROUP DRIVER FOR MASTER NODE (Add section to kubeadm-config.yaml if you are using a supprted OS for SystemD and change the kubernetesVersion to the actual one installed by kubeadm)

    apiVersion: kubeadm.k8s.io/v1beta3
    kind: ClusterConfiguration
    kubernetesVersion: 1.30.0
    controlPlaneEndpoint: "masternode:6443"
    networking:
      podSubnet: 10.200.0.0/16

    ---
    apiVersion: kubelet.config.k8s.io/v1beta1
    kind: KubeletConfiguration
    cgroupDriver: systemd

CONFIRGURE HOSTNAME FOR MASTER NODE
Open file : nano /ect/hosts
Add Master Node's Static IP and preferred Hostname (10.0.96.11 masternode)

INITIALIZE KUBEADM ON MASTER NODE (Remember to save the token hash)
kubeadm init --config=kubeadm-config.yaml --upload-certs | tee kubeadm-init.out

Logout from ROOT if you are stilL ROOT

To make kubectl work for your non-root user, run these commands, which are also part of the kubeadm init output:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

INSTALL A POD NETWORKING INTERFACE ON MASTER NODE
Download and Install the Tigera Calico operator and custom resource definitions.
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.24.5/manifests/tigera-operator.yaml

Download and Install Calico by creating the necessary custom resource.
Before installing remember to change the CALICO_IPV4POOL_CIDR to the POD_CIDR (10.200.0.0/16)
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.24.5/manifests/custom-resources.yaml

JOIN WORKER NODES TO THE CONTROL PLANE (MASTER NODE)
Run the command below inside each worker node with the token you got from the cli when you initialized kubeadm on Master Node:
kubeadm join masternode:6443 --token n0smf1.ixdasx8uy109cuf8 --discovery-token-ca-cert-hash sha256:f6bce2764268ece50e6f9ecb7b933258eac95b525217b8debb647ef41d49a898