DEV Community: Ian Kiprotich

Exploring Longhorn: A Game-Changer for Kubernetes Storage

Ian Kiprotich — Wed, 24 May 2023 06:43:35 +0000

Exploring Longhorn: A Game-Changer for Kubernetes Storage

Welcome back to my blog, a dedicated space where we dive into the vast ocean of Kubernetes and its thriving ecosystem. If you’re a developer, a DevOps professional, or simply a tech enthusiast, you know that Kubernetes has fundamentally transformed the way we build, deploy, and manage applications. But the story doesn’t end there.

Today, we shift our focus towards a crucial yet often challenging aspect of Kubernetes — storage. Ensuring persistent and reliable storage in a Kubernetes environment can sometimes feel like a complex puzzle. However, there’s a solution that can help us put these pieces together seamlessly — Longhorn and runs on any enviroment.

Longhorn, an innovative open-source project by Rancher Labs, offers a reliable, lightweight, and user-friendly distributed block storage system for Kubernetes. It’s changing the game by making it simpler to run stateful applications in a Kubernetes environment without the storage headaches.

In this blog post, we will dive into the world of Longhorn, exploring its features, architecture, and why it might be the perfect fit for your Kubernetes storage needs. Whether you’re already navigating the Kubernetes landscape or just beginning your journey, this comprehensive guide to Longhorn will provide valuable insights into managing Kubernetes storage effectively.

Longhorn: Origins, Architecture, and Purpose

Born out of Rancher Labs, Longhorn began as an open-source project in 2014. The goal was clear: to simplify, streamline, and democratize the management of persistent storage volumes in Kubernetes environments. Originally an internal project at Rancher Labs, it was donated to the Cloud Native Computing Foundation (CNCF) in 2020, cementing its position in the Kubernetes ecosystem.

What is Longhorn?

At its core, Longhorn is a lightweight, reliable, and easy-to-use distributed block storage system for Kubernetes. In essence, it transforms commodity hardware and cloud volumes into a reliable and distributed block storage solution, providing software-defined storage (SDS) for Kubernetes.

How Does Longhorn Work?

Longhorn operates by using containers to create a storage volume and replicates it synchronously across multiple nodes in a Kubernetes cluster. It ensures high availability of the data stored in these volumes by intelligently distributing data replicas across different nodes and disks. If a node or disk fails, Longhorn automatically switches workload to another replica to maintain availability.

The Purpose of Longhorn

The principal aim of Longhorn is to make it as easy as possible to manage persistent storage for Kubernetes workloads. Before tools like Longhorn, managing stateful workloads in Kubernetes was quite challenging. Developers and system administrators often had to rely on external storage systems, manually provision storage, or be restricted to specific cloud providers.

With Longhorn, this changes. It allows users to create, manage, and automatically scale persistent volumes directly from the Kubernetes interface or through Kubernetes APIs. It also provides intuitive interfaces for volume backups, snapshots, and storage management — all out of the box.

Longhorn is designed to be platform-agnostic. This means it can work across bare-metal, on-premises, and cloud-based Kubernetes installations, providing a consistent experience no matter where your cluster is hosted.

Features and Benefits of Using Longhorn in a Kubernetes Environment

Longhorn provides a rich set of features designed to address various needs in a Kubernetes environment. These features make it an excellent choice for handling persistent storage in Kubernetes, simplifying the management of stateful applications.

1. Easy Installation and Operation

Longhorn can be installed on any Kubernetes cluster with a single command, making it extremely straightforward to get started. It provides an intuitive graphical user interface that simplifies management tasks, such as creating and attaching volumes, taking and managing snapshots, and setting up backups.

2. Lightweight and Reliable

Longhorn has a small footprint and doesn’t require any additional services to run, making it a lightweight solution. It ensures high availability by synchronously replicating volumes across different nodes in the cluster. If a node or disk fails, Longhorn automatically re-replicates data to other nodes, maintaining data safety.

3. Cloud-Native and Kubernetes-Native

As a cloud-native solution, Longhorn runs on any Kubernetes cluster, regardless of whether it’s on-premises, in the cloud, or in a hybrid environment. Longhorn volumes are natively integrated into Kubernetes, which means you can manage them just like any other Kubernetes resource.

4. Disaster Recovery Volumes

Longhorn supports cross-cluster disaster recovery volumes, enabling standby volumes to be quickly activated in case of a disaster. This feature is vital for business continuity and meets the high availability requirements of enterprise applications.

5. Incremental Snapshots and Backups

Longhorn allows you to take incremental snapshots of your volumes, which can then be backed up to a secondary storage. This approach optimizes storage usage and speeds up backup and recovery operations.

6. ReadWriteMany (RWX) Volumes Support

Starting from Longhorn version 1.1.0, Longhorn supports ReadWriteMany (RWX) volumes, allowing a volume to be read and written by many nodes at the same time, which is a key requirement for certain types of applications.

7. Active Monitoring and Alerts

Longhorn provides active monitoring and alerting through integration with Prometheus, a popular open-source monitoring solution.

8. Built-In Data Locality and Affinity Features

Longhorn supports data locality to try and keep the data and the workload in the same node to improve performance. It also supports volume and node affinity/anti-affinity policies for advanced volume scheduling.

Longhorn Architecture and Operation

The architecture of Longhorn is designed to be lightweight and fully integrated with Kubernetes. It consists of a set of components that work together to deliver a reliable, distributed block storage solution for your Kubernetes workloads. Here’s an overview of the key components:

Longhorn Manager:

The Longhorn Manager runs on each node in the Kubernetes cluster. It’s responsible for orchestrating the other components, handling scheduling, detecting node failures, and maintaining the state of the cluster.

Longhorn Engine:

The Longhorn Engine is the data plane component responsible for reading and writing data to the storage backend. Each Longhorn volume has a corresponding Longhorn Engine instance. The engine captures snapshots, creates backups, and replicates data across different nodes for high availability.

Longhorn UI:

The Longhorn UI is a graphical user interface that allows users to easily manage volumes, take snapshots, set up backups, and monitor the state of the storage system.

Longhorn CSI (Container Storage Interface) Driver:

The CSI driver allows Kubernetes to use Longhorn as its storage provider. It translates Kubernetes volume operations into corresponding Longhorn operations, such as creating a volume, attaching/detaching a volume to/from a node, and mounting/unmounting a volume to/from a pod.

Longhorn Instance Manager:

There are two types of instance managers, one for managing Longhorn Engine instances and another for managing Longhorn Replica instances. Each node in the cluster runs one instance manager pod for each type.

Longhorn Replicas:

These are the copies of the data that are synchronously created by the Longhorn engine on different nodes for redundancy and high availability.

How Longhorn Operates

In Longhorn, when you create a persistent volume claim (PVC) in Kubernetes, the Longhorn CSI driver communicates this request to the Longhorn Manager. The Manager then provisions a new Longhorn volume and a corresponding Longhorn Engine.

The Engine stores the actual data of the volume and manages the replication to the Longhorn Replicas spread across different nodes. This replication ensures that your data remains safe and available even if a node fails.

Moreover, when you take a snapshot, the Engine creates a point-in-time copy of the data without affecting the running workload. These snapshots can be used for backups, which are incremental and can be stored in secondary storage like AWS S3 or NFS server.

To sum it up, the architecture of Longhorn is built to operate seamlessly with Kubernetes, with a focus on simplicity, reliability, and ease of use. The system ensures your data is always safe, available, and can be managed effortlessly right from the Kubernetes interface or via Kubernetes APIs.

Installing Longhorn

Prerequisites

Before installing Longhorn, ensure that your Kubernetes cluster meets the following requirements:

Kubernetes v1.14 or higher
At least 1 worker node
Helm 3 installed (If you plan to use Helm for installation)
Open-iscsi installed on all nodes (in most cloud Kubernetes services like GKE, EKS, AKS, this comes preinstalled)

Installation Steps

Using kubectl

First, download the Longhorn YAML file from the Longhorn releases page. For example, if you’re installing Longhorn 1.1.0, you would use:

https://raw.githubusercontent.com/longhorn/longhorn/v1.1.0/deploy/longhorn.yaml

Then, apply the YAML file with kubectl:

kubectl apply -f longhorn.yaml

Using Helm

First, add the Longhorn Helm repository:

helm repo add longhorn https://charts.longhorn.io

Then, update your Helm repository:

helm repo update

Create the namespace that longhorn will be installed to

kubectl create namespace longhorn-system

Finally, install Longhorn:

helm install longhorn longhorn/longhorn --namespace longhorn-system

Now that Longhorn is installed we can view the pods running

kubectl get pods -n longhorn-system

Next we can view the services and be able to view the longhorn ui

kubectl get svc -n longhorn-system

Next we want to view the longhorn ui. I will port forward but you can use any applicable form

kubectl port-forward -n longhorn-system svc/longhorn-frontend 8080:80

Get the IP address and view the UI

From the dashboard, we can view different aspects. From the top bar, you can click on backups and be able to view different backups done or restore to the previous backup

Creating a Volume in Longhorn

From the UI you can create volumes or via the Kubernetes Persistent Volume Claim (PVC)

Click on the Volume tab at the top of the page.
Click on the Create button. This will open the Create Volume page.
Fill in the required details like Name, Size, Number of Replicas etc. Then click on Create

Through a Kubernetes Persistent Volume Claim (PVC):

You can also create a Longhorn volume by creating a PVC in Kubernetes. Here is an example of how to do this:

Create a PersistentVolumeClaim YAML file. For example, you might name it longhorn-volume-pvc.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: longhorn-vol-claim
spec:
  storageClassName: longhorn
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

Apply the PVC yaml

kubectl apply -f longhorn-volume-pvc.yaml

You can now view the PVC from the dashboard and play around with it you can create backups and restore them.

Closing Remarks

As we bring our discussion on Longhorn to a close, it’s clear to see the many advantages this cloud-native, lightweight, and easy-to-use storage system brings to Kubernetes environments. From its straightforward installation to its cloud-agnostic nature, right down to its high-availability and robust data protection features, Longhorn is built to simplify storage management for Kubernetes workloads.

With Longhorn, users can focus more on building their applications rather than worrying about the underlying storage layer. The features we’ve highlighted — such as incremental snapshot and backup, support for RWX volumes, and cross-cluster disaster recovery — are just a taste of what Longhorn can offer. Not to mention its intuitive user interface, which makes managing and troubleshooting your storage resources a breeze.

That being said, I believe Longhorn is a robust storage solution that can cater to both small-scale and enterprise-level Kubernetes deployments. If you haven’t tried it yet, I encourage you to do so and explore how it can streamline your storage management.

Remember, as with any technology, the best way to understand it is to get hands-on. Install Longhorn, create some volumes, and see how it improves your Kubernetes experience.

Thank you for joining me on this deep dive into Longhorn. We hope it has given you valuable insights and would love to hear your experiences and thoughts on this remarkable storage solution for Kubernetes. Stay tuned to this blog for more insights into the ever-evolving world of Kubernetes and its ecosystem.

Happy Kubernetes-ing!

How to Provision your EKS cluster using Terraform

Ian Kiprotich — Mon, 13 Mar 2023 22:45:18 +0000

In this article, we will be able to provision our cluster using terraform and make it ready for application deployments.

Prerequisite

Already have an account in AWS
Understand Terraform basics have a different blog that talks about what terraform is and how it is used you can find it here.
Basic Knowledge of kubectl

AWS’s Elastic Kubernetes Service (EKS) is a comprehensive and sophisticated managed service that enables you to effortlessly deploy, oversee, and grow containerized applications on Kubernetes with ease.

This tutorial will walk you through the process of deploying an EKS cluster using Terraform, a robust infrastructure automation tool. Afterwards, you will configure kubectl with Terraform output and confirm that your cluster is ready for use.

Why Deploy With Terraform

Why choose to use Terraform for deploying EKS clusters? While it’s possible to use AWS’s built-in provisioning methods (such as the UI, CLI, or CloudFormation), Terraform provides several benefits:

*Unified Workflow *— If you already utilize Terraform for deploying AWS infrastructure, you can use the same process for deploying both EKS clusters and the applications that run on them.

Lifecycle Management — Terraform handles the creation, updates, and removal of tracked resources, without requiring manual API inspection to identify those resources.

Graph of Relationships — Terraform identifies and observes the dependencies between resources. For instance, if an AWS Kubernetes cluster requires specific VPC and subnet configurations, Terraform won’t attempt to create the cluster unless the VPC and subnet have been provisioned first.

First clone this repo

https://github.com/onai254/terraform-eks-provision/tree/main

Then view the files in your preferred editor and move to the directory where the files are located in the terminal.

The configuration for this deployment specifies a fresh VPC where the cluster will be provisioned. The public EKS module is used to build the necessary resources such as Auto Scaling Groups, security groups, and IAM Roles and Policies.

You can inspect the module configuration by opening the main.tf file. The eks_managed_node_groups parameter is set to deploy three nodes distributed over two node groups for the cluster.

Lets look at each component in the repository

Main.tf

provider "aws" {
  region = var.region
}

data "aws_availability_zones" "available" {}

locals {
  cluster_name = var.Cluster_name
}

resource "random_string" "suffix" {
  length  = 8
  special = false
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "3.19.0"

  name = var.VPC_name

  cidr = "10.0.0.0/16"
  azs  = slice(data.aws_availability_zones.available.names, 0, 3)

  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.4.0/24", "10.0.5.0/24", "10.0.6.0/24"]

  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true

  public_subnet_tags = {
    "kubernetes.io/cluster/${var.Cluster_name}" = "shared"
    "kubernetes.io/role/elb"                      = 1
  }

  private_subnet_tags = {
    "kubernetes.io/cluster/${var.Cluster_name}" = "shared"
    "kubernetes.io/role/internal-elb"             = 1
  }
}

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "19.5.1"

  cluster_name    = var.Cluster_name
  cluster_version = "1.24"

  vpc_id                         = module.vpc.vpc_id
  subnet_ids                     = module.vpc.private_subnets
  cluster_endpoint_public_access = true

  eks_managed_node_group_defaults = {
    ami_type = "AL2_x86_64"

  }

  eks_managed_node_groups = {
    one = {
      name = "node-group-1"

      instance_types = ["t3.small"]

      min_size     = 2
      max_size     = 4
      desired_size = 3
    }

    two = {
      name = "node-group-2"

      instance_types = ["t3.small"]

      min_size     = 2
      max_size     = 4
      desired_size = 3
    }
  }
}

# https://aws.amazon.com/blogs/containers/amazon-ebs-csi-driver-is-now-generally-available-in-amazon-eks-add-ons/ 
data "aws_iam_policy" "ebs_csi_policy" {
  arn = "arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy"
}

module "irsa-ebs-csi" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-assumable-role-with-oidc"
  version = "4.7.0"

  create_role                   = true
  role_name                     = "AmazonEKSTFEBSCSIRole-${var.Cluster_name}"
  provider_url                  = module.eks.oidc_provider
  role_policy_arns              = [data.aws_iam_policy.ebs_csi_policy.arn]
  oidc_fully_qualified_subjects = ["system:serviceaccount:kube-system:ebs-csi-controller-sa"]
}

resource "aws_eks_addon" "ebs-csi" {
  cluster_name             = var.Cluster_name
  addon_name               = "aws-ebs-csi-driver"
  addon_version            = "v1.5.2-eksbuild.1"
  service_account_role_arn = module.irsa-ebs-csi.iam_role_arn
  tags = {
    "eks_addon" = "ebs-csi"
    "terraform" = "true"
  }
}

This Terraform code provisions an Amazon Elastic Kubernetes Service (EKS) cluster with an EBS CSI driver. The code defines a new VPC, with private and public subnets, and uses a public EKS module to create the required resources, including auto-scaling groups, security groups, and IAM roles and policies.

The eks_managed_node_groups parameter configures the EKS cluster with three nodes across two node groups. The code also creates an IAM role with the necessary permissions to use the EBS CSI driver, and attaches this role to a Kubernetes service account. Finally, the code provisions the EBS CSI driver as an EKS add-on using the previously created IAM role.

Overall, this code enables the creation of a fully-managed EKS cluster with EBS CSI driver support using Terraform, providing a unified workflow and full lifecycle management capabilities.

Output.tf

output "cluster_endpoint" {
  description = "Endpoint for EKS control plane"
  value       = module.eks.cluster_endpoint
}

output "cluster_security_group_id" {
  description = "Security group ids attached to the cluster control plane"
  value       = module.eks.cluster_security_group_id
}

output "region" {
  description = "AWS region"
  value       = var.region
}

output "cluster_name" {
  description = "Kubernetes Cluster Name"
  value       = module.eks.cluster_name
}

These are the output values that Terraform will show after successfully running the code.

“cluster_endpoint”: This is the endpoint for EKS control plane.
“cluster_security_group_id”: This is the security group ids attached to the cluster control plane.
“region”: This is the AWS region used in the deployment.
“cluster_name”: This is the Kubernetes Cluster Name used in the deployment.

These outputs are useful for accessing the deployed resources and verifying the deployment.

Terraform.tf

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.47.0"
    }

    random = {
      source  = "hashicorp/random"
      version = "~> 3.4.3"
    }

    tls = {
      source  = "hashicorp/tls"
      version = "~> 4.0.4"
    }

    cloudinit = {
      source  = "hashicorp/cloudinit"
      version = "~> 2.2.0"
    }
  }

  required_version = "~> 1.3"
}

This is the Terraform configuration block that declares the required providers and the minimum required version of Terraform. In this specific example, it requires the AWS, Random, TLS, and Cloudinit providers at specific version constraints to be installed in order to run the Terraform code. Additionally, it sets the required version of Terraform to be at least version 1.3.

Variable.tf

variable "region" {
  description = "AWS region"
  type        = string
  default     = "us-east-2"
}

variable "Cluster_name" {
  description = "Cluster Name"
  type = string
  default = "Giovanni"
}

variable "VPC_name" {
  description = "VPC Name"
  type = string
  default = "VPC_giovanni"

}

These are the variable definitions in the Terraform code:

region: A string variable that specifies the AWS region to use. It has a default value of "us-east-2" and a description explaining what it is used for.
Cluster_name: A string variable that specifies the name of the Kubernetes cluster. It has a default value of "Giovanni" and a description explaining what it is used for.
VPC_name: A string variable that specifies the name of the VPC. It has a default value of "VPC_giovanni" and a description explaining what it is used for.

These variables can be overridden when the Terraform module is used, allowing users to customize the infrastructure to their needs. The descriptions provided can help users understand what each variable is used for and make informed decisions when setting their values.

Lets now initialize our environment

In order for us to configure the infrastructure we need first to initialize the environment before

terraform init

terraform init is a command used to initialize a new or existing Terraform working directory. It is typically the first command to run after writing or cloning a Terraform configuration, or when starting to work on a new module or environment.

When you run terraform init, it will perform the following steps:

Download and install the required provider plugins and modules declared in the configuration.
Initialize a backend if one is not already configured, which is used to store the state of the infrastructure that Terraform manages.
Check for updates to the installed provider plugins and modules, and display a message if updates are available.

Running terraform init is a necessary step before using any other Terraform command, and it should be run again whenever new providers or modules are added or updated in the configuration.

Terraform plan

terraform plan

terraform plan will show you the execution plan for the infrastructure changes that will be applied when you run terraform apply. It will check your configuration files and compare them with the current state of your infrastructure, and then display a summary of the changes that Terraform will make to bring your infrastructure to the desired state.

Before running terraform plan, you should make sure that you have initialized your Terraform working directory by running terraform init and that your AWS credentials are properly configured.

Assuming your configuration files are in the current directory, you can run terraform plan by opening a terminal or command prompt, navigating to the directory where your configuration files are located, and running the following command:

Terraform apply

terraform apply

When you run the terraform apply command, Terraform will compare the current state of your infrastructure, as defined by your Terraform configuration files (in this case, your main.tf file), with the desired state described in the configuration.

Terraform will then determine what changes need to be made to the current infrastructure to bring it into the desired state, and prompt you to approve these changes before applying them.

If you approve the changes, Terraform will make the necessary API calls to create, update, or delete resources as needed.

Once the apply is complete, Terraform will output any relevant information, such as the IDs of the resources it created, and update the state file with the new state of the infrastructure.

Configuring your Kubectl

Once your cluster has been created using Terraform, you’ll need to configure kubectl to interact with it. To do this, you'll need to open the outputs.tf file to review the output values. In particular, you'll use the region and cluster_name outputs from this file to configure kubectl

Run the following command to retrieve the access credentials for your cluster and configure kubectl.

aws eks --region $(terraform output -raw region) update-kubeconfig \
    --name $(terraform output -raw cluster_name)

You can now use your kubectl to manage your cluster and deploy Kubernetes configurations to it.

You can then verify the cluster if it is configured correctly

kubectl cluster-info

Notice that the Kubernetes control plane location matches the cluster_endpoint value from the terraform apply output above.

You can then verify the worker nodes as part of the cluster

kubectl get nodes

Terraform Destroy

terraform destroy

Destroy the resources you created in this tutorial to avoid incurring extra charges. Respond yes to the prompt to confirm the operation.

That was easy with a few steps you are able to have a cluster up and running. Check out my other blogs and follow me on Twitter and LinkedIn for more content. You can also check the UI in AWS cluster to check your cluster if it is running.

How to Deploy Prometheus Operator in Your Kubernetes Cluster EKS

Ian Kiprotich — Mon, 13 Mar 2023 21:26:46 +0000

In this blog we will be able to deploy prometheous operator in our EKS cluster and be able to view different dashboards like the grafana and the prometheus

Preriqities

EKS cluster running

What is Prometheus Operator

Let's first define what Prometheus Operator is. It is an open-source project that simplifies the deployment and management of Prometheus and related monitoring components on Kubernetes.

It simplifies the deployment and management of Prometheus-based monitoring by automating tasks such as setting up monitoring targets, configuring alerting rules, and scaling the Prometheus servers.

Prometheus Operator is widely used in Kubernetes clusters to monitor applications and infrastructure. It is particularly useful for microservices-based architectures where applications are deployed as Kubernetes services or deployments.

Simply put: the core benefit of Prometheus Operator is simple and scalable deployment of a full Prometheus monitoring stack.

Traditionally, without Prometheus Operator customization and configuration of Prometheus is complex. With Prometheus Operator, K8s custom resources allow for easy customization of Prometheus, Alertmanager, and other components. Additionally, the Prometheus Operator Helm Chart includes all the dependencies required to stand up a full monitoring stack.

Prometheus Operator allows users to easily configure and manage Prometheus and related monitoring tools, including Grafana dashboards, Alertmanager, and exporters.

It provides a set of Kubernetes custom resources, such as Prometheus, Alertmanager, and ServiceMonitor, which can be used to configure and manage Prometheus instances in a Kubernetes environment.

Here are some of the benefits of using the Prometheus Operator:

Simplified configuration: With the Prometheus Operator, you can define and manage Prometheus instances using Kubernetes custom resources, which are easier to manage and maintain than traditional YAML manifests.
Automated management: The Prometheus Operator automates many of the tasks involved in deploying and managing Prometheus instances, such as scaling, rolling upgrades, and configuration changes.
Enhanced security: The Prometheus Operator provides built-in security features, such as authentication and authorization, that can help you secure your Prometheus instances.
Extensible monitoring: The Prometheus Operator makes it easy to extend the monitoring capabilities of Prometheus by integrating with other monitoring tools and services, such as Grafana and Kubernetes metrics.

To use the Prometheus Operator, you need to install it in your Kubernetes cluster and then create custom resources to define and configure your Prometheus instances.

Here are the basic steps:

Install the Prometheus Operator: You can install the Prometheus Operator using the helm package manager or by deploying the Kubernetes manifests directly.
Define a Prometheus resource: Use the Prometheus custom resource to define your Prometheus instance's configuration, such as the data retention period, alerting rules, and scrape targets.
Define an Alertmanager resource: Use the Alertmanager custom resource to define the configuration for your Alertmanager instance, such as the receivers and notification channels.
Define a ServiceMonitor resource: Use the ServiceMonitor custom resource to define the monitoring configuration for your applications running in Kubernetes.
Deploy and manage: The Prometheus Operator will create and manage your Prometheus instances and related components based on the custom resources you define.

By using the Prometheus Operator, you can simplify the deployment and management of Prometheus instances in Kubernetes and take advantage of Kubernetes’ native features for monitoring and scaling.

Understanding Prometheus Operator CRD

Prometheus Operator orchestrates Prometheus, Alertmanager and other monitoring resources by acting on a set of Kubernetes Custom Resource Definitions (CRDs).

Understanding what each of these CRDs does will allow you to better optimize your monitoring stack. The supported CRDs are:

Prometheus- Defines the desired state of a Prometheus Deployment
**Alertmanager- **Defines the desired state of a Alertmanager Deployment
**ThanosRuler- **Defines the desired state of a ThanosRuler Deployment
**ServiceMonitor- **Declaratively specifies how groups of Kubernetes services should be monitored. Relevant Prometheus scrape configuration is automatically generated.
**PodMonitor- **Declaratively specifies how groups of Kubernetes pods should be monitored. Relevant Prometheus scrape configuration is automatically generated.
**Probe- **Declaratively specifies how ingresses or static targets should be monitored. Relevant Prometheus scrape configuration is automatically generated.
**PrometheusRule- **Defines the desired state of a Prometheus Alerting and/or recording rules. Relevant Prometheus rules file is automatically generated.
**AlertmanagerConfig- **Declaratively specifies subsections of the Alertmanager configuration, allowing routing of alerts to custom receivers, and setting inhibit rules.

Installing Prometheus Operator using Helm

To kickstart your monitoring stack, the initial step would be to deploy Prometheus Operator and its corresponding Custom Resource Definitions (CRDs) in your Kubernetes cluster. In order to establish a complete monitoring stack, Prometheus necessitates the deployment of Grafana, node-exporter, and kube-state-metrics. The good news is that all of these essential components are included as dependency charts in the Prometheus Operator Helm Chart, which facilitates their automatic installation and integration with Prometheus Operator.

Before installing the Prometheus operator make sure your cluster is up and running since the Prometheus operator requires more resources then start the cluster with more resources using the — resource field.

minikube start --cpus 4 --memory 4096

The above command will create a cluster with 4cpus and a memory of 4096

$ helm install prom-operator-run prometheus-community/kube-prometheus-stack

Give it a minute or so to install all the Prometheus components after that run the

kubectl get pods

You should be able to see the different components installed by Helm. All the components from StatefullSet to the pods.

From the above, we see different pods created by Prometheus Operator. The operator creates other things also in our cluster you can view all by

kubectl get all

Configure-port forwarding for Grafana

Since we have already installed the different components of Prometheus using Prometheus Operator we can now be able to view the Grafana dashboard in our cluster using port forwarding since the service available is the ClusterIP

The command for accessing the Grafana UI interface is

kubectl port-forward svc/prom-operator-01-grafana 3000:80

Open the browser and go to localhost 3000 The prompt for Grafana login is displayed

Username = admin

Password = prom-operator

You can also be able to change the password.

with the Grafana up and running we can be able to view different scraped data from Prometheus, edit the dashboard and so on.

You can also view the Prometheus UI also by port forwarding.

Important notes

Prometheus operator watches for namespaces for the Prometheus/Alertmanager CR. In most scenarios, it watches all namespaces but you can configure it to watch for a particular namespace as well. This can be changed by using the --namespaces= flag on the Prometheus operator namespace.
spec.selector.matchLabels MUST match your app’s name (k8s-app: my-app in our example) for ServiceMonitor to find the corresponding endpoints during deployment.
You can access the Prometheus UI by port-forwarding to the Prometheus container or creating a service on top of it. In a production scenario, you should not expose Prometheus since it should only act as a Grafana data-source. However, if you need to expose it is recommended to use an Ingress.
Make sure Prometheus instances are configured to store metric data in persistent volumes so that it is preserved in between restarts.

Next, we will upload an application in the Kubernetes cluster and be able to monitor it using Prometheus.

Understand Networking in Kubernetes

Ian Kiprotich — Mon, 13 Mar 2023 21:12:51 +0000

Networking in Kubernetes is an important topic to understand, as it enables applications running on a Kubernetes cluster to communicate with each other, as well as with external services.

In a Kubernetes cluster, each pod has its own unique IP address that is routable within the cluster. This IP address is typically assigned by the Kubernetes network plugin, which is responsible for configuring the networking for the cluster. The network plugin is a key component of the Kubernetes networking architecture, and there are multiple options available, such as Calico, Flannel, and Weave Net.

In addition to the network plugin, Kubernetes also includes a number of other components that are involved in networking, such as the kube-proxy, which is responsible for load balancing network traffic across multiple pods, and the Service object, which provides a stable IP address and DNS name for a set of pods.

Understanding how networking works in Kubernetes is important for anyone working with Kubernetes, as it can have a significant impact on the performance and reliability of applications running on the cluster.

Ingress

In Kubernetes, an Ingress is an API object that provides a way to configure external access to services in a cluster. An Ingress resource typically defines a set of rules that specify how incoming traffic should be routed to the appropriate service, based on the incoming request’s host and path. It exposes HTTP and HTTPS routes from outside the cluster to services within the cluster.

When a request comes into the Kubernetes cluster, it first hits the Ingress controller, which is responsible for managing and routing traffic according to the rules defined in the Ingress resource. The Ingress controller uses a set of rules defined in the Ingress resource to determine where the traffic should be routed. In order for ingress to work you need to use an Ingress controller i.e Nginx, AWS.

Once the Ingress controller has determined the appropriate service to route the traffic to, it forwards the request to the service’s ClusterIP, which is an internal IP address assigned to the service by the Kubernetes network plugin. The service then uses its own rules to determine which pod(s) to forward the traffic to.

The traffic then reaches the target pod, where it is handled by the application running inside the pod. The application’s response is then sent back through the same path, via the service and the Ingress controller, to the original request sender.

It’s important to note that the Ingress resource is only responsible for routing traffic to services within the cluster, and does not handle any authentication or security. Therefore, it is often used in combination with other security measures such as TLS termination or Web Application Firewalls (WAFs) to provide a secure and reliable service.

DNS — Domain Name System

Domain Name System (DNS) is used to provide a way for services and pods to discover and communicate with each other. It is a service that is responsible for translating or resolving a service name to its IP address. Each pod in a Kubernetes cluster is assigned a unique hostname, which is derived from the pod’s name and the namespace it is running in. By default, each hostname is resolvable within the cluster’s DNS namespace.

When a service is created in Kubernetes, it is assigned a stable DNS name that is used to access the service from other pods and services within the cluster.

The DNS name is typically in the format servicename.namespace.svc.cluster.local, where servicename is the name of the service,

namespace is the Kubernetes namespace in which the service is running, cluster.local is the default DNS domain for the cluster.

When a pod wants to communicate with a service, it can simply use the service’s DNS name to connect to it. The Kubernetes DNS service will resolve the DNS name to the corresponding ClusterIP assigned to the service, and route the traffic to the appropriate pod.

In addition to resolving DNS names for services, Kubernetes also supports custom DNS configurations, which allow you to specify additional DNS servers or search domains that should be used when resolving DNS queries. This can be useful if you need to resolve DNS names outside of the Kubernetes cluster, such as for accessing external services or APIs.

Overall, DNS is a critical component of networking in Kubernetes, as it allows services and pods to discover and communicate with each other, and is an essential part of the infrastructure for deploying reliable and scalable applications on a Kubernetes cluster.

Core DNS

CoreDNS is a popular DNS server implementation used in Kubernetes for service discovery and DNS resolution. It is the default DNS server for Kubernetes and ensures pods and services have a Fully Qualified Domain Name (FQDN). CoreDNS is a flexible and extensible DNS server that is designed to be easily integrated into Kubernetes clusters and can be customized to support a wide range of use cases. Without CoreDNS the cluster's communication would cease to work.

In Kubernetes, CoreDNS is typically deployed as a pod in the cluster and is responsible for resolving DNS queries for services and pods. CoreDNS uses the Kubernetes API to retrieve information about services and pods and automatically generates DNS records for each of them.

One of the benefits of using CoreDNS in Kubernetes is that it is highly configurable, and can be extended to support custom plugins and DNS providers. For example, you can use CoreDNS plugins to add support for custom DNS zones or to integrate with external DNS providers.

Another benefit of CoreDNS is that it provides better performance and scalability than the previous default DNS server in Kubernetes, kube-dns. CoreDNS is written in Go and is designed to be lightweight and efficient, which makes it well-suited for handling large volumes of DNS queries in high-traffic Kubernetes environments.

To use CoreDNS in your Kubernetes cluster, you can deploy it as a pod using a Kubernetes manifest file or Helm chart. Once deployed, you can configure the CoreDNS server to meet your specific needs, such as by adding custom DNS providers, defining custom DNS zones, or integrating with other Kubernetes components such as Ingress or ExternalDNS.

Overall, CoreDNS is a powerful and flexible DNS server implementation that is well-suited for use in Kubernetes clusters and provides a solid foundation for service discovery and DNS resolution in modern cloud-native applications.

FQDN — Fully Qualified Domain Name is a domain name that specifies its exact location in the tree hierarchy, also known as an absolute domain functionality of core DNS.

Probes

In Kubernetes, probes are used to determine the health of a container running in a pod. Probes are a critical part of Kubernetes’ self-healing and auto-scaling capabilities, as they provide a way for the cluster to automatically detect and recover from unhealthy containers. Probes are used to detect the state of a container.

There are three types of probes in Kubernetes:

Liveness Probe

This type of probe is used to determine if a container is still running and is healthy. The liveness probe sends periodic requests to the container, and if the container fails to respond or responds with an error, the probe will mark the container as unhealthy and trigger a restart.

A Liveness probe could catch a deadlock where an application is running but unable to make any progress instead it restarts the container this can make the application more available despite the bugs.

Readiness Probe

This type of probe is used to determine if a container is ready to start receiving traffic. The readiness probe sends periodic requests to the container, and if the container responds with a success code, it will be marked as ready to receive traffic. If the container fails to respond or responds with an error, it will be marked as not ready, and will not receive any traffic until it becomes ready again.

Startup Probe

This type of probe is used to determine if a container is in the process of starting up. The startup probe sends periodic requests to the container, and if the container responds with a success code, it will be marked as ready to receive traffic. If the container fails to respond or responds with an error, the startup probe will keep sending requests until the container becomes ready, or until a configurable timeout is reached.

Probes in Kubernetes are defined in the pod’s spec section, using YAML configuration. Each probe is defined with a set of parameters, such as the type of probe, the endpoint to probe, the probe timeout, the probe period, and the success and failure thresholds.

Overall, probes are a powerful feature in Kubernetes that enable containers to be automatically monitored and restarted in the event of failures or unresponsiveness, which helps to improve the reliability and availability of applications running on Kubernetes clusters.

Netfilter

In Kubernetes, Netfilter is used to implement network policies, which are used to control the flow of network traffic between pods in a cluster. Network policies are Kubernetes objects that define rules for how traffic is allowed to flow between pods, and they use Netfilter rules to enforce those policies.

When a network policy is applied to a Kubernetes cluster, the Kubernetes API server communicates with the network plugin, which is responsible for configuring the networking rules in the underlying network infrastructure. The network plugin, in turn, generates Netfilter rules to enforce the network policy.

The Netfilter rules generated by the network plugin are based on the selectors specified in the network policy. Selectors are used to identify which pods should be affected by the policy, and they can be based on a wide range of criteria, such as the pod’s labels, namespace, or IP address. The network plugin generates Netfilter rules that match the specified selectors, and then applies the action specified in the policy to the matching packets.

For example, a network policy might be defined to allow traffic to flow only between pods that have a specific label. The network plugin would generate Netfilter rules to match packets between pods with that label, and then allow those packets to flow through the network. Similarly, a network policy might be defined to deny traffic between two specific pods, in which case the network plugin would generate Netfilter rules to drop packets between those pods.

Overall, Netfilter is a critical component of the network policy implementation in Kubernetes, as it allows for granular control over the flow of network traffic between pods in a cluster, and provides a powerful mechanism for enforcing security and access control policies.

IPTables

IPTables is a Linux-based firewall tool that allows users to configure and manage network filtering rules. In Kubernetes, IPTables are used to implement network policies, which control the flow of traffic between pods and services.

When a network policy is created in Kubernetes, the kube-proxy component creates IPTables rules to enforce the policy. These rules are applied to network traffic as it passes through the node where the pod or service is located.

The IPTables rules generated by Kubernetes are based on the network policy’s selectors and rules. Selectors identify which pods the policy applies to, while rules define what traffic should be allowed or denied. For example, a network policy could be created that only allows traffic to specific ports on pods with a certain label.

The IPTables rules generated by Kubernetes are inserted into the kernel’s IPTables chains, which determine how network traffic is processed. These chains are evaluated in a specific order, with the first matching rule determining the action taken on the packet.

Kubernetes also uses IPTables to implement Kubernetes Services, which provide a stable IP address and DNS name for accessing a set of pods. When a Service is created in Kubernetes, kube-proxy creates an IPTables rule to forward traffic to the appropriate pod based on the Service’s selector.

Overall, IPTables are an important tool for implementing network policies and Services in Kubernetes, as it allows for fine-grained control over the flow of network traffic, and provides a reliable and scalable mechanism for load balancing and service discovery.

IPVS

IPVS (IP Virtual Server) is a Linux kernel module that provides network load-balancing capabilities. In Kubernetes, IPVS is used as an alternative to kube-proxy and IPTables for implementing Services.

When a Service is created in Kubernetes and the Service type is set to “LoadBalancer”, IPVS is used to create a virtual IP address (VIP) for the Service. The VIP is used as the target address for client traffic and is associated with a set of pods that provide the actual service.

IPVS works by intercepting incoming traffic to the VIP and distributing it among the available pods using a load-balancing algorithm. There are several load-balancing algorithms available in IPVS, including round-robin, least-connection, and weighted least-connection.

IPVS also provides health checks to ensure that traffic is only sent to healthy pods. When a pod fails a health check, IPVS removes it from the list of available pods and redistributes traffic among the remaining healthy pods.

IPVS has several advantages over kube-proxy and IPTables, including better scalability and performance, and more flexible load-balancing algorithms. IPVS can handle large numbers of connections and is optimized for high throughput and low latency. It also supports more advanced load-balancing features, such as session persistence and connection draining.

However, IPVS requires additional configuration and setup compared to kube-proxy and IPTables, and may not be compatible with all network environments. IPVS also requires kernel support and may not be available on all Linux distributions.

Proxy

A proxy is a server application that acts as an intermediary between client requesting a resource and the server providing that resource

Kubectl Proxy

The Kubectl Proxy is a command-line tool that enables a user to create a secure tunnel between their local machine and a Kubernetes API server. This allows the user to access the Kubernetes API server without the need for direct network access or complex authentication configurations. Kubectl Proxy is used for various purposes, such as accessing the Kubernetes Dashboard or using kubectl commands against a remote cluster.

For example, suppose a user wants to access the Kubernetes Dashboard running on a remote cluster. They can use Kubectl Proxy to create a secure tunnel and then access the Dashboard through a local web browser.

Kube-Proxy

On the other hand, Kube-Proxy is a component that runs on each node in a Kubernetes cluster and is responsible for implementing Kubernetes Services. Kube-Proxy listens for changes to Services and then updates the local IPTables or IPVS rules accordingly. This ensures that traffic is correctly routed to the appropriate pods in the cluster.

For example, suppose a Service is created in Kubernetes that maps to a set of pods with the label “app=myapp”. Kube-Proxy will create IPTables or IPVS rules that direct traffic to the appropriate pod based on the Service’s selector.

Both Kubectl Proxy and Kube-Proxy have benefits and limitations. Kubectl Proxy is simple to set up and provides secure access to the Kubernetes API server, but it can be slow and may not be suitable for production environments.

Kube-Proxy is reliable and scalable, but it can be complex to configure and may not be suitable for all network environments.

Envoy

In addition to Kube-Proxy, another popular proxy used in Kubernetes is Envoy. Envoy is a high-performance proxy that provides advanced traffic management and load-balancing capabilities. Envoy can be used as a replacement for Kube-Proxy to implement Kubernetes Services or can be used as an independent component to provide advanced traffic management features.

Envoy is used in many production environments and can provide benefits such as advanced load-balancing algorithms, circuit breaking, and distributed tracing.

However, Envoy requires additional setup and configuration compared to Kube-Proxy, and may not be compatible with all network environments. Additionally, Envoy is generally used in more complex scenarios, such as multi-cluster or multi-cloud environments, and may be overkill for simpler use cases.

Container Networking Interface

The Container Networking Interface (CNI) is a specification and set of tools for configuring networking in containerized environments, such as those provided by Kubernetes. The goal of CNI is to provide a common standard for network plugins so that container runtimes and orchestration systems can work with any networking solution that supports the CNI API.

CNI defines a standard way for container runtimes, such as Docker or CRI-O, to call networking plugins to configure the network interfaces of containers. The plugins are responsible for creating and configuring network interfaces for the containers, as well as configuring the network namespace and routing tables.

In Kubernetes, CNI is used by the kubelet to configure the network interfaces of pods. When a pod is created, the kubelet invokes the CNI plugin to configure the pod’s network. The CNI plugin then creates and configures the network interfaces for the pod, sets up any necessary routing rules, and adds the pod’s IP address to the appropriate network namespace.

CNI plugins can be either built into the container runtime or provided as standalone binaries. There are many CNI plugins available, each with its own strengths and weaknesses. Some popular CNI plugins include Calico, Flannel, and Weave Net.

The use of CNI provides several benefits in containerized environments. First, it allows for a common standard that can be used by multiple container runtimes and orchestration systems. This means that network plugins can be developed independently of the container runtime or orchestration system, which promotes flexibility and compatibility.

Second, CNI provides a modular and extensible architecture that allows for easy integration with other networking solutions. This enables users to choose the best networking solution for their specific use case and avoid vendor lock-in.

Finally, CNI provides a simple and flexible API for configuring container networking, which makes it easy for developers to create and deploy custom networking solutions tailored to their needs.

How to Monitor your Application using Prometheus

Ian Kiprotich — Mon, 13 Mar 2023 21:04:27 +0000

In this Blog, we will be able to deploy our application in an EKS cluster and monitor it with Prometheus
In the previous blog, we set up a Prometheus Operator to help us monitor our applications. Now, we’ll deploy our application and use Prometheus to monitor it.

The prerequisites you need to archive this goal are:

You should be having an EKS cluster running
Your application yaml files

Before we deploy the application let's talk about the ground things such as ServiceMonitor

ServiceMonitor

ServiceMonitor is a Kubernetes object that specifies how Prometheus should monitor a set of targets that belong to a Kubernetes service.

For example, suppose we have a Kubernetes service called “myapp” that has several replicas running across different pods. We want to monitor these replicas using Prometheus. To do this, we can create a ServiceMonitor object in Kubernetes that specifies the labels and endpoints for the pods running the “myapp” service.

Here’s an example ServiceMonitor YAML file:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: myapp-monitor
  namespace: mynamespace
  labels:
    app: myapp
spec:
  selector:
    matchLabels:
      app: myapp
  endpoints:
  - port: web
    interval: 30s
    path: /metrics

In this example, we define a ServiceMonitor called “myapp-monitor” in the “mynamespace” namespace. We specify that we want to monitor the pods that have the label “app=myapp”. We also specify that we want to scrape the “web” port on these pods every 30 seconds and that the metrics can be found at the “/metrics” endpoint.

Once we apply this ServiceMonitor YAML file to Kubernetes, Prometheus will automatically discover and monitor the targets that belong to the “myapp” service based on the label selector defined in the ServiceMonitor.

Overall, ServiceMonitors provide an easy way to configure Prometheus to scrape metrics from multiple pods that belong to a single Kubernetes service, allowing for more efficient and scalable monitoring of Kubernetes-based applications.

Since we have already configured PrometheusOperator in the cluster and we already viewed the Grafana Dashboard let us now view the Prometheus dashboard and see all the metrics being scraped by Prometheus

Prometheus Dashboard

To display Prometheus Dashboard we need to pass the port-forward attribute to the Prometheus service.

First, get the service name and the port running

Then expose the port outside using port-forward

kubectl port-forward svc/prometheus-operated 9090 -n monitor

The IP address is displayed go check out Prometheus

From the dashboard, we can check on the service discovery that Prometheus has discovered.

You can also check on the targets that Prometheus is scraping with the /metrics endpoint.

In our demo, we will deploy a microservice architecture and crape the metrics using Prometheus. But one more thing before we deploy our application it is important to know what Exporters are.

Exporters

What are exporters in Prometheus?

Exporters are software components that expose metrics in a format that can be understood by Prometheus. They act as intermediaries between Prometheus and the systems and applications that are being monitored, and they provide a standard way to collect metrics from a wide variety of sources.

Node-Exporter they are translator between application metrics to data that Prometheus will understand. An exporter itself will expose /metrics endpoint so that Prometheus can be able to scrape from there. They collect metrics data from applications and they can be a separate deployment in the cluster.

In the context of Kubernetes, exporters are typically deployed as sidecar containers in the same pod as the application or system component they are monitoring. The exporter is responsible for collecting metrics from the system or application and exposing them over HTTP or some other protocol, so that Prometheus can scrape them.

There are many different exporters available for Prometheus, covering a wide range of systems and applications. Some of the most common exporters include the Node Exporter (for collecting metrics from Linux servers), the Prometheus MySQL Exporter (for monitoring MySQL databases), and the Prometheus Blackbox Exporter (for monitoring network services).

How to use exporters in Kubernetes with Prometheus?

To use exporters in Kubernetes with Prometheus, you will typically follow these steps:

Identify the metrics you want to monitor, you will need to identify the metrics you want to monitor from your systems and applications. This will depend on the specific use case and the systems being monitored.
Install the appropriate exporter: Once you have identified the metrics you want to monitor, you will need to install the appropriate exporter for each system or application. This can be done using either the Helm chart or YAML manifests.
Configure Prometheus to scrape the exporter: Finally, you will need to configure Prometheus to scrape the metrics from the exporter. This is typically done by adding a scrape configuration to the prometheus.yaml file, specifying the URL of the exporter endpoint.

For example, if you want to monitor the CPU usage and memory usage of a Kubernetes pod, you can deploy the Node Exporter as a sidecar container in the same pod, and configure Prometheus to scrape the metrics from the Node Exporter’s endpoint. Once this is done, Prometheus will start collecting the metrics and storing them in its time series database, where they can be analyzed and visualized using Prometheus’s query language and dashboarding tools.

Why are exporters useful in Prometheus?

Exporters are useful in Prometheus for several reasons:

Support for diverse systems and applications: Exporters provide a standard way to collect metrics from a wide variety of systems and applications, making it easy to monitor different components of a Kubernetes environment.
Easy deployment and configuration: Exporters can be deployed as sidecar containers in Kubernetes pods, making it easy to configure and manage them alongside the systems and applications they are monitoring.
Consistent metric format: Exporters expose metrics in a consistent format that can be understood by Prometheus, making it easy to write queries and build dashboards that work across different systems and applications.
High performance: Exporters are typically designed to be highly performant and lightweight, so that they can collect metrics with minimal overhead and without impacting the performance of the systems they are monitoring.

In summary, exporters are a powerful and flexible feature of Prometheus that make it easy to monitor diverse systems and applications in Kubernetes environments. By using exporters, you can collect detailed metrics on the health and performance of your systems, and use this data to optimize and troubleshoot

Hands-on Demo

In the previous demo, we deployed Prometheus in our cluster we also exposed the ports for Grafana and Prometheus. Next let's install the application for this demo we will clone a git repository for a microservices application developed by google.

git clone https://github.com/GoogleCloudPlatform/microservices-demo

After installing the application we can view different components of the application

We can also view the application by port-forward, or using ingress or load balancers.

Next, we will install the exporter, and let's start with the Redis exporter. Open the Prometheus exporter page, we can search and find the Redis exporter. Exporters are also available as docker images and you can find them in the Docker Hub.

Components you need to have when deploying an exporter

The application itself in the docker image that exposes the /metrics endpoints
1. A service so that Prometheus can connect to the exporter
2. Service monitor to tell Prometheus that a new service has been created and it should start scraping it.

To install the Redis exporter in the most simplified way to use it we can search for a helm chart that is ready and configured to install the exporter.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

The above command basically installs the Prometheus helm charts although we had already installed the chart when installing Prometheus. Next, let's install the node exporter chart.

But before we deploy the exporter there are some values that we need to change from the chart. Let's create values.yaml configuration that we will use to pass while using helm to install the radis node exporter

redisAddress: redis://redis-cart:6379
#the service we will monitor and the port 

serviceMonitor:
  additionalLabels:
    release: prometheus

Save the file before running the helm install command giving your node exporter the name and passing the -f values.yaml file.

helm install redis-exporter prometheus-community/prometheus-redis-exporter -f values.yaml

You can then check the service monitor running in the cluster to see if we installed the Service monitor correctly.

Tomorrow we will finish from there: checking on the service discovery function and if Prometheus discovered our service monitor.

How to Deploy Istio to manage your Network in your Cluster

Ian Kiprotich — Mon, 13 Mar 2023 20:58:49 +0000

In our previous blog post, we discussed the installation of istioctl and provided an overview of the various components of Istio. In this blog post, our focus will be on the installation of Istio in our cluster and the execution of our initial application on Istio.

Prerequisite

In order to follow along with this demo you have to have the following

Your cluster can either be minikube or a cluster running in the cloud
Istioctl installed and configured
Your application yaml files

Installing Istio

In order to install Istio in our cluster we can run the command

istioctl install

In the above definition, we are running Isitio in the demo profile we also touched upon different Istio profiles and how we can use them in our previous blog

When we installed Istio in our cluster it created its own Istiosystem namespace and installed some pods. Now we can be able to view the pods installed in the Istio-system namespace.

kubectl get pods -n istio-system

Also, we can be able to verify and see all the components that Istio installed into our cluster by running the following commands

istioctl verify-install

These components are just a subset of those that will be installed. To proceed with the application installation, we need to modify the label to enable Istio to discover each microservice in our application. This can be achieved by performing the following steps. Run the following code to change the labels.

kubectl label namespace default istio-injection=enabled

The command is used to modify the label of the default namespace in Kubernetes and enable automatic sidecar injection of the Istio Envoy proxy for all the pods running in the namespace.

When executed, this command adds the label “istio-injection=enabled” to the default namespace’s metadata. This tells Istio to automatically inject the Envoy sidecar proxy into all pods that are created in the namespace. By default, Istio only injects the Envoy sidecar proxy into pods that have the “istio-injection=enabled” label set on their metadata.

Enabling automatic sidecar injection with this command simplifies the process of deploying microservices on the Istio service mesh. Once the default namespace is labeled, you can deploy your microservices as usual and Istio will automatically inject the Envoy sidecar proxy into each pod, allowing Istio to control and monitor network traffic between the microservices.

After we have labeled the default namespace we can now deploy the application and see what happens. In this example, we will use the demo application by Istio to understand more. After deploying the application check for the pods running.

In the provided example, we can see that there are pods currently active within the default namespace. Each of these pods contains two containers: the first container is responsible for executing the business logic of the pod, while the second container is an envoy proxy that has been automatically deployed on each pod. This automatic deployment was made possible by labeling the default namespace with “istio-injection=enabled”. Should additional pods be added to this namespace, the istio service discovery component will detect their presence and deploy a corresponding envoy container. We can also check this by describing the pod.

Kiali

Next, we will talk about Kiali and other add-ons in Istio. Kiali is a web-based graphical user interface (GUI) that provides observability and visualization features for Istio service mesh. Istio is a powerful service mesh platform that can help manage and secure microservices-based applications in a distributed system. Kiali is a tool that can help you visualize and understand the traffic flow, service dependencies, and health status of your Istio service mesh.

Kiali provides a number of useful features for Istio users, including:

Service topology visualization: Kiali can display a real-time topology map of the services in your Istio service mesh, including their dependencies and traffic flows.
Metrics and telemetry: Kiali can display detailed metrics and telemetry data for individual services and traffic flows, including traffic rates, latency, and error rates.
Distributed tracing: Kiali can integrate with distributed tracing systems like Jaeger to provide detailed tracing information for service calls and transactions.
Service health monitoring: Kiali can provide real-time health status for individual services and the overall service mesh, including alerts and notifications for service failures and other issues.
Configuration validation: Kiali can validate the configuration of your Istio service mesh, including routing rules, security policies, and other settings, to help ensure that your services are running as expected.

Overall, Kiali is a powerful tool for Istio users who need to monitor and manage complex microservices-based applications in a distributed system. By providing real-time visibility and insights into your Istio service mesh, Kiali can help you diagnose and resolve issues quickly, optimize your system performance, and ensure that your services are running smoothly and securely.

Installing Kiali

There are multiple methods available for installing the Kiali demonstration using helm charts or the downloaded addons folder as part of the Istio installation process. To continue, we will carry out the installation of Istio on our cluster.

kubectl apply samples/addons/kiali.yaml

We can verify if we have installed Kiali by checking the pods that are running in the istio-system

kubectl get pods -n istio-system

After installing Kiali in our cluster we can view the Dashboard by using the command.

istioctl dashboard kiali

This will automatically open the Kiali dashboard running in Localhost

After successfully configuring Kiali within our cluster and accessing the Kiali dashboard, we can proceed to direct traffic towards the application and monitor it using the dashboard. To achieve this, we need to create a Gateway that will enable our service mesh to receive traffic from external sources outside the cluster.

But first, lets get to know some terms and definitions in order to understand more about Traffic management in Istio

Gateway

A Gateway resource is used to configure load-balancing of traffic entering the service mesh from outside the cluster. It defines a set of protocols, ports, and workload instances to which traffic should be directed.

The Gateway acts as an entry point for external traffic to enter the service mesh, and it can be configured to perform various tasks, such as TLS termination, header manipulation, or authentication.

With the help of the Gateway resource, Istio can control and secure the incoming traffic to the service mesh, providing an additional layer of control and visibility over network traffic.

A Gateway resource in Istio consists of two main components:

A set of workload instances — that represent the backend services that will handle incoming traffic, along with the protocol and port numbers they are listening on.
A set of networking configuration parameters that define how incoming traffic should be handled, such as TLS encryption settings, routing rules, and load-balancing algorithms.

Here’s an example of how a Gateway is used in Istio:

Let’s say we have a service mesh consisting of several microservices that are running on Kubernetes pods. These microservices are accessible only within the cluster, and we want to enable external users to access them from the internet. To achieve this, we can create a Gateway resource that defines the external IP address, port number, and TLS settings, and associate it with a set of workload instances representing the backend services.

For example, let’s say we have a microservice called “product” that runs on Kubernetes pods and listens on port 8080. We can create a Gateway resource that defines an external IP address of 203.0.113.0 and port 80, and associate it with the “product” service. When a user sends a request to http://203.0.113.0:80/product, the Gateway resource will receive the request and route it to the backend “product” service.

We can also configure the Gateway resource to use TLS encryption to secure the traffic between the external user and the service mesh. For example, we can configure Istio to use a certificate authority to issue and manage SSL/TLS certificates, and configure the Gateway to use these certificates to encrypt and decrypt the incoming traffic.

In summary, a Gateway in Istio is a Kubernetes resource that enables external traffic to enter a service mesh, and it can be used to configure load balancing, routing, and security for incoming traffic.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: my-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "product"

The spec field has two main components:

selector: A set of labels used to select the gateway workload instances that will handle incoming traffic. In this example, it selects the default ingress gateway using the label istio: ingressgateway.
servers: A list of servers that the gateway will listen on, including the port number, protocol, and hostnames that should be routed to this server. In this example, it defines a server that listens on port 80 using HTTP protocol.

To view the gateway in Kubernetes run the command

kubectl get gateway

Virtual Service

A VirtualService is a Kubernetes resource that defines a set of routing rules for incoming traffic to a service mesh. It allows you to control the routing of network traffic within the mesh, such as directing traffic to different versions of a service or sending traffic to a specific subset of a service.

A VirtualService resource consists of two main components:

A set of routing rules that define how incoming traffic should be directed to one or more destination services. The routing rules can be based on various criteria such as HTTP headers, URI paths, or traffic weights.
A set of destination services that represent the backend services that will handle incoming traffic.

The VirtualService resource provides a powerful way to control traffic within a service mesh, allowing you to implement advanced traffic management features such as canary releases, A/B testing, blue-green deployments, and fault injection.

Here’s an example of how a VirtualService can be used in Istio:

Let’s say we have a service mesh consisting of several versions of a microservice called “product” running on Kubernetes pods. We want to direct incoming traffic to the latest version of the “product” service, while still allowing a small percentage of traffic to be sent to the previous version for canary testing. To achieve this, we can create a VirtualService resource that defines two subsets of the “product” service, representing the latest version and the previous version respectively, and set traffic weights to direct 99% of the traffic to the latest version and 1% of the traffic to the previous version.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: product
spec:
  hosts:
  - product-service
  http:
  - route:
    - destination:
        host: product-service
        subset: v1
      weight: 99
    - destination:
        host: product-service
        subset: v2
      weight: 1

In this example, we define a VirtualService resource called “product” that routes traffic to the “product-service”. It defines two routes, one for each subset of the “product-service”. The first route directs 99% of the traffic to the “v1” subset, while the second route directs 1% of the traffic to the “v2” subset. This allows us to test the new version of the “product” service with a small percentage of traffic, while still directing most of the traffic to the stable version.

In this next example, we will look at a service that can be accessed through various microservices

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: bookinfor
spec:
  hosts:
  - bookinfor.app
  gateway:
  - bookinfor-gateway
  http:
  - match:
    - uri:
        exact: /productpage
    - uri
        prefix: /static
    - uri
        exact: /login
    - uri
        exact: /logout
    - uri
        prefix: api/vi/products

   - route:
    - destination:
        host: productpage
        port:
           number: 9080

In the above YAML file we are routing all traffic from different points to one destination that is “productpage”

Destination Rule

DestinationRule is a Kubernetes resource that defines policies for routing traffic to a specific version of a service. It allows you to configure various aspects of how traffic is directed to a service, such as load-balancing, connection pool settings, TLS settings, and more.

A DestinationRule resource consists of two main components:

A set of rules that define how traffic should be routed to the service. This can include subsets of the service, load balancing policies, circuit breaker settings, and more.
A set of policies that define how traffic should be secured when communicating with the service. This can include mTLS (mutual TLS) settings, certificate authorities, and more.

Here’s an example of how a DestinationRule can be used in Istio:

Let’s say we have a service mesh consisting of several versions of a microservice called “product” running on Kubernetes pods. We want to configure the load balancing and connection pool settings for the “product” service. To achieve this, we can create a DestinationRule resource that defines the “product” service and sets the load balancing algorithm to round-robin and the maximum number of connections to 100.

Here’s an example YAML file for a DestinationRule:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: product
spec:
  host: product-service
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    connectionPool:
      http:
        http1MaxPendingRequests: 100
        maxRequestsPerConnection: 5

In this example, we define a DestinationRule resource called “product” that sets the load balancing algorithm for the “product-service” to round-robin using the simple property. We also set the maximum number of pending requests to 100 and the maximum number of requests per connection to 5.

This allows us to fine-tune the load balancing and connection pool settings for the “product” service, providing better control and management of traffic in the service mesh.

In summary, a DestinationRule in Istio is a Kubernetes resource that defines policies for routing traffic to a specific version of a service, allowing you to configure various aspects of how traffic is directed to a service, such as load-balancing, connection pool settings, TLS settings, and more.

TrafficPolicies

Traffic Policies are rules and configurations that control the behavior of traffic within the service mesh. They provide fine-grained control over how traffic is routed, load balanced, secured, and monitored in the service mesh.

Traffic policies can be defined for individual services or for groups of services, and they can be specified at various levels in the service mesh, such as virtual services, destination rules, gateways, and more. Some of the common traffic policies in Istio include:

Load balancing policies: These policies control how traffic is distributed among different instances of a service. For example, you can specify a round-robin load-balancing algorithm or a weighted load-balancing algorithm based on the performance and availability of each instance.
Connection pool settings: These policies control how many connections are allowed to a service at any given time, and how long they can stay open. This helps prevent connection overload and optimizes resource utilization.
**Traffic routing policies: **These policies control how traffic is routed to different versions of a service, based on various factors such as HTTP headers, cookies, or client IP addresses.
Security policies: These policies control how traffic is secured within the service mesh. This can include mutual TLS (mTLS) authentication, certificate management, and access control policies.
Traffic monitoring policies: These policies control how traffic is monitored and logged in the service mesh. This can include configuring telemetry data collection, setting up tracing and logging, and more.

By using traffic policies in Istio, you can fine-tune and optimize the behavior of traffic in your service mesh, ensuring better performance, reliability, and security of your microservices-based applications.

let's talk briefly about the Load Balancing Policies

Load Balancing Policies

load balancing policies are a key aspect of traffic management in Istio. They determine how traffic is distributed among different instances of a service within the service mesh, ensuring optimal utilization of resources and better performance and availability of microservices-based applications.

There are several load-balancing policies that can be used in Istio, depending on the specific needs of the application. Some of the common load-balancing policies include:

Round-robin: In this policy, traffic is evenly distributed among all instances of a service, one after the other, in a cyclic order. This ensures that each instance of the service receives an equal share of the traffic.
Weighted: In this policy, traffic is distributed based on the relative weights assigned to each instance of a service. Instances with higher weights receive more traffic than instances with lower weights, allowing you to control the distribution of traffic based on the performance and availability of each instance.
Locality-based: In this policy, traffic is distributed based on the geographic location of the client and the instances of the service. This helps optimize the routing of traffic and reduce latency by directing traffic to the closest instance of the service.
Least connections: In this policy, traffic is directed to the instance with the least number of active connections, ensuring that each instance of the service is not overloaded with too many connections.
Random: In this policy, traffic is randomly distributed among all instances of a service, allowing you to achieve a more uniform distribution of traffic.

You can specify the load-balancing policy for a service or a group of services in Istio by creating a DestinationRule resource and setting the appropriate load-balancing algorithm. This can be done using the simple property for simple round-robin or random load balancing, or the weighted property for weighted load balancing. Once the DestinationRule is created, Istio will automatically apply the load-balancing policy to the specified service or group of services.

How to Deploy ArgoCD in EKS cluster For Continuous Deployment

Ian Kiprotich — Mon, 13 Mar 2023 20:32:57 +0000

In this blog, we will be able to deploy ArgoCd in an EKS cluster and use it for Continuous Deployment process

Prerequisites

Have an EKS cluster running
Have your yaml configuration microservices in a git repository

What is ArgoCD?

Argo CD is a declarative continuous delivery tool for Kubernetes applications. It helps automate the deployment of applications to Kubernetes clusters by providing a GitOps approach to managing and deploying applications.

With Argo CD, you can define the desired state of your application in a Git repository and have it automatically synced with your Kubernetes cluster. This allows you to easily manage your application deployments across multiple environments, such as development, staging, and production.

Argo CD provides a web-based user interface as well as a command-line interface for managing your application deployments. It also integrates with other tools in the Kubernetes ecosystem, such as Helm charts, Kustomize, and Jsonnet.

Overall, Argo CD helps simplify the deployment process for Kubernetes applications and provides a reliable, scalable, and secure way to manage your deployments.

Demo overview steps

Install ArgoCD in K8s cluster
Configure ArgoCD with “Applicatin” CRD
Install the ArgoCD CLI.
Test our setup by updating Deployment.yaml files

Let's get started by first installing ArgoCd

First, create a namespace

kubectl create namespace argocd

Next, let's apply the yaml configuration files for ArgoCd

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Now we can view the pods created in the ArgoCD namespace.

Next we can acess the ArgoCD ui from the services deployed

kubectl port-forward svc/argocd-server 8080:443 -n argocd

The username is admin

The password is autogenerated and stored in the secret called argocd-initial-admin-secret in the argocd installation namespace.

kubectl get secret argocd-initial-admin-secret -n argo
cd -o yaml

Then you can decode the password

echo R215N3hvZ21LSEEwRGlIag== | base64 --decode

The ArgoCD is empty because we have to configure it thats gonna be the next step.

Let's write a configuration file for ArgoCD to connect it to the git repository where the configuration files are hosted

since we have to connect ArgoCD with our cluster we have to get the endpoint for our cluster

aws eks describe-cluster --name <cluster-name> --query "cluster.endpoint"

You can also view the endpoint using the following steps

To find the endpoint using the AWS Management Console:

Open the EKS console.
Select your cluster.
Under the “Configuration” tab, you should see the “Kubernetes endpoint” listed.

Next, we have to configure the CLI to our ArgoCD instance. the easiest way is to use the homebrew command

brew install argocd

After you install you can log in to the CLI using the command

argocd login 127.0.0.1:443

Fill in the Username and Password

This step registers a cluster’s credentials to Argo CD, and is only necessary when deploying to an external cluster. When deploying internally (to the same cluster that Argo CD is running in), https://kubernetes.default.svc should be used as the application’s K8s API server address.

First list all clusters contexts in your current kubeconfig:

kubectl config get-contexts -o name

Add your cluster

argocd cluster add arn:aws:eks:us-east-2:758659350150:cluster/Giovanni

The above command installs a ServiceAccount (argocd-manager), into the kube-system namespace of that kubectl context, and binds the service account to an admin-level ClusterRole. Argo CD uses this service account token to perform its management tasks (i.e. deploy/monitoring).

Next, we will configure the git repository https://gitlab.com/onai254/giovanni.git to deploy the application but before that, we need to set the current namespace to ArgoCD by running the following command:

kubectl config set-context --current --namespace=argocd

Next, we will create the application using the following command

argocd app create myapp --repo https://github.com/argoproj/argocd-example-apps.git --path . --revision master --dest-server https://11E5FA4C5A83DEA0033E57F09D23DFC5.gr7.us-east-2.eks.amazonaws.com --dest-namespace default

With this, our application is deployed automatically by ArgoCD and also it watches the repository for any changes.

You can also view more functionality of ArgoCD and view the different parts of the application.