DEV Community: Subodh Bagde

Kubernetes in Action: Architecture, Features and Free Resources

Subodh Bagde — Mon, 15 Jul 2024 15:49:19 +0000

Introduction

In today’s fast-paced tech world, applications need to be developed, deployed, and scaled rapidly. Kubernetes, an open-source container orchestration platform, has become a cornerstone in achieving this efficiency. This blog post will explore what Kubernetes is, its benefits, and provide a simplified explanation of its architecture.

Kubernetes Architecture

1. Cluster:

A Kubernetes cluster is a set of machines, called nodes, that run containerized applications. You have one master node and multiple worker nodes.

2. Master Node:

Control Plane: The master node manages the Kubernetes cluster. It consists of several key components:
API Server: This is the front end of the Kubernetes control plane. All communication goes through this server.
etcd: This is a key-value store used to store all the cluster data.
Controller Manager: This ensures the desired state of the cluster is maintained, like managing replication and handling node failures.
Scheduler: This assigns workloads (containers) to the worker nodes based on resource availability and other constraints.

3. Worker Nodes:

These nodes run the applications. Each worker node has essential components to manage containerized applications:
Kubelet: This is the agent that runs on each worker node. It communicates with the master node and ensures that the containers are running as expected.
Kube-proxy: This is a network proxy that maintains network rules on nodes, allowing communication to your pods from inside or outside the cluster.
Container Runtime: This is the software responsible for running containers. Examples include Docker, containerd, and CRI-O.

4. Pods:

A pod is the smallest and simplest Kubernetes object. It represents a single instance of a running process in your cluster. A pod can contain one or more containers.

5. Controllers:

Controllers manage the state of your applications. Some common types are:
ReplicaSet: Ensures a specified number of pod replicas are running.
Deployment: Manages ReplicaSets and provides declarative updates to applications.
StatefulSet: Manages stateful applications (apps that require persistent storage).
DaemonSet: Ensures that all (or some) nodes run a copy of a pod.

6. Services:

Services in Kubernetes provide a way to expose an application running on a set of pods as a network service. Services can be of different types:
ClusterIP: Exposes the service on a cluster-internal IP.
NodePort: Exposes the service on each node’s IP at a static port.
LoadBalancer: Exposes the service externally using a cloud provider’s load balancer.

Simple Analogy to understand K8s Architecture

Imagine you are running a large restaurant kitchen:

Master Node is the head chef who manages the kitchen and decides who does what.
Worker Nodes are the individual chefs who actually cook the dishes.
API Server is like the order window where orders are placed and communicated to the head chef.
etcd is the recipe book where all recipes (data) are stored.
Controller Manager is like the head chef making sure all the stations are staffed and functioning correctly.
Scheduler is like the head chef assigning specific tasks to each chef based on their skills and the current workload.
Kubelet is like each chef ensuring they follow the orders given to them.
Kube-proxy is like the kitchen door staff managing the flow of dishes in and out.
Pods are the individual dishes being prepared, each potentially consisting of multiple ingredients (containers).
Services are the wait staff making sure the dishes get to the correct tables (customers).

Real-World Use Cases

1. Microservices Architecture

Example: Netflix utilizes Kubernetes to manage its vast microservices architecture, ensuring seamless streaming and content delivery.

2. CI/CD Pipelines

Example: Companies like Shopify use Kubernetes to automate their CI/CD pipelines, leading to faster release cycles and improved code quality.

3. Big Data Processing

Example: Spotify uses Kubernetes to run its big data processing tasks, enabling them to handle large volumes of data for music recommendations and user analytics.

4. Hybrid and Multi-Cloud Deployments

Example: CERN, the European Organization for Nuclear Research, uses Kubernetes for its hybrid cloud strategy, combining on-premises infrastructure with public clouds to handle their extensive computational needs.

Free Kubernetes Resources

1. Kubernetes Official Documentation

Link: Kubernetes Documentation

2. Minikube

Link: Minikube

3. KubeAcademy by VMware

Link: KubeAcademy

4. KodeKloud

Link: KodeKloud

Conclusion

By understanding the basics of Kubernetes and its architecture, you can begin to harness its power to build scalable, resilient, and efficient applications. Whether you are managing a small application or a large-scale enterprise system, Kubernetes provides the tools you need to succeed.

Feel free to reach out if you have any questions. Happy containerizing!

A Guide to Installing Jenkins, Configuring it and expose to outside world.

Subodh Bagde — Sat, 15 Jun 2024 12:22:09 +0000

What is Jenkins?

Jenkins is an open-source Java automation server used for software development lifecycle automation of repetitive operations. Being able to integrate with nearly all tools in the CI/CD pipeline with its many plugin support makes it a vital tool for developers and DevOps engineers.

AWS EC2 Instance

Go to AWS Console
Instances(running)
Launch instances

Install Jenkins

Pre-Requisites:

Java (JDK)

Run the below commands to install Java and Jenkins

Install Java

sudo apt update sudo apt install openjdk-11-jre

Verify Java is Installed

java --version

Now, you can proceed with installing Jenkins

curl -fsSL https://pkg.jenkins.io/debian/jenkins.io-2023.key | sudo tee \
  /usr/share/keyrings/jenkins-keyring.asc > /dev/null
echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \
  https://pkg.jenkins.io/debian binary/ | sudo tee \
  /etc/apt/sources.list.d/jenkins.list > /dev/null
sudo apt-get update
sudo apt-get install jenkins

By default, Jenkins will not be accessible to the external world due to the inbound traffic restriction by AWS. Open port 8080 in the inbound traffic rules as show below.

Add inbound traffic rules as shown in the image:

Login to Jenkins using the below URL:

http://:8080 [You can get the ec2-instance-public-ip-address from your AWS EC2 console page]

After you login to Jenkins,
- Run the command to copy the Jenkins Admin Password - sudo cat /var/lib/jenkins/secrets/initialAdminPassword
- Enter the Administrator password

Click on Install suggested plugins

Wait for the Jenkins to Install suggested plugins.

Create First Admin User or Skip the step.

Jenkins Installation is Successful. You can now starting using the Jenkins

Install the Docker Pipeline plugin in Jenkins:

Log in to Jenkins.
Go to Manage Jenkins > Manage Plugins.
In the Available tab, search for "Docker Pipeline".
Select the plugin and click the Install button.
Restart Jenkins after the plugin is installed.

Wait for the Jenkins to be restarted.

Docker Slave Configuration

Run the below command to Install Docker

sudo apt update
sudo apt install docker.io

Grant Jenkins user and Ubuntu user permission to docker daemon.

sudo su - 
usermod -aG docker jenkins
usermod -aG docker ubuntu
systemctl restart docker

You can run the below command to check if docker is up and running:

docker run hello-world

Once you are done with the above steps, it is better to restart Jenkins.

http://<ec2-instance-public-ip>:8080/restart

The docker agent configuration is now successful.

You can also refer my GitHub Repo:

https://github.com/SubodhBagde/Jenkins-Demo-Pipeline/tree/main

A simple Jenkins pipeline to verify if the docker agent configuration is working as expected.

Follow the steps to implement your first pipeline in Jenkins.

Click on the New Item tab on the left side of your Jenkins UI. Give a name and select Pipeline option and click on OK.

Now configure the pipeline as shown below in the image.

Finally click on Build Now tab on the left to start the process.

Here's the output, simply cick on the console output to view it.

Multi Stage Multi Agent

Set up a multi stage jenkins pipeline where each stage is run on a unique agent. This is a very useful approach when you have multi language application
or application that has conflicting dependencies.

Steps involved in it are pretty much similar to the previous one.

You have to just replace the Script path my this folder name, as shown below.

Run the following docker command to check whether the container has been build or not. You can see that once the execution is done the Jenkins application automatically deletes the container.

docker ps

The Multi Stage Multi Agent process is successful.

AWS CloudWatch: The Gatekeeper for Your AWS Environment

Subodh Bagde — Sun, 09 Jun 2024 16:03:08 +0000

Introduction

Maintaining your infrastructure’s operational health and peak performance is crucial in the world of cloud computing. AWS CloudWatch provides robust monitoring, alerting, reporting, and logging features, acting as a gatekeeper for your AWS environment. This article explores AWS CloudWatch, its benefits, and a real-world example of setting up an alarm to track CPU usage.

What is AWS CloudWatch?

Amazon Web Services (AWS) offers a flexible monitoring and management solution called AWS CloudWatch. You may use it to trigger alarms, monitor and evaluate metrics, and get real-time insights about your AWS apps and resources. By serving as a central repository for all monitoring data, CloudWatch enables you to keep your infrastructure operating efficiently and in good condition.

Advantages of AWS CloudWatch

AWS CloudWatch is a vital tool for managing AWS environments because of its many important benefits, which include:

Monitoring: Continuously observes your AWS resources and applications to ensure they are functioning correctly.
Real-Time Metrics: Provides up-to-date data on resource utilization, enabling informed decision-making.
Alarms: Automatically notifies you when specific metrics exceed
predefined thresholds, allowing for timely intervention.
Log Insights: Centralizes and manages logs, simplifying troubleshooting and application behavior monitoring.
Custom Metrics: Tracks specific metrics relevant to your application or business needs.
Cost Optimization: Monitors resource usage and sets up billing alarms to help manage and optimize AWS costs.

Creating an Alarm in CloudWatch

Let’s explore a practical use case where we create an alarm in CloudWatch to notify us via email when the CPU utilization of an instance spikes to 50% or above.

Log in to the AWS Management Console and navigate to the CloudWatch service.
Click on “Alarms” in the left-hand menu and select “Create alarm.”
Choose “Select metric” and pick “EC2” from the metric source.
Under “Namespace,” select “AWS/EC2.”
For “Metric Name,” choose “CPUUtilization.”
Select the specific EC2 instance you want to monitor from the “Instance ID” dropdown. So I have created a specific EC2 instance for this purpose called “cloud-watch-demo”.
Scaling: Scales with your AWS environment, handling millions of events per minute.

Select your EC2 instance and then click on monitoring tab to see Graphical analysis of your instance based on various parameters such as CPU Utilization.

Under “Statistic,” select “Average” to monitor the average CPU utilization over a period.
In the “Period” field, enter the desired time window for averaging CPU usage (e.g., 5 minutes).
For “Comparison operator,” choose “Greater than (>)”
In the “Threshold” value, enter “50” to trigger the alarm when CPU utilization exceeds 50%.
Leave the “Evaluation periods” set to “1” for the alarm to trigger if the average CPU utilization is above 50% for the chosen time period.
Under “Alarm name,” enter a descriptive name for your alarm (e.g., “High CPU Utilization on [Instance ID]”).
Now you need to configure the notification for the alarm. Click on “Add action” and choose “SNS topic.”
If you haven’t already, create a new SNS topic or select an existing one where you want to receive notifications.
Click “Next” and review the alarm configuration.
Finally, click “Create alarm” to set up your CloudWatch alarm.
Once the alarm is created you can click on it to see a detailed view.

Click on the Metrics tab in the AWS CloudWatch and then search for metric name “CPUUtilization”. After this select your EC2 Instance to see the graph. (In my case, I’ve selected “cloud-watch-demo”)

Now to check the working of alarm I’ve used a python program that generates CPU Spikes which will in turn affect the CPU Utilization of our instance. Credits for the python script:- Abhishek Veeramalla

import time

def simulate_cpu_spike(duration=30, cpu_percent=80):
    print(f"Simulating CPU spike at {cpu_percent}%...")
    start_time = time.time()

    # Calculate the number of iterations needed to achieve the desired CPU utilization
    target_percent = cpu_percent / 100
    total_iterations = int(target_percent * 5_000_000)  # Adjust the number as needed

    # Perform simple arithmetic operations to spike CPU utilization
    for _ in range(total_iterations):
        result = 0
        for i in range(1, 1001):
            result += i

    # Wait for the rest of the time interval
    elapsed_time = time.time() - start_time
    remaining_time = max(0, duration - elapsed_time)
    time.sleep(remaining_time)

    print("CPU spike simulation completed.")

if __name__ == '__main__':
    # Simulate a CPU spike for 30 seconds with 80% CPU utilization
    simulate_cpu_spike(duration=30, cpu_percent=80)

Run this python script on your EC2 Instance. After this you will have to wait for 2–5 minutes then check your email which you have provided for the SNS topic. You will see the results on your AWS Cloud watch alarm dashboard.

Conclusion

AWS CloudWatch is a crucial tool for monitoring and managing your AWS resources and applications. It offers real-time insights, automated alarms, centralized logging, and cost management features, enabling you to maintain the health and performance of your infrastructure. By setting up alarms, such as the one for monitoring CPU utilization, you can proactively address issues and ensure your applications run smoothly.

Thank you for reading, and I hope you found this blog post helpful in your AWS journey!