DEV Community: Rakesh Jain

Prometheus and AlertManager step by step configuration with blackbox exporter on Ubuntu 18.04

Rakesh Jain — Thu, 20 Aug 2020 12:04:18 +0000

Prometheus is a monitoring tool designed for recording real-time metrics in a time-series database. It is an open-source software project, written in Go. The Prometheus metrics are collected using HTTP pulls, allowing for higher performance and scalability.

For complete tutorial please follow the below link ..

https://medium.com/@rakeshjain_17559/prometheus-and-alertmanager-step-by-step-configuration-with-blackbox-exporter-on-ubuntu-18-04-f2b76f52f1ad

Real Life Cron Examples

Rakesh Jain — Thu, 20 Aug 2020 12:01:11 +0000

This tutorial is about How to write a crontab schedule expression for diff scenarios ..

Go through the following tutorial for more details ..
https://medium.com/@rakeshjain_17559/cron-examples-4233b873e5f9

Kubernetes hands on series - Upgrading Production Kubernetes cluster created with kubeadm

Rakesh Jain — Thu, 20 Aug 2020 11:58:34 +0000

This tutorial will explain how we can upgrade a Kubernetes cluster created with kubeadm from version 1.17.x to version 1.18.x or from version 1.18.x to 1.18.y (where y > x).``

Please go through the link below for full tutorial ..

https://medium.com/@rakeshjain_17559/kubernetes-hands-on-series-upgrading-production-kubernetes-cluster-created-with-kubeadm-6340821bc92

The Best and Safest way to clean up boot partition-Ubuntu 14.04/16.04/18.04 LTS

Rakesh Jain — Tue, 11 Aug 2020 09:11:04 +0000

When you have apt working and boot partition has some space left :

Check the current kernel version
$ uname -r 3.19.0-64-generic

Remove the OLD kernels
List the old kernel (it wont include the current kernel)

$ sudo dpkg --list 'linux-image*'|awk '{ if ($1=="ii") print $2}'|grep -v `uname -r`

You will get the list of images something like below:

linux-image-3.19.0-25-generic linux-image-3.19.0-56-generic linux-image-3.19.0-58-generic linux-image-3.19.0-59-generic linux-image-3.19.0-61-generic linux-image-3.19.0-65-generic linux-image-extra-3.19.0-25-generic linux-image-extra-3.19.0-56-generic linux-image-extra-3.19.0-58-generic linux-image-extra-3.19.0-59-generic linux-image-extra-3.19.0-61-generic

Now its time to remove old kernel one by one as

$sudo apt-get purge linux-image-3.19.0-25-generic $ sudo apt-get purge linux-image-3.19.0-56-generic $ sudo apt-get purge linux-image-3.19.0-58-generic $ sudo apt-get purge linux-image-3.19.0-59-generic $ sudo apt-get purge linux-image-3.19.0-61-generic $ sudo apt-get purge linux-image-3.19.0-65-generic

When you’re done removing the older kernels, you can run this to remove ever packages you won’t need anymore:
$sudo apt-get autoremove

And finally you can run this to update grub kernel list:
$sudo update-grub

When boot partition has zero space left and apt not working:

Get the list of kernel images (it wont include the current kernel)
Get the list of kernel images and determine what you can do without.

This command will show installed kernels except the currently running one
$sudo dpkg --list 'linux-image*'|awk '{ if ($1=="ii") print $2}'|grep -vuname -r

You will get the list of images something like below:

linux-image-3.19.0-25-generic
linux-image-3.19.0-56-generic
linux-image-3.19.0-58-generic
linux-image-3.19.0-59-generic
linux-image-3.19.0-61-generic
linux-image-3.19.0-65-generic
linux-image-extra-3.19.0-25-generic
linux-image-extra-3.19.0-56-generic
linux-image-extra-3.19.0-58-generic
linux-image-extra-3.19.0-59-generic
linux-image-extra-3.19.0-61-generic

Prepare for old kernel delete
Craft a command to delete all files in /boot for kernels that don’t matter to you using brace expansion to keep you sane. Remember to exclude the current and two newest kernel images. From above Example, it’s
$sudo rm -rf /boot/*-3.19.0-{25,56,58,59,61,65}-*

Clean up apt for old partial installs
sudo apt-get -f install

Run Autoremove to clear old images
Finally, autoremove to clear out the old kernel image packages that have been orphaned by the manual boot clean.
$sudo apt-get autoremove

Update Grub
$sudo update-grub

Now you can update and install packages
sudo apt-get update sudo apt-get install lshw -y

For an all in one try this:
sudo dpkg --list 'linux-image*'|awk '{ if ($1=="ii") print $2}'|grep -v uname -r | while read -r line; do sudo apt-get -y purge $line;done;sudo apt-get autoremove; sudo update-grub; apt-get update; apt-get install lshw -y

Ref:https://help.ubuntu.com/community/RemoveOldKernels#Configure_Unattended_Upgrades_to_Remove_Unneeded_Kernels_Automatically

Hope you like the tutorial. Please like & share and let me know your feedback in the response section.
Happy Learning!

Kubernetes kubectl Tips and Tricks you must know

Rakesh Jain — Sun, 09 Aug 2020 04:24:03 +0000

As we know that kubectl is a powerful tool which we use every day. It has broad functionality.

Which kubectl tip or tricks, you found out and said wish I knew it before.

I will start with my experience wish I would have known earlier.

Before we begin, familiarize yourself with the following command parameters:

--dry-run : By default, as soon as this is executed, the resource will be created.

--dry-run=client : This will NOT create the resource. Instead, kubernetes will tell you if it is possible to summon the object.

-o yaml : This will output the resource definition file in a YAML format on the screen.

Use the last two in combination to conveniently generate a resource definition file. Later, you can modify it as required, instead of writing the file from scratch.

For example, the following generates a POD manifest file:
kubectl run nginx --image=nginx --dry-run=client -o yaml

Let’s look at more imperative commands for generating PODs.

Create an NGINX Pod:

$kubectl run nginx --image=nginx

Generate a POD manifest YAML file:

$kubectl run nginx --image=nginx --dry-run=client -o yaml

As a reminder, the -o yaml parameter tells kubectl to output a yaml file, while — dry-run instructs kubernetes to NOT create the POD.

Let’s now learn about deployment related imperative commands.

Deployment Commands
To create a deployment from the command line:

$kubectl create deployment --image=nginx nginx

Generate a deployment YAML file template:

$kubectl create deployment --image=nginx nginx --dry-run=client -o yaml

kubectl create deployment does not have a — replicas option.

Having said that, if you wish to scale the deployment, you would have to first create it and then scale it using the kubectl scale command.

The following command will create a deployment YAML definition file with the name “nginx-deployment.yaml”:

$kubectl create deployment --image=nginx nginx --dry-run=client -o yaml > nginx-deployment.yaml

Again, the -o parameter tells kubectl to output a yaml file, while the > nginx-deployment.yaml argument provides kubectl with the desired output file name.
You can then modify the YAML file to your needs.

Services Commands
The following command creates a service named “redis-service” of type ClusterIP, with the purpose of exposing the redis pod on port 6379.

$kubectl expose pod redis --port=6379 --name redis-service --dry-run=client -o yaml

Note: This will automatically use the POD’s labels as selectors
Alternative:

$kubectl create service clusterip redis --tcp=6379:6379 --dry-run=client -o yaml

The above command will not use the POD’s labels as selectors, instead it will assume selectors as app=redis. You cannot pass in selectors as an option. Alternatively, generate the file and modify the selectors before creating the service.

The following command creates a service named nginx of type NodePort to expose the nginx pod’s port 80 on port 30080 on the nodes:

$kubectl expose pod nginx --port=80 --name nginx-service --type=NodePort --dry-run=client -o yaml

Note: This will automatically use the pod’s labels as selectors, but you cannot specify the node port. You have to generate a definition file and then add the node port in manually before creating the service with the pod.
Or

$kubectl create service nodeport nginx --tcp=80:80 --node-port=30080 --dry-run=client -o yaml

This will not use the pod’s labels as selectors.
The last couple commands have their own challenges: One cannot accept a selector, the other cannot accept a node port. In my opinion, I recommend going with the kubectl expose command.

If you need to specify a node port, generate a definition file using the same command and manually input the nodeport before creating the service.

Hope you like the tutorial. Please like and share:) if you like it and let me know your feedback in the response section.
Happy Learning!

Kubernetes hands on series: Pods

Rakesh Jain — Sat, 08 Aug 2020 04:47:16 +0000

Pods are the smallest deployable units of computing that you can create and manage in Kubernetes.
A pod is a collection of containers and its storage inside a node of a Kubernetes cluster.

Single Container Pod -->
They can be simply created with the kubectl run command, where you have a defined image on the Docker registry which we will pull while creating a pod.

Method 1: Creating Pod from command line with kubectl -->

$kubectl run "name of pod" --image="name of the image from registry"

$kubectl run myapache --image httpd --port=80

$kubectl get pods -w
NAME READY STATUS RESTARTS AGE
myapache 0/1 ContainerCreating 0 2s
myapache 1/1 Running 0 13s

$kubectl describe pods myapache
Name: myapache
Namespace: default
Priority: 0
Node: kworker-rj2/172.42.42.202
Start Time: Fri, 07 Aug 2020 08:06:31 +0000
Labels: run=myapache
Annotations: cni.projectcalico.org/podIP: 172.16.213.204/32
cni.projectcalico.org/podIPs: 172.16.213.204/32
Status: Running
IP: 172.16.213.204
IPs:
IP: 172.16.213.204
Containers:
myapache:
Container ID: docker://f2d794c10236e2470612aa8530158ec8ad51d487bed90c415f7c94768d343f81
Image: httpd
Image ID: docker-pullable://httpd@sha256:3cbdff4bc16681541885ccf1524a532afa28d2a6578ab7c2d5154a7abc182379
Port: 80/TCP
Host Port: 0/TCP
State: Running
Started: Fri, 07 Aug 2020 08:06:43 +0000
Ready: True
Restart Count: 0
Environment:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-sdv2m (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-sdv2m:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-sdv2m
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message

Normal Scheduled 23s default-scheduler Successfully assigned
default/myapache to kworker-rj2
Normal Pulling 21s kubelet, kworker-rj2 Pulling image "httpd"
Normal Pulled 12s kubelet, kworker-rj2 Successfully pulled image "httpd"
Normal Created 11s kubelet, kworker-rj2 Created container myapache
Normal Started 10s kubelet, kworker-rj2 Started container myapache

This can also be done by creating the yaml file and then running the kubectl create command. But before lets understand what a Kubernetes resource yaml file means.

Method 2: Creating Pod from a yaml file -->
Understanding resource yaml file -->

In the .yaml file for the Kubernetes object you want to create, you'll need to set values for the following fields:

apiVersion - Which version of the Kubernetes API you're using to create this object

kind - What kind of object you want to create

metadata - Data that helps uniquely identify the object, including a name string, UID, and optional namespace and Labels

spec - What state you desire for the object

Once the above yaml file is created, we will save the file with the name of apache.yml and run the create command to run the document.
$kubectl create –f apache.yml

It will create a pod with the name of apache. We can use the describe command along with kubectl to describe the pod as we have done that above.

How to go inside a container within a pod -->
Here we will learn how can we go inside a running container of a Pod.

We have deployed an apache container and its IP is 172.16.213.204.
Lets curl the container IP first and see the output.
$curl 172.16.213.204:80
It works!

Now go inside this container and change the index file content and exit from the container. And test again with curl.

$kubectl -it exec myapache -- bash
root@myapache:/usr/local/apache2# echo “Hello from myapache port” > htdocs/index.html
root@myapache:/usr/local/apache2# exit
curl 172.16.213.204:80
Hello from myapache port

Multi Container Pod -->
Multi container pods are created using yaml file with the definition of the containers.

Pod Phases -->
Here are the possible values for phase:
Pending: The Pod has been accepted by the Kubernetes cluster, but one or more of the containers has not been set up and made ready to run. This includes time a Pod spends waiting to be scheduled as well as the time spent downloading container images over the network.

Running: The Pod has been bound to a node, and all of the containers have been created. At least one container is still running, or is in the process of starting or restarting.

Succeeded: All containers in the Pod have terminated in success, and will not be restarted.

Failed: All containers in the Pod have terminated, and at least one container has terminated in failure. That is, the container either exited with non-zero status or was terminated by the system.

Unknown: For some reason the state of the Pod could not be obtained. This phase typically occurs due to an error in communicating with the node where the Pod should be running.
If a node dies or is disconnected from the rest of the cluster, Kubernetes applies a policy for setting the phase of all Pods on the lost node to Failed.

Container restart policy -->
The spec of a Pod has a restartPolicy field.

Possible values -> Always, OnFailure, and Never. The default value is Always.

The restartPolicy applies to all containers in the Pod. restartPolicy only refers to restarts of the containers by the kubelet on the same node. After containers in a Pod exit, the kubelet restarts them with an exponential back-off delay (10s, 20s, 40s, …), that is capped at five minutes. Once a container has executed with no problems for 10 minutes without any problems, the kubelet resets the restart backoff timer for that container.

Container imagePullPolicy policy -->
imagePullPolicy: IfNotPresent: the image is pulled only if it is not already present locally.

imagePullPolicy: Always: every time the kubelet launches a container, the kubelet queries the container image registry to resolve the name to an image digest.

imagePullPolicy: Never: the image is assumed to exist locally. No attempt is made to pull the image.

Forced Pod termination -->
You must specify an additional flag --force along with --grace-period=0 in order to perform force deletions.

$kubectl delete pods myapache --force --grace-period=0
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "myapache" force deleted

Hope you like the tutorial. Please like and share:) if you like it and let me know your feedback in the response section.
Happy Learning!

Production Ready Kubernetes Cluster Deployment on Ubuntu

Rakesh Jain — Mon, 03 Aug 2020 06:47:21 +0000

Min HW requirement -
Master — 2GB RAM, 2vCPU’s, min 1GBPS nw
Nodes — 1GB RAM, 2vCPU’s, min 1GBPS nw

SW requirement -
Docker or any container mgmt systems should be installed.
kubelet service should be running on each node.

We have 3 Ubuntu 18.04 systems running with following configuration -

Kubernetes Master Node ->
172.42.42.200 kmaster-rj.example.com/kmaster-rj
4 GB RAM, 2 vCPU’s, 64 GB HDD

Kubernetes Worker Nodes ->
172.42.42.201 kworker-rj1.example.com/kworker-rj1
2 GB RAM, 2 vCPU’s, 64 GB HDD
172.42.42.202 kworker-rj2.example.com/kworker-rj2
2 GB RAM, 2 vCPU’s, 64 GB HDD
Note: You can use any number of worker nodes with a single master node.

Installation ->
Install the following software packages on all 3 servers;

Install and enable Docker
apt-get update
apt install docker.io
systemctl enable docker

Add the Kubernetes gpg key
apt install curl && curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add

Add the Xenial Kubernetes repository
apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"

Install kubeadm
apt install kubeadm
kubeadm version

Kubernetes Deployment -->

Disable swap -
If you run nodes with (traditional to-disk) swap, you lose a lot of the isolation properties that make sharing machines viable. You have no predictability around performance or latency or IO. So its recommended to off the swap.
swapoff -a && sed -i ‘/ swap / s/^/#/’ /etc/fstab
Kubernetes Cluster Initialization (On Master)->

Method 1 of deployment (from command line)->
Initialize the cluster, also by passing a flag that is later needed for the networking plugin (CNI).

kubeadm init --apiserver-advertise-address $(hostname -i) --pod-network-cidr=172.16.213.0/24

Note: If 172.16.213.0/24 is already in use within your network you must select a different pod network CIDR, replacing 172.16.213.0/24 in the above command as well as in any manifests applied below.

Method 2 of deployment (using config file)->
You can initialize the cluster by passing the kubeadm-config.yaml file with specific values as given below
https://github.com/TroubleshooteRJ/kubernetes-demo/blob/master/kubeadm-config.yaml

Here you can change the kubernetesVersion, external IP addresses and domain names you would use to connect to the Kubernetes API server, Pod CIDR and Service CIDR based on your networking requirements etc.

You can init the master with the following command.

curl https://github.com/TroubleshooteRJ/kubernetes-demo/blob/master/kubeadm-config.yaml -o kubeadm-config.yaml# update the file as necessary and then run belowkubeadm init --config kubeadm-config.yaml

This will output a join command. Save this command somewhere as this will later be used to connect worker nodes to the master.

Configure kubectl as it lets you connect to the cluster from the master (The command for this step is given in the output of the previous command).

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Type the below command to check if kubectl is working (The status will be ‘NotReady’ as we haven’t set up our networking yet. )
kubectl get nodes

Now let’s deploy a Pod Network through the Master Node.
Here we will use Calico as the network of choice.

Install the Calico Networking Plugin in the Cluster (This will handle networking between different pods and nodes).This can be done by applying a yaml file that describes the objects needed to create in the cluster.

kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

Wait until each pod has the STATUS of Running.
Confirm that all of the pods are running with the following command.

kubectl get pods --all-namespaces -w

Now check if kubectl is working (The status should be ‘Ready’ now ).
kubectl get nodes -o wide

Now let’s add the Worker nodes to the Network in order to form a Cluster.
The kubeadm init command that you ran previously on the master should output a kubeadm join command containing a token and hash. If you have copied that command from the master and saved it somewhere, run it on both worker nodes with sudo to connect them to the master.

kubeadm join 172.42.42.200:6443 --token bl4tmf.p5vld9mlo3zbqy66
--discovery-token-ca-cert-hash sha256:2aba01741a419a17d257b60f4aa2ae7b13185056a24986669347bfbcf9f6ecbe

Now check if the 2 worker nodes are connected (on master).
sudo kubectl get nodes

Make sure all 3 of your nodes are listed with the above command and have a status of ‘Ready’.

With the following command you can see some of back-end system pods that are fully up and running.

Now we have a Kubernetes cluster fully up and running on Ubuntu!

Hope you like the tutorial. Let me know your feedback in the response section.

Happy Learning!