DEV Community: Unni P

Prometheus - HTTPS & Authentication - Part 4

Unni P — Tue, 22 Aug 2023 15:31:52 +0000

In this article, we will look how we can configure HTTPS and Authentication on both Prometheus and Node Exporter

Prerequisites

In my previous article, we looked at how we can set up Prometheus and Node Exporter as systemd services on an Ubuntu instance.

Prometheus - Installation on Amazon EC2 (Ubuntu) - Part 3

In this article, we will look how to install and configure Prometheus and Node Exporter on Amazon EC2 Ubuntu instance

iamunnip.hashnode.dev

But in the above setup, we were accessing the Prometheus expression browser and Node Exporter metrics endpoints via HTTP and there was no authentication enabled.

We are going to address these issues in this article.

HTTPS

Node Exporter

Create a new directory for storing the Node Exporter configuration file and change its ownership to "node_exporter" user

$ sudo mkdir -p /etc/node_exporter

$ sudo chown node_exporter:node_exporter /etc/node_exporter

Create a configuration file for Node Exporter and change its ownership to "node_exporter" user

$ sudo touch /etc/node_exporter/node_exporter.yml

$ sudo chown node_exporter:node_exporter /etc/node_exporter/node_exporter.yml

Generate a certificate and key using OpenSSL

$ sudo openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 -keyout prom.key -out prom.crt -subj "/C=US/ST=California/L=Oakland/O=MyOrg/CN=localhost" -addext "subjectAltName = DNS:localhost"

$ ls
prom.crt  prom.key

Copy the certificate and key files to the Node Exporter configuration directory and change their ownership to "node_exporter" user

$ sudo cp prom.* /etc/node_exporter

$ sudo chown node_exporter:node_exporter prom.crt

$ sudo chown node_exporter:node_exporter prom.key

Add the tls_server_config details to the configuration file

$ sudo vi /etc/node_exporter/node_exporter.yml

tls_server_config:
  cert_file: prom.crt
  key_file: prom.key

Update the systemd unit file of node_exporter service to include the above configuration file

$ sudo vi /etc/node_exporter/node_exporter.yml

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter \
        --web.config.file /etc/node_exporter/node_exporter.yml

[Install]
WantedBy=multi-user.target

Restart the node_exporter service and verify its status

$ sudo systemctl restart node_exporter

$ sudo systemctl status node_exporter

The metrics endpoint is now accessible via HTTPS

Open Prometheus expression browser and navigate to Status -> Targets, we can see our node_exporter target is showing down

Prometheus

Copy the certificate and key files to the Prometheus configuration directory and change its ownership to "prometheus" user

$ sudo cp prom.* /etc/prometheus

$ sudo chown prometheus:prometheus /etc/prometheus/prom.crt

$ sudo chown prometheus:prometheus /etc/prometheus/prom.key

Update the Prometheus configuration file to include scheme and tls_config for the "node_exporter" job

$ sudo vi /etc/prometheus/prometheus.yml

global:
  scrape_interval: 15s
  scrape_timeout: 10s

scrape_configs:
  - job_name: "node_exporter"
    scheme: https
    tls_config:
      ca_file: prom.crt
      insecure_skip_verify: true
    static_configs:
      - targets: ["172.31.81.113:9100"]

Validate the configuration file using the promtool

$ promtool check config /etc/prometheus/prometheus.yml
Checking /etc/prometheus/prometheus.yml
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

Restart the prometheus service to take effect the new configuration changes

$ sudo systemctl restart prometheus

$ sudo systemctl status prometheus

Open Prometheus expression browser and navigate to Status -> Targets, we can see our node_exporter target is showing up

Now we have enabled secure communication between the Prometheus server and Node Exporter but still our Prometheus expression browser is using an HTTP connection

Create a new configuration file for configuring HTTPS connection and change its ownership to "prometheus" user

$ sudo touch /etc/prometheus/webconfig.yml

$ sudo chown prometheus:prometheus /etc/prometheus/webconfig.yml

Add the tls_server_config details to the newly created configuration file

$ sudo vi /etc/prometheus/webconfig.yml

tls_server_config:
  cert_file: prom.crt
  key_file: prom.key

Update the systemd unit file of prometheus service to include the above configuration file

$ sudo vi /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus \
    --web.console.templates /etc/prometheus/consoles \
    --web.console.libraries /etc/prometheus/console_libraries \
    --web.config.file /etc/prometheus/webconfig.yml

[Install]
WantedBy=multi-user.target

Now we can access the Prometheus expression browser using HTTPS

Authentication

Install the apache2 utils package to generate a password

$ sudo apt update

$ sudo apt install apache2-utils

Generate the password using the htpasswd tool

$ htpasswd -nBC 16 admin | tr -d ':\n'
New password:
Re-type new password:
admin$2y$16$fMgRIvex1Rn67dHErc.Ft.CSI3ng5b457FOe9JIZkMB7k7p3PfS8O

Node Exporter

Update the Node Exporter configuration file to include basic authentication

$ sudo vi /etc/node_exporter/node_exporter.yml

tls_server_config:
  cert_file: prom.crt
  key_file: prom.key
basic_auth_users:
  admin: $2y$16$fMgRIvex1Rn67dHErc.Ft.CSI3ng5b457FOe9JIZkMB7k7p3PfS8O

Restart the node_exporter service and verify its status

$ sudo systemctl restart node_exporter

$ sudo systemctl status node_exporter

Now access the Node Exporter metrics endpoint and it will show a login prompt

Open Prometheus expression browser and navigate to Status -> Targets, we can see our node_exporter target is showing down

Prometheus

Update the Prometheus configuration file to include basic authentication for the "node_exporter" job

$ sudo vi /etc/prometheus/prometheus.yml

global:
  scrape_interval: 15s
  scrape_timeout: 10s

scrape_configs:
  - job_name: "node_exporter"
    scheme: https
    tls_config:
      ca_file: prom.crt
      insecure_skip_verify: true
      basic_auth:
        username: admin
        password: Password!
    static_configs:
      - targets: ["172.31.81.113:9100"]

Validate the configuration file using the promtool

$ promtool check config /etc/prometheus/prometheus.yml
Checking /etc/prometheus/prometheus.yml
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

Restart the prometheus service to effect new configuration changes

$ sudo systemctl restart prometheus

$ sudo systemctl status prometheus

Open the Prometheus expression browser and navigate to Status -> Targets, we can see our node_exporter target is showing up

Now we have enabled basic authentication between the Prometheus server and Node Exporter but we need to enable authentication for the Prometheus server

Update the below configuration file to include basic authentication on the Prometheus server

$ sudo vi /etc/prometheus/webconfig.yml

tls_server_config:
  cert_file: prom.crt
  key_file: prom.key

basic_auth_users:
  admin: $2y$16$fMgRIvex1Rn67dHErc.Ft.CSI3ng5b457FOe9JIZkMB7k7p3PfS8O

Restart the prometheus service for the new changes and check its status

$ sudo systemctl restart prometheus

$ sudo systemctl status prometheus

Now access the Prometheus expression browser and it will show a login prompt

That's all for now

Reference

https://prometheus.io/docs/prometheus/latest/configuration/https/

https://kodekloud.com/courses/prometheus-certified-associate-pca/

Prometheus - Installation on Amazon EC2 (Ubuntu) - Part 3

Unni P — Mon, 21 Aug 2023 16:48:56 +0000

In my previous article, we looked at how we can quickly set up Prometheus and Node Exporter.

Prometheus - Quick Setup on Amazon EC2 (Ubuntu) - Part 2

In this article, we will look at how we can quickly setup Prometheus and Node Exporter on a Ubuntu instance on Amazon EC2

iamunnip.hashnode.dev

But in that setup, Prometheus and Node Exporter processes are running in the foreground and it won't start the process after a reboot of the VM.

We are going to fix that issue in this article.

Prerequisites

Setup an EC2 instance of type t2.small
Ubuntu 22.04 LTS as AMI
30 GB of hard disk space
Open ports 22 for SSH, 9090 for Prometheus and 9100 for Node Exporter

Installation

Prometheus

$ ssh -i prometheus.pem ubuntu@3.80.133.119

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.2 LTS
Release:        22.04
Codename:       jammy

Create a new user for managing the Prometheus process

Upon executing the command it will create a user called prometheus without a home directory and shell as /bin/false

$ sudo useradd --shell /bin/false --no-create-home prometheus

Create a new directory for storing our Prometheus configuration file and update the ownership of the directory

$ sudo mkdir -p /etc/prometheus

$ sudo chown prometheus:prometheus /etc/prometheus

Create another directory for storing Prometheus time series data and update the ownership of the directory

$ sudo mkdir -p /var/lib/prometheus

$ sudo chown prometheus:prometheus /var/lib/prometheus

Download and extract the latest Prometheus release from their GitHub page

$ wget https://github.com/prometheus/prometheus/releases/download/v2.46.0/prometheus-2.46.0.linux-amd64.tar.gz

$ tar -xzvf prometheus-2.46.0.linux-amd64.tar.gz

Change the directory and install Prometheus and Promtool binaries

$ cd prometheus-2.46.0.linux-amd64

$ sudo install prometheus /usr/local/bin

$ sudo install promtool /usr/local/bin

Change the permission of these binaries to prometheus user

$ sudo chown prometheus:prometheus /usr/local/bin/prometheus

$ sudo chown prometheus:prometheus /usr/local/bin/promtool

Check the version of prometheus or promtool binaries

$ prometheus --version
prometheus, version 2.46.0 (branch: HEAD, revision: cbb69e51423565ec40f46e74f4ff2dbb3b7fb4f0)
  build user:       root@42454fc0f41e
  build date:       20230725-12:31:24
  go version:       go1.20.6
  platform:         linux/amd64
  tags:             netgo,builtinassets,stringlabels

$ promtool --version
promtool, version 2.46.0 (branch: HEAD, revision: cbb69e51423565ec40f46e74f4ff2dbb3b7fb4f0)
  build user:       root@42454fc0f41e
  build date:       20230725-12:31:24
  go version:       go1.20.6
  platform:         linux/amd64
  tags:             netgo,builtinassets,stringlabels

Copy the consoles and console_libraries directory which is used for dashboarding and visualization

$ sudo cp -rv consoles /etc/prometheus

$ sudo cp -rv console_libraries /etc/prometheus

Change the ownerships of the directories recursively to prometheus user

$ sudo chown -R prometheus:prometheus /etc/prometheus/consoles

$ sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries

Copy the configuration file to the above-mentioned directory and change its ownership to prometheus user

$ sudo cp prometheus.yml /etc/prometheus

$ sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml

Create a systemd unit file for the Prometheus service

$ sudo vi /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus \
    --web.console.templates /etc/prometheus/consoles \
    --web.console.libraries /etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

Start the prometheus process and enable the process at boot

$ sudo systemctl daemon-reload

$ sudo systemctl start prometheus

$ sudo systemctl enable prometheus

Check the prometheus service status

$ sudo systemctl is-active prometheus
active

$ sudo systemctl is-enabled prometheus
enabled

Open http://3.80.133.119:9090 in the browser and you can see the Prometheus expression browser

Node Exporter

Create a new user for managing the node exporter process

$ sudo useradd --shell /bin/false --no-create-home node_exporter

Download and extract the latest Node Exporter release from their GitHub page

$ wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz

$ tar -xzvf node_exporter-1.6.1.linux-amd64.tar.gz

Change the directory and install the node_exporter binary

$ cd node_exporter-1.6.1.linux-amd64

$ sudo install node_exporter /usr/local/bin

Change the ownership of the binary to node_exporter user

$ sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

Check the version of the node_exporter binary

$ node_exporter --version
node_exporter, version 1.6.1 (branch: HEAD, revision: 4a1b77600c1873a8233f3ffb55afcedbb63b8d84)
  build user:       root@586879db11e5
  build date:       20230717-12:10:52
  go version:       go1.20.6
  platform:         linux/amd64
  tags:             netgo osusergo static_build

Create a systemd unit file for the Node Exporter service

$ sudo vi /etc/systemd/system/node_exporter.service

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

Start the node_exporter process and enable the process at boot

$ sudo systemctl daemon-reload

$ sudo systemctl start node_exporter

$ sudo systemctl enable node_exporter

Check the status of the node_exporter process

$ sudo systemctl is-active node_exporter
active

$ sudo systemctl is-enabled node_exporter
enabled

Open http://3.80.133.119:9100/metrics in your browser to view all the metrics

Configuration

Prometheus

Now we have installed Prometheus and Node Exporter on the server and we need to modify the Prometheus configuration file to add new targets to scrape metrics

In the installation step, we already copied a default Prometheus configuration file to the /etc/prometheus location. You can edit this file or replace it with the below contents.

Here I have added a new job called "node_exporter" to scrape metrics from the same VM, instead of using localhost I have used the private IP of the instance

$ sudo vi /etc/prometheus/prometheus.yml

global:
  scrape_interval: 15s
  scrape_timeout: 10s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: "node_exporter"
    static_configs:
      - targets: ["172.31.93.117:9100"]

Validate the configuration file using the promtool

$ promtool check config /etc/prometheus/prometheus.yml

Checking /etc/prometheus/prometheus.yml
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

Restart the Prometheus service to effect new configuration changes

$ sudo systemctl restart prometheus

$ sudo systemctl status prometheus

Open the expression browser and navigate to Status -> Targets to view the newly added "node_exporter" target

We can view an example metric called "node_memory_Active_bytes" using the expression browser

Select the Graph option to view it as a graph

That's all for now

Reference

https://prometheus.io/docs/prometheus/latest/getting_started/

https://kodekloud.com/courses/prometheus-certified-associate-pca/

Infisical - Open Source SecretOps - Kubernetes Setup

Unni P — Thu, 17 Aug 2023 17:21:51 +0000

In this article, we will talk about Infisical, an open-source secret management tool and how we can setup on a local Kubernetes cluster

Introduction

An open-source, end-to-end secret management platform
Enables teams to easily manage and sync their environment variables, API keys, secrets and other configurations

Features

Intuitive dashboard
Client SDK's
Infisical CLI
Native platform integrations
Automatic Kubernetes deployment secret reloads
Complete control of data when hosted by ourself
Secret versioning
Point-in-Time recovery
Role-based access control
Secret scanning and leak prevention
Effortless on-premise deployment

Prerequisites

kubectl

Check my article to install kubectl

$ kubectl version --client --short
Client Version: v1.27.4
Kustomize Version: v5.0.1

Helm

Check my article to install Helm

$ helm version
version.BuildInfo{Version:"v3.12.1", GitCommit:"f32a527a060157990e2aa86bf45010dfb3cc8b8d", GitTreeState:"clean", GoVersion:"go1.20.4"}

Kubernetes cluster

For this demo, I'm using Rancher Desktop on Windows

$ kubectl get nodes

NAME              STATUS   ROLES                  AGE   VERSION
rancher-desktop   Ready    control-plane,master   27s   v1.27.4+k3s1

Installation

Add the Infisical Helm repository

$ helm repo add infisical-helm-charts 'https://dl.cloudsmith.io/public/infisical/helm-charts/helm/charts/' 

$ helm repo update

Install Infisical using the below command.

It will install all the necessary components in the infisical namespace and also install Nginx ingress controller

$ helm install -n infisical infisical infisical-helm-charts/infisical --set ingress.nginx.enabled=true --create-namespace

NAME: infisical
LAST DEPLOYED: Thu Aug 17 19:43:06 2023
NAMESPACE: infisical
STATUS: deployed
REVISION: 1
TEST SUITE: None

Check the status of the components and the load balancer service created by Nginx ingress controller

Copy the Load Balancer IP, we will need it in the next step

$ kubectl -n infisical get pods

NAME                                                  READY   STATUS    RESTARTS   AGE
infisical-frontend-8588c9f65-sxz8d                    1/1     Running   0          3m19s
infisical-frontend-8588c9f65-nsnjr                    1/1     Running   0          3m19s
mongodb-0                                             1/1     Running   0          3m19s
infisical-backend-6777b66f58-79d84                    1/1     Running   0          3m19s
infisical-backend-6777b66f58-g2fwb                    1/1     Running   0          3m19s
infisical-ingress-nginx-controller-7785998dc6-lqxgk   1/1     Running   0          3m19s

```bash
$ kubectl -n infisical get svc

NAME                                           TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE
infisical-backend                              ClusterIP      10.43.203.3     <none>           4000/TCP                     9m47s
infisical-frontend                             ClusterIP      10.43.41.186    <none>           3000/TCP                     9m47s
infisical-ingress-nginx-controller-admission   ClusterIP      10.43.42.222    <none>           443/TCP                      9m47s
mongodb                                        ClusterIP      10.43.112.139   <none>           27017/TCP                    9m47s
infisical-ingress-nginx-controller             LoadBalancer   10.43.63.38     172.31.134.241   80:30035/TCP,443:31538/TCP   9m47s
```

Initial Setup

In our local setup, we are skipping SMTP configuration and accessing the dashboard via the LoadBalancer IP address http://172.31.134.241/signup
You can also set up the hostname using ingress.hostName=<your-hostname> option during the installation
Create an account for the administrator by clicking the "Continue with Email" option

Enter your email id and click the "Get Started" option

Enter the details accordingly and click "Sign Up" option

Once you sign up, you will need to download "Emergency Kit" and save it somewhere safe. If you get locked out of your account, we can use this emergency kit to unlock it

Now we are redirected to the homepage of Infisical

Configuration

Create a new project by clicking the "Add New Project" button from the dashboard and name your project "MyApp"

Once our project is created, we will get an interface like below. Here we can see different environments like Development, Staging and Production. We are going to add the secrets in the Development environment by clicking the "Go to Development" option

We can copy secrets from other environments, upload env files etc. We can create a new secret by clicking the "Add a new secret" option

Add the required secrets and save the changes

Secrets Operator Setup

First, we need to generate a Service Token from our project settings

Select the "Create token" option and enter a name for the service token. Select the environment, secrets path, expiration and permissions according to your use case and click on "Create" option

Once the service token is created, copy and save it somewhere safe. We need this token to configure the secrets operator

Back in our Kubernetes cluster, we need to install and configure the Infisical secrets operator to sync our secrets to the cluster

$ helm install -n infisical infisical-secrets-operator infisical-helm-charts/secrets-operator

NAME: infisical-secrets-operator
LAST DEPLOYED: Thu Aug 17 21:05:56 2023
NAMESPACE: infisical
STATUS: deployed
REVISION: 1
TEST SUITE: None

Check the infisical namespace, we can see a secret controller manager pod is created and it's up and running

$ kubectl -n infisical get pods

NAME                                                  READY   STATUS    RESTARTS   AGE
infisical-frontend-8588c9f65-sxz8d                    1/1     Running   0          84m
infisical-frontend-8588c9f65-nsnjr                    1/1     Running   0          84m
mongodb-0                                             1/1     Running   0          84m
infisical-backend-6777b66f58-79d84                    1/1     Running   0          84m
infisical-backend-6777b66f58-g2fwb                    1/1     Running   0          84m
infisical-ingress-nginx-controller-7785998dc6-lqxgk   1/1     Running   0          84m
infisical-secre-controller-manager-56c6f9b6d-lqk2v    2/2     Running   0          100s

Create a new namespace for our application

$ kubectl create ns myapp

namespace/myapp created

Create a Kubernetes secret containing the Service Token

$ kubectl -n myapp create secret generic myapp-service-token --from-literal=infisicalToken=st.64de3f83a96c27c805827382.a7acb98a535125353af9135009f9b974.3d41003562f52a730823347dd4a01f96

secret/myapp-service-token created

```bash
$ kubectl -n myapp get secrets

NAME                  TYPE     DATA   AGE
myapp-service-token   Opaque   1      25s
```

Now we can sync our secrets to our cluster by creating an InfisicalSecret CRD

apiVersion: secrets.infisical.com/v1alpha1
kind: InfisicalSecret
metadata:
  name: myapp-infisical-secret
  namespace: myapp
spec:
  hostAPI: http://infisical-backend.infisical.svc.cluster.local:4000/api
  resyncInterval: 10
  authentication:
    serviceToken:
      serviceTokenSecretReference:
        secretName: myapp-service-token
        secretNamespace: myapp
      secretsScope:
        envSlug: dev
        secretsPath: "/"
  managedSecretReference:
    secretName: myapp-managed-secret
    secretNamespace: myapp

```bash
$ kubectl apply -f myapp-infisical-secret.yml
infisicalsecret.secrets.infisical.com/myapp-infisical-secret created
```

Once it's created, check the status of the CRD

From the status, we can see our secrets are synced to our cluster
```
$ kubectl -n myapp get infisicalsecrets

NAME                     AGE
myapp-infisical-secret   42s
```

```bash
$ kubectl -n myapp describe infisicalsecrets myapp-infisical-secret

Name:         myapp-infisical-secret
Namespace:    myapp
Labels:       <none>
Annotations:  <none>
API Version:  secrets.infisical.com/v1alpha1
Kind:         InfisicalSecret
Metadata:
  Creation Timestamp:  2023-08-17T16:07:04Z
  Generation:          1
  Resource Version:    4927
  UID:                 bb156757-1528-4827-afe9-09916fcd4372
Spec:
  Authentication:
    Service Token:
      Secrets Scope:
        Env Slug:      dev
        Secrets Path:  /
      Service Token Secret Reference:
        Secret Name:       myapp-service-token
        Secret Namespace:  myapp
  Host API:                http://infisical-backend.infisical.svc.cluster.local:4000/api
  Managed Secret Reference:
    Secret Name:       myapp-managed-secret
    Secret Namespace:  myapp
  Resync Interval:     10
Status:
  Conditions:
    Last Transition Time:  2023-08-17T16:07:04Z
    Message:               Infisical controller has located the Infisical token in provided Kubernetes secret
    Reason:                OK
    Status:                True
    Type:                  secrets.infisical.com/LoadedInfisicalToken
    Last Transition Time:  2023-08-17T16:07:05Z
    Message:               Infisical controller has started syncing your secrets
    Reason:                OK
    Status:                True
    Type:                  secrets.infisical.com/ReadyToSyncSecrets
    Last Transition Time:  2023-08-17T16:07:05Z
    Message:               Infisical has found 0 deployments which are ready to be auto redeployed when secrets change
    Reason:                OK
    Status:                True
    Type:                  secrets.infisical.com/AutoRedeployReady
Events:                    <none>
```

We can verify the secrets have been synced to our cluster by checking the myapp-managed-secret

$ kubectl -n myapp get secrets myapp-managed-secret

NAME                   TYPE     DATA   AGE
myapp-managed-secret   Opaque   4      5m16s

```bash
$ kubectl -n myapp describe secrets myapp-managed-secret

Name:         myapp-managed-secret
Namespace:    myapp
Labels:       <none>
Annotations:  secrets.infisical.com/version: W/"a5a-hBx83Z5L+nuphYXlN17htm8jAjo"

Type:  Opaque

Data
====
MYSQL_HOST:      9 bytes
MYSQL_PASSWORD:  13 bytes
MYSQL_PORT:      4 bytes
MYSQL_DATABASE:  13 bytes
```

Application Deployment

Deploy a sample Nginx application using the below manifest file

The annotation secrets.infisical.com/auto-reload: "true" ensures that it automatically redeploys when managed secrets are changed

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: myapp
  labels:
    app: myapp
  annotations: 
    secrets.infisical.com/auto-reload: "true"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: maypp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: nginx:1.25.2
        envFrom:
        - secretRef:
            name: myapp-managed-secret
        ports:
        - containerPort: 80

```bash
$ kubectl apply -f .\myapp-deployment.yml
deployment.apps/myapp created
```

Once our application is up and running, exec into the pod and list the environment variables to view our secrets

$ kubectl -n myapp get pods

NAME                     READY   STATUS    RESTARTS   AGE
myapp-77c586c9ff-5xsxv   1/1     Running   0          48s

```bash
$ kubectl -n myapp exec -it myapp-77c586c9ff-5xsxv -- bash

root@myapp-77c586c9ff-5xsxv:/# env | grep -i MYSQL
MYSQL_PORT=3306
MYSQL_PASSWORD=mysqlPassword
MYSQL_HOST=mysqlHost
MYSQL_DATABASE=mysqlDatabase
```

That's all for now

Reference

https://infisical.com/docs/integrations/platforms/kubernetes

https://docs.rancherdesktop.io/getting-started/installation/

Uptime Kuma on Amazon EC2 (Ubuntu) - An Open-source Monitoring Tool

Unni P — Wed, 03 May 2023 15:01:07 +0000

In this article, we will look at how we can install and configure Uptime Kuma an open-source monitoring tool on Amazon EC2 Ubuntu instance

Introduction

An open-source self-hosted monitoring tool
Monitor uptime for HTTP, HTTPS, DNS etc
SSL certificate information
Notifications via Email, Slack, Discord etc
20 seconds interval
Support for proxy and multi-factor authentication

Prerequisites

Setup an EC2 instance of type t2.micro
Ubuntu 22.04 LTS as AMI
10 GB of hard disk space
Open port 22 for SSH and 3001 for Uptime Kuma
Install and configure Docker
Need a temporary webpage with SSL enabled

Installation

$ ssh -i uptime-kuma.pem ubuntu@54.234.8.87

Create a Docker volume for Uptime Kuma

$ docker volume create uptime-kuma

$ docker volume ls
DRIVER    VOLUME NAME
local     uptime-kuma

Start the container and verify the status

$ docker container run -d --restart=always -p 3001:3001 -v uptime-kuma:/app/data --name uptime-kuma louislam/uptime-kuma:1
Unable to find image 'louislam/uptime-kuma:1' locally
1: Pulling from louislam/uptime-kuma
3689b8de819b: Pull complete
4178a276654a: Pull complete
b46162c13de5: Pull complete
4d3ac03f17d8: Pull complete
b935255dae7e: Pull complete
792f129a81f3: Pull complete
4110002867ba: Pull complete
390f8662c74f: Pull complete
9dd174cf6e30: Pull complete
4f4fb700ef54: Pull complete
703bad70ccf2: Pull complete
Digest: sha256:cf61d3262b29e1c48cc2ac284c9264227bbc46168f408e5f4c4d6301f0629e41
Status: Downloaded newer image for louislam/uptime-kuma:1
fcdfcc5a5d1470ca3b1dd1e23d29a4975d909e2a5ab2f53e1e2bef0fa4a58665

$ docker container ls -a
CONTAINER ID   IMAGE                    COMMAND                  CREATED              STATUS                        PORTS                                       NAMES
fcdfcc5a5d14   louislam/uptime-kuma:1   "/usr/bin/dumb-init …"   About a minute ago   Up About a minute (healthy)   0.0.0.0:3001->3001/tcp, :::3001->3001/tcp   uptime-kuma

Open http://54.234.8.87:3001 in your browser

Initially, we need to set up a username and password for accessing the dashboard

Once done, we will redirect to the dashboard

Configuration

I have already configured an Nginx website and enabled SSL using Let’s Encrypt

We need to set up Notifications to get notified whenever a host is down or a certificate is going to expire

If we are using Email (SMTP) as a notification type then we need to do configuration in our Gmail account

Enable multi-factor authentication
Create app password

From the dashboard, navigate to Settings → Notifications → Setup Notification and configure as below to receive email notifications

Notification Type: Email (SMTP)
Friendly Name: Alerts
Hostname: smtp.gmail.com
Port: 587
Security: None/STARTTLS (25, 587)

Username: <your-email-id>
Password: <app-password>
From Email: <your-email-id>
To Email: <your-email-id>

Custom Subject: {{NAME}} {{STATUS}}
Default enabled: Enable

Once the required information is entered click the Test and Save button and you will receive a test mail in your inbox

Let's add our URL to monitor by clicking the Add New Monitor button and filling in the details and clicking the Save button

Friendly Name: 158b5452c43c.mylabserver.com
URL: https://158b5452c43c.mylabserver.com
Heartbeat Interval: 20
Certificate Expiry Notification: Enable

From the dashboard, we can see the uptime, response time of our URL and certificate expiry date

Now make a downtime by stopping the Nginx service and we can see our URL is showing the status as DOWN and will receive an email in your inbox

Let's start the Nginx service now our URL status is showing as UP and will receive an email in your inbox

As you see from the above status, the Let’s Encrypt certificate is going to expire in 89 days.

Let's configure Uptime Kuma to receive an alert for certificate expiry

Navigate to Settings → Notifications → TLS Certificate Expiry

**By default, we will get certificate expiry notifications in 7, 14 and 21 days but we are going configure it as 89 days because my certificate will expire in 89 days click the **Save button and wait to receive an email in your inbox

Reference

https://github.com/louislam/uptime-kuma

https://mariushosting.com/synology-activate-gmail-smtp-for-docker-containers/

Setting up IBM Db2 Community Edition on Amazon EC2 (Ubuntu)

Unni P — Wed, 03 May 2023 13:51:39 +0000

In this article we will look how we can install IBM Db2 community edition on Amazon EC2 Ubuntu instance

Introduction

Db2 is a cloud-native database
Low latency transactions and highly resilient
Supports structured and unstructured data
Entry-level edition of the Db2 data server for the developer and partner community
Available for Linux, Windows, and AIX and also available as a Docker image
Supports all core Db2 features
Up to 4 cores and 16 GB RAM
Always-on security
Db2 Community support

Prerequisites

Setup an EC2 instance of type t2.medium
Ubuntu 20.04 LTS as AMI
30 GB of hard disk space
Open port 22 for SSH and 25000 for Db2
Create an IBM account for downloading the Db2 community edition

Installation

$ ssh -i ibm-db2.pem ubuntu@54.234.180.34

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.6 LTS
Release:        20.04
Codename:       focalDownload and extract Db2 community edition tarball on EC2 instance
You can download the community edition from this URL

Download the latest IBM Db2 Linux (x64) version on your local machine
Once the download is completed, copy the downloaded file to your EC2 instance

$ scp -i ibm-db2.pem v11.5.8_linuxx64_server_dec.tar.gz ubuntu@54.234.180.34:/home/ubuntu

$ pwd
/home/ubuntu

$ ls
v11.5.8_linuxx64_server_dec.tar.gz

Extract the tarball in your home directory

$ tar -xzvf v11.5.8_linuxx64_server_dec.tar.gz

$ ls
server_dec  v11.5.8_linuxx64_server_dec.tar.gz

Move to the extracted directory and execute db2prereqcheck command This will check the prerequisite requirements for installing Db2

$ cd server_dec

$ ls
db2  db2_deinstall  db2_install  db2checkCOL.tar.gz  db2checkCOL_readme.txt  db2ckupgrade  db2ls  db2prereqcheck  db2setup  installFixPack

$ sudo ./db2prereqcheck

==========================================================================

Sun Apr 30 04:51:27 2023
Checking prerequisites for DB2 installation. Version "11.5.8.0". Operating system "Linux"

Validating "kernel level " ...
   Required minimum operating system kernel level: "3.10.0".
   Actual operating system kernel level: "5.15.0".
   Requirement matched.

Validating "Linux distribution " ...
   Required minimum "UBUNTU" version: "16.04"
   Actual version: "20.04"
   Requirement matched.

Validating "ksh symbolic link" ...
   WARNING : Requirement not matched.
ERROR:
   The 'strings' utility that is used to detect prerequisite libraries
   is not present on this system.  Please use your package or software
   manager to install the GNU Binary Utilities.

Validating "C++ Library version " ...
   Required minimum C++ library: "libstdc++.so.6"
   Standard C++ library is located in the following directory: "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28".
DBT3512W  The db2prereqcheck utility failed to determine the currently-installed version of the C++ standard library, libstdc++.
   Requirement matched.

Validating "libaio.so version " ...
DBT3553I  The db2prereqcheck utility successfully loaded the libaio.so.1 file.
   Requirement matched.

Validating "libnuma.so version " ...
DBT3610I  The db2prereqcheck utility successfully loaded the libnuma.so.1 file.
   Requirement matched.

Validating "/lib/i386-linux-gnu/libpam.so*" ...
   DBT3514W  The db2prereqcheck utility failed to find the following 32-bit library file: "/lib/i386-linux-gnu/libpam.so*".
   WARNING : Requirement not matched.
Requirement not matched for DB2 database "Server" . Version: "11.5.8.0".
Summary of prerequisites that are not met on the current system:
   DBT3514W  The db2prereqcheck utility failed to find the following 32-bit library file: "/lib/i386-linux-gnu/libpam.so*".


DBT3619W  The db2prereqcheck utility detected that ksh is not linked to ksh or ksh93. This is required for Db2 High Availability Feature with Tivoli SA MP.

From the above output, we can see the requirement checks are failed and we need to fix them
Enable 32-bit architecture on your instance and install the required packages

$ sudo dpkg --add-architecture i386

$ sudo apt update

$ sudo apt install -y ksh ksh93 lib32stdc++6 libpam0g:i386 binutils

Once the required packages are installed, rerun the db2prereqcheck command again and verify all the requirements are matched

$ sudo ./db2prereqcheck

==========================================================================

Sun Apr 30 04:55:30 2023
Checking prerequisites for DB2 installation. Version "11.5.8.0". Operating system "Linux"

Validating "kernel level " ...
   Required minimum operating system kernel level: "3.10.0".
   Actual operating system kernel level: "5.15.0".
   Requirement matched.

Validating "Linux distribution " ...
   Required minimum "UBUNTU" version: "16.04"
   Actual version: "20.04"
   Requirement matched.

Validating "ksh symbolic link" ...
   Requirement matched.

Validating "C++ Library version " ...
   Required minimum C++ library: "libstdc++.so.6"
   Standard C++ library is located in the following directory: "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28".
   Actual C++ library: "CXXABI_1.3.1"
   Requirement matched.


Validating "32 bit version of "libstdc++.so.6" " ...
   Found the 64 bit "/lib/x86_64-linux-gnu/libstdc++.so.6" in the following directory "/lib/x86_64-linux-gnu".
   Found the 32 bit "/lib32/libstdc++.so.6" in the following directory "/lib32".
   Requirement matched.

Validating "libaio.so version " ...
DBT3553I  The db2prereqcheck utility successfully loaded the libaio.so.1 file.
   Requirement matched.

Validating "libnuma.so version " ...
DBT3610I  The db2prereqcheck utility successfully loaded the libnuma.so.1 file.
   Requirement matched.

Validating "/lib/i386-linux-gnu/libpam.so*" ...
   Requirement matched.
DBT3533I  The db2prereqcheck utility has confirmed that all installation prerequisites were met.

Multiple installation methods available for Db2 for specific use cases
We are going to install Db2 using db2_install command as root user and wait for the installation to complete

$ sudo ./db2_install
Read the license agreement file in the db2/license directory.

***********************************************************
To accept those terms, enter "yes". Otherwise, enter "no" to cancel the install process. [yes/no]
yes


Default directory for installation of products - /opt/ibm/db2/V11.5

***********************************************************
Install into default directory (/opt/ibm/db2/V11.5) ? [yes/no]
yes


Specify one of the following keywords to install DB2 products.

  SERVER
  CONSV
  CLIENT
  RTCL

Enter "help" to redisplay product names.

Enter "quit" to exit.

***********************************************************
SERVER
***********************************************************
Do you want to install the DB2 pureScale Feature? [yes/no]
no
DB2 installation is being initialized.

Once the installation is completed, you will see the below message and we can check the installation log file for post-installation steps

The execution completed successfully.

For more information see the DB2 installation log at
"/tmp/db2_install.log.5295".

$ cat /tmp/db2_install.log.5295
Post-installation instructions
-------------------------------

Required steps:
Set up a DB2 instance to work with DB2.

Optional steps:
Notification SMTP server has not been specified. Notifications cannot be sent to contacts in your contact list until this is specified. For more information see the DB2 administration documentation.

To validate your installation files, instance, and database functionality, run the Validation Tool, /opt/ibm/db2/V11.5/bin/db2val. For more information, see "db2val" in the DB2 Information Center.

Let’s validate our installation by executing the db2val command and verify the log file We can see that everything is OK

$ sudo /opt/ibm/db2/V11.5/bin/db2val
DBI1379I  The db2val command is running. This can take several minutes.

DBI1335I  Installation file validation for the DB2 copy installed at
      /opt/ibm/db2/V11.5 was successful.

DBI1343I  The db2val command completed successfully. For details, see
      the log file /tmp/db2val-230430_051357.log.

$ cat /tmp/db2val-230430_051357.log
Installation file validation for the DB2 copy installed at "/opt/ibm/db2/V11.5" starts.

Task 1: Validating Installation file sets.
Status 1 : Success

Task 2: Validating embedded runtime path for DB2 executables and libraries.
Status 2 : Success

Task 3: Validating the accessibility to the installation path.
Status 3 : Success

Task 4: Validating the accessibility to the /etc/services file.
Status 4 : Success

DBI1335I  Installation file validation for the DB2 copy installed at
      /opt/ibm/db2/V11.5 was successful.

Installation file validation for the DB2 copy installed at "/opt/ibm/db2/V11.5" ends.

DBI1343I  The db2val command completed successfully. For details, see
      the log file /tmp/db2val-230430_051357.log.

Post Installation

Create required groups for Db2

$ sudo groupadd -g 998 db2iadm1

$ sudo groupadd -g 997 db2fsdm1

$ sudo groupadd -g 996 dasadm1

Create required users for Db2

$ sudo useradd -u 1004 -g db2iadm1 -m -d /home/db2inst1 db2inst1

$ sudo useradd -u 1003 -g db2fsdm1 -m -d /home/db2fenc1 db2fenc1

$ sudo useradd -u 1002 -g dasadm1 -m -d /home/dasusr1 dasusr1

Set passwords for users

$ sudo passwd db2inst1
New password:
Retype new password:
passwd: password updated successfully

$ sudo passwd db2fenc1
New password:
Retype new password:
passwd: password updated successfully

$ sudo passwd dasusr1
New password:
Retype new password:
passwd: password updated successfully

Create an instance for Db2 using db2icrt command and verify the log file for information about connecting to the database

$ sudo /opt/ibm/db2/V11.5/instance/db2icrt -a server -u db2fenc1 db2inst1
DBI1446I  The db2icrt command is running.


DB2 installation is being initialized.

 Total number of tasks to be performed: 4
Total estimated time for all tasks to be performed: 309 second(s)

Task #1 start
Description: Setting default global profile registry variables
Estimated time 1 second(s)
Task #1 end

Task #2 start
Description: Initializing instance list
Estimated time 5 second(s)
Task #2 end

Task #3 start
Description: Configuring DB2 instances
Estimated time 300 second(s)
Task #3 end

Task #4 start
Description: Updating global profile registry
Estimated time 3 second(s)
Task #4 end

The execution completed successfully.

For more information see the DB2 installation log at "/tmp/db2icrt.log.85927".
DBI1070I  Program db2icrt completed successfully.

$ cat /tmp/db2icrt.log.85927

Required steps:
You can connect to the DB2 instance "db2inst1" using the port number "25000". Record it for future reference.

Let’s start the database instance using the below commands

$ sudo su - db2inst1

$ db2ls

Install Path                       Level   Fix Pack   Special Install Number   Install Date                  Installer UID
---------------------------------------------------------------------------------------------------------------------
/opt/ibm/db2/V11.5               11.5.8.0        0                            Sun Apr 30 05:08:04 2023 UTC             0

$ . sqllib/userprofile

$ db2ilist
db2inst1

$ db2start
04/30/2023 05:36:30     0   0   SQL1063N  DB2START processing was successful.
SQL1063N  DB2START processing was successful.

Now the database instance is started and we can connect to the instance

$ db2
(c) Copyright IBM Corporation 1993,2007
Command Line Processor for DB2 Client 11.5.8.0

You can issue database manager commands and SQL statements from the command
prompt. For example:
    db2 => connect to sample
    db2 => bind sample.bnd

For general help, type: ?.
For command help, type: ? command, where command can be
the first few keywords of a database manager command. For example:
 ? CATALOG DATABASE for help on the CATALOG DATABASE command
 ? CATALOG          for help on all of the CATALOG commands.

To exit db2 interactive mode, type QUIT at the command prompt. Outside
interactive mode, all commands must be prefixed with 'db2'.
To list the current command option settings, type LIST COMMAND OPTIONS.

For more detailed help, refer to the Online Reference Manual.

db2 =>

Create a test database and connect to the database using the below commands

db2 => create database ibm
DB20000I  The CREATE DATABASE command completed successfully.

db2 => connect to ibm

   Database Connection Information

 Database server        = DB2/LINUXX8664 11.5.8.0
 SQL authorization ID   = DB2INST1
 Local database alias   = IBM

Enable automatic start of database instance after reboot Execute the below commands as db2inst1 user

$ db2greg -getinstrec instancename='db2inst1'
Retrieved record:
   Service      = |DB2|
   Version      = |11.5.8.0|
   InstanceName = |db2inst1|
   InstancePath = |/home/db2inst1/sqllib|
   Usage        = |N/A|
   StartAtBoot  = 1
   Maintenance  = 0
   InstallPath  = |/opt/ibm/db2/V11.5|
   RemoteProf   = |N/A|
   Comment      = |N/A|

$ db2iauto -on db2inst1

Reboot the EC2 instance and verify the Db2 process

$ sudo su - db2inst1

$ ps -ef | grep -i db2
root         469       1  0 05:48 ?        00:00:00 /opt/ibm/db2/V11.5/bin/db2fmcd
root        1127    1096  0 05:48 pts/0    00:00:00 sudo su - db2inst1
root        1128    1127  0 05:48 pts/0    00:00:00 su - db2inst1
db2inst1    1129    1128  0 05:48 pts/0    00:00:00 -sh
root        1626       1  0 05:49 ?        00:00:00 db2wdog 0 [db2inst1]
db2inst1    1628    1626  1 05:49 ?        00:00:00 db2sysc 0
root        1635    1626  0 05:49 ?        00:00:00 db2ckpwd 0
root        1636    1626  0 05:49 ?        00:00:00 db2ckpwd 0
root        1637    1626  0 05:49 ?        00:00:00 db2ckpwd 0
db2inst1    1639    1626  0 05:49 ?        00:00:00 db2vend (PD Vendor Process - 1) 0
db2inst1    1647    1626  1 05:49 ?        00:00:00 db2acd 0 ,0,0,0,1,0,0,00000000,0,0,0000000000000000,0000000000000000,00000000,00000000,00000000,00000000,00000000,00000000,0000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000c3834000,0000000000000000,0000000000000000,1,0,0,,,,,a89f30,14,1e014,2,0,1,0000000000041fc0,0x240000000,0x240000000,1600000,2,2,13
db2inst1    1673    1129  0 05:49 pts/0    00:00:00 ps -ef
db2inst1    1674    1129  0 05:49 pts/0    00:00:00 grep -i db2

$ systemctl status db2fmcd
● db2fmcd.service - DB2 v11.5.8.0
     Loaded: loaded (/etc/systemd/system/db2fmcd.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2023-04-30 05:48:42 UTC; 11min ago
   Main PID: 469 (db2fmcd)
      Tasks: 58 (limit: 4686)
     Memory: 1.0G
     CGroup: /system.slice/db2fmcd.service
             ├─ 469 /opt/ibm/db2/V11.5/bin/db2fmcd
             ├─1626 db2wdog 0 [db2inst1]
             ├─1628 db2sysc 0
             ├─1635 db2ckpwd 0
             ├─1636 db2ckpwd 0
             ├─1637 db2ckpwd 0
             ├─1639 db2vend (PD Vendor Process - 1) 0
             ├─1647 db2acd 0 ,0,0,0,1,0,0,00000000,0,0,0000000000000000,0000000000000000,00000000,00000000,00000000,00000000,00000000,00000000,0000,00000000,00000000,00000000,00000000,00000000>
             └─2450 db2fmp ( ,1,0,0,0,0,0,00000000,0,0,0000000000000000,0000000000000000,00000000,00000000,00000000,00000000,00000000,00000000,0000,00000000,00000000,00000000,00000000,00000000>

Connect to our test database and we can see everything is OK

$ db2
(c) Copyright IBM Corporation 1993,2007
Command Line Processor for DB2 Client 11.5.8.0

You can issue database manager commands and SQL statements from the command
prompt. For example:
    db2 => connect to sample
    db2 => bind sample.bnd

For general help, type: ?.
For command help, type: ? command, where command can be
the first few keywords of a database manager command. For example:
 ? CATALOG DATABASE for help on the CATALOG DATABASE command
 ? CATALOG          for help on all of the CATALOG commands.

To exit db2 interactive mode, type QUIT at the command prompt. Outside
interactive mode, all commands must be prefixed with 'db2'.
To list the current command option settings, type LIST COMMAND OPTIONS.

For more detailed help, refer to the Online Reference Manual.

db2 => connect to ibm

   Database Connection Information

 Database server        = DB2/LINUXX8664 11.5.8.0
 SQL authorization ID   = DB2INST1
 Local database alias   = IBM

Reference

https://www.dbi-services.com/blog/setting-up-ibm-db2-on-linux-root-installation/

https://www.ibm.com/docs/en/db2/11.5

Prometheus - Quick Setup on Amazon EC2 (Ubuntu) - Part 2

Unni P — Wed, 03 May 2023 13:32:39 +0000

In this article, we will look at how we can quickly setup Prometheus and Node Exporter on a Ubuntu instance on Amazon EC2

Prerequisites

Setup an EC2 instance of type t2.micro
Ubuntu 22.04 LTS as AMI
10 GB of hard disk space
Open ports 22 for SSH, 9090 for Prometheus and 9100 for Node Exporter

Installation

Prometheus

$ ssh -i <key_name>.pem ubuntu@<ip_address>

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.2 LTS
Release:        22.04
Codename:       jammy

Download and extract the latest release from their GitHub page

$ wget https://github.com/prometheus/prometheus/releases/download/v2.43.0/prometheus-2.43.0.linux-amd64.tar.gz

$ tar -xzvf prometheus-2.43.0.linux-amd64.tar.gz
prometheus-2.43.0.linux-amd64/
prometheus-2.43.0.linux-amd64/LICENSE
prometheus-2.43.0.linux-amd64/consoles/
prometheus-2.43.0.linux-amd64/consoles/prometheus.html
prometheus-2.43.0.linux-amd64/consoles/node-disk.html
prometheus-2.43.0.linux-amd64/consoles/node-overview.html
prometheus-2.43.0.linux-amd64/consoles/prometheus-overview.html
prometheus-2.43.0.linux-amd64/consoles/index.html.example
prometheus-2.43.0.linux-amd64/consoles/node-cpu.html
prometheus-2.43.0.linux-amd64/consoles/node.html
prometheus-2.43.0.linux-amd64/prometheus
prometheus-2.43.0.linux-amd64/promtool
prometheus-2.43.0.linux-amd64/NOTICE
prometheus-2.43.0.linux-amd64/console_libraries/
prometheus-2.43.0.linux-amd64/console_libraries/prom.lib
prometheus-2.43.0.linux-amd64/console_libraries/menu.lib
prometheus-2.43.0.linux-amd64/prometheus.yml

Change the directory and run the Prometheus binary

$ cd prometheus-2.43.0.linux-amd64/

$ ./prometheus
ts=2023-04-28T14:52:07.254Z caller=main.go:520 level=info msg="No time or size retention was set so using the default time retention" duration=15d
ts=2023-04-28T14:52:07.255Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)"
ts=2023-04-28T14:52:07.255Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)"
ts=2023-04-28T14:52:07.255Z caller=main.go:570 level=info host_details="(Linux 5.15.0-1031-aws #35-Ubuntu SMP Fri Feb 10 02:07:18 UTC 2023 x86_64 ip-172-31-84-196 (none))"
ts=2023-04-28T14:52:07.255Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)"
ts=2023-04-28T14:52:07.255Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2023-04-28T14:52:07.257Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
ts=2023-04-28T14:52:07.258Z caller=main.go:1005 level=info msg="Starting TSDB ..."
ts=2023-04-28T14:52:07.261Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any"
ts=2023-04-28T14:52:07.261Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.434µs
ts=2023-04-28T14:52:07.261Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while"
ts=2023-04-28T14:52:07.264Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9090
ts=2023-04-28T14:52:07.264Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9090
ts=2023-04-28T14:52:07.265Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
ts=2023-04-28T14:52:07.265Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=36.679µs wal_replay_duration=3.153164ms wbl_replay_duration=721ns total_replay_duration=3.306296ms
ts=2023-04-28T14:52:07.267Z caller=main.go:1026 level=info fs_type=EXT4_SUPER_MAGIC
ts=2023-04-28T14:52:07.267Z caller=main.go:1029 level=info msg="TSDB started"
ts=2023-04-28T14:52:07.267Z caller=main.go:1209 level=info msg="Loading configuration file" filename=prometheus.yml
ts=2023-04-28T14:52:07.274Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=prometheus.yml totalDuration=6.593992ms db_storage=1.674µs remote_storage=2.431µs web_handler=939ns query_engine=1.285µs scrape=6.143405ms scrape_sd=30.548µs notify=32.577µs notify_sd=12.129µs rules=1.905µs tracing=7.357µs
ts=2023-04-28T14:52:07.274Z caller=main.go:990 level=info msg="Server is ready to receive web requests."
ts=2023-04-28T14:52:07.274Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..."

Open http://public_ip:9090 in the browser and you can see the Prometheus expression browser

Navigate to Status → Targets to view all targets configured in the configuration file

Navigate to Status → Configuration to view the configuration file contents

Navigate to Status → TSDB Status to view time-series database details

Open http://public_ip:9090/metrics in the browser to view metrics

Press ctrl+c to stop running the Prometheus server

Node Exporter

A popular exporter that collects system-level metrics from Linux and Unix-based systems
Provides a wide range of metrics that can be used to monitor system health
Different metrics are CPU usage, memory usage, disk usage, network statistics etc
Download and extract the latest release from their GitHub page

$ wget https://github.com/prometheus/node_exporter/releases/download/v1.5.0/node_exporter-1.5.0.linux-amd64.tar.gz

$ tar -xzvf node_exporter-1.5.0.linux-amd64.tar.gz
node_exporter-1.5.0.linux-amd64/
node_exporter-1.5.0.linux-amd64/LICENSE
node_exporter-1.5.0.linux-amd64/NOTICE
node_exporter-1.5.0.linux-amd64/node_exporter

Change the directory and run the Node Exporter binary

$ cd node_exporter-1.5.0.linux-amd64/

$ ./node_exporter
ts=2023-04-28T15:27:11.226Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)"
ts=2023-04-28T15:27:11.226Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)"
ts=2023-04-28T15:27:11.227Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/)
ts=2023-04-28T15:27:11.227Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
ts=2023-04-28T15:27:11.227Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$
ts=2023-04-28T15:27:11.228Z caller=node_exporter.go:110 level=info msg="Enabled collectors"
ts=2023-04-28T15:27:11.228Z caller=node_exporter.go:117 level=info collector=arp
ts=2023-04-28T15:27:11.228Z caller=node_exporter.go:117 level=info collector=bcache
ts=2023-04-28T15:27:11.228Z caller=node_exporter.go:117 level=info collector=bonding
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=btrfs
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=conntrack
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=cpu
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=cpufreq
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=diskstats
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=dmi
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=edac
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=entropy
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=fibrechannel
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=filefd
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=filesystem
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=hwmon
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=infiniband
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=ipvs
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=loadavg
ts=2023-04-28T15:27:11.229Z caller=node_exporter.go:117 level=info collector=mdadm
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=meminfo
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=netclass
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=netdev
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=netstat
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=nfs
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=nfsd
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=nvme
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=os
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=powersupplyclass
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=pressure
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=rapl
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=schedstat
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=selinux
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=sockstat
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=softnet
ts=2023-04-28T15:27:11.230Z caller=node_exporter.go:117 level=info collector=stat
ts=2023-04-28T15:27:11.231Z caller=node_exporter.go:117 level=info collector=tapestats
ts=2023-04-28T15:27:11.231Z caller=node_exporter.go:117 level=info collector=textfile
ts=2023-04-28T15:27:11.231Z caller=node_exporter.go:117 level=info collector=thermal_zone
ts=2023-04-28T15:27:11.231Z caller=node_exporter.go:117 level=info collector=time
ts=2023-04-28T15:27:11.231Z caller=node_exporter.go:117 level=info collector=timex
ts=2023-04-28T15:27:11.231Z caller=node_exporter.go:117 level=info collector=udp_queues
ts=2023-04-28T15:27:11.231Z caller=node_exporter.go:117 level=info collector=uname
ts=2023-04-28T15:27:11.231Z caller=node_exporter.go:117 level=info collector=vmstat
ts=2023-04-28T15:27:11.231Z caller=node_exporter.go:117 level=info collector=xfs
ts=2023-04-28T15:27:11.231Z caller=node_exporter.go:117 level=info collector=zfs
ts=2023-04-28T15:27:11.231Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100
ts=2023-04-28T15:27:11.232Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100

Open http://public_ip:9100/metrics to view all the metrics

Press ctrl+c to stop running Node Exporter

Reference

https://prometheus.io/docs/introduction/first_steps/

https://prometheus.io/docs/guides/node-exporter/

Prometheus - Architecture - Part 1

Unni P — Wed, 03 May 2023 12:48:47 +0000

In the first part of the series we will look Prometheus introduction, features and its components

Introduction

Prometheus is an open-source monitoring and alerting tool
Originally built by SoundCloud in 2012
Donated to Cloud Native Computing Foundation in 2016
Second project hosted by CNCF after Kubernetes
CNCF graduated project
Collects and stores its metrics as time series data
Metrics information is stored with the timestamp at which it was recorded
Optional key-value pairs called labels are also stored

Features

Multi-dimensional data model — allows to define and store time-series data with multiple dimensions
PromQL — a query language to perform complex queries and calculations on metrics data
Time-series collection — can collect metrics data from a wide range of sources, including servers, applications, and databases
Data retention — can set retention policies on the metrics data
Service discovery — can automatically discover and monitor new services in a dynamic infrastructure
Exporters — different types of agents available for collecting metrics from different types of sources
Grafana integration — can easily be integrated with Grafana for visualization and custom dashboards
Alerting — can alert stakeholders when an issue occurs

Architecture

Prometheus Server

The main component of Prometheus
Collects, stores and serves metrics data
Follows a pull-based model for collecting metric data
Periodically queries the targets configured in the scrape configuration
Uses HTTP or HTTPS protocol
Stores the data in a time-series database
Time-series database allows Prometheus to do operations like querying, aggregation and visualization

Exporters

Agents that collect and expose metrics data from applications or third-party systems
Written in a wide variety of programming languages
Designed to be lightweight and easy to deploy
Scrape metrics data from a target and expose it on an HTTP endpoint
Metrics data is returned in a text-based format such as plain text, JSON, or protobuf
Official exporters are maintained by Prometheus GitHub Organization like Node Exporter, MySQL Exporter etc
Unofficial exporters are externally contributed and maintained like Apache Exporter, PostgreSQL Exporter etc

Alertmanager

Manages alerts generated by Prometheus
Receives alerts from Prometheus and groups them based on criteria like severity
Based on routing rules, it sends a notification to various receivers like Email, Slack, Pagerduty etc
Applies a set of deduplication rules to ensure that alerts are not sent multiple times for the same issue
Provides web interface that allows users to view and manage alerts

Pushgateway

Prometheus follows a pull-based model for collecting metrics data
Sometimes we need to push metrics to Prometheus which cannot be scraped
Helps to push metrics to Prometheus instead of pulling metric
Best suitable for short-lived jobs
Provides an HTTP API for pushing metrics from short-lived jobs

Service Discovery

Process of identifying and monitoring systems automatically
Helps Prometheus to keep track of different services which are running
Supports a variety of service discovery options for discovering scrape targets like Kubernetes, Consul, Docker etc
If you need a service discovery system that is currently not supported, we can use file-based service discovery
Enables you to list scrape targets in a JSON file along with metadata
Metadata is a piece of useful information about targets, like the name of the service, description etc
HTTP service discovery helps to discover targets over an HTTP endpoint an alternative to file-based service discovery

Client Libraries

Set of libraries that allows developers to instrument their applications to expose metrics to Prometheus
It exposes an HTTP endpoint and Prometheus can scrape metrics from these HTTP endpoints and store them in a time-series database
Once the metrics are stored, Prometheus can generate graphs, reports and alerts based on the data
Helps developers to get insights into their application performance
Official client libraries are available in many programming languages like Go, Python, Java etc
Unofficial client libraries are also available for C, C++, Node.js etc

Reference

https://prometheus.io/docs/introduction/overview/

Upgrading a Kubernetes Cluster using kubeadm

Unni P — Wed, 03 May 2023 12:21:23 +0000

In this article, we will look how we can upgrade a Kubernetes cluster using kubeadm

Prerequisites

We need a working Kubernetes cluster created using kubeadm
If you haven’t created the cluster, please check my previous article below

This will guide you to create a Kubernetes v1.27.0 cluster

Building a Kubernetes Cluster using kubeadm

Unni P ・ May 3 ・ 4 min read

#kubernetes #cloudnative #opensource

Steps

Control Plane

Verify the version of our cluster by executing the below command in the [control-plane] instance

$ kubectl get nodes
NAME            STATUS   ROLES           AGE    VERSION
control-plane   Ready    control-plane   7m5s   v1.27.0
node-1          Ready    <none>          91s    v1.27.0
node-2          Ready    <none>          17s    v1.27.0

Update the package index and find the latest kubeadm patch available in the repository

$ sudo apt update

$ sudo apt-cache madison kubeadm

Unhold the kubeadm package for upgrading

$ sudo apt-cache unhold kubeadm

Upgrade the kubeadm package to version v1.27.1 and hold the package from automatic upgrading


$ sudo apt install -y kubeadm=1.27.1-00

$ sudo apt-mark hold kubeadm

Check the upgraded kubeadm version and verify the upgrade plan

$ sudo kubeadm version

$ sudo kubeadm upgrade plan

Once our plan is verified, we can upgrade the [control-plane] by executing the below command

$ sudo kubeadm upgrade apply v1.27.1

Prepare the [control-plane] node for maintenance by marking it unschedulable and evicting the workloads

$ kubectl drain control-plane --ignore-daemonsets

Unhold the kubelet and kubectl packages for an upgrade

$ sudo apt-mark unhold kubelet kubectl

Upgrade kubelet and kubectl packages to version v1.27.1 and hold the packages from automatic upgrading

$ sudo apt install kubelet=1.27.1-00 kubectl=1.27.1-00

$ sudo apt-mark hold kubelet kubectl

Restart the kubelet on [control-plane]

$ sudo systemctl daemon-reload

$ sudo systemctl restart kubelet

Uncordon the [control-plane] for marking it as schedulable

$ kubectl uncordon control-plane

Verify the nodes, now we can see [control-plane] is upgraded to v1.27.1

$ kubectl get nodes
NAME            STATUS   ROLES           AGE    VERSION
control-plane   Ready    control-plane   118m   v1.27.1
node-1          Ready    <none>          112m   v1.27.0
node-2          Ready    <none>          111m   v1.27.0

Nodes

Update the package index on [node-1] and unhold the kubeadm package for upgrading

$ sudo apt update

$ sudo apt-mark unhold kubeadm

Upgrade the kubeadm package to version v1.27.1 on [node-1] and hold the package from automatic upgrading

$ sudo apt install kubeadm=1.27.1-00

$ sudo apt-mark hold kubeadm

Upgrade the cluster configuration on [node-1] using the below command

$ sudo kubeadm upgrade node

Prepare the [node-1] for maintenance by marking it unschedulable and evicting the workloads Execute the below drain command on [control-plane]

$ kubectl drain node-1 --ignore-daemonsets --force

Unhold the kubelet and kubectl packages for upgrade

$ sudo apt-mark unhold kubelet kubectl

Upgrade kubelet and kubectl packages to version v1.27.1 and hold the packages from automatic upgrading

$ sudo apt install kubelet=1.27.1-00 kubectl=1.27.1-00

$ sudo apt-mark hold kubelet kubectl

Restart the kubelet on [node-1]

$ sudo systemctl daemon-reload

$ sudo systemctl restart kubelet

Uncordon the [node-1] for marking it as schedulable Execute the below uncordon command on [control-plane]

$ kubectl uncordon node-1

Repeat the above steps for [node-2] also
Now our cluster is fully upgraded to v1.27.1

Verify the same by executing the below command on [control-plane]

$ kubectl get nodes
NAME            STATUS   ROLES           AGE    VERSION
control-plane   Ready    control-plane   136m   v1.27.1
node-1          Ready    <none>          131m   v1.27.1
node-2          Ready    <none>          129m   v1.27.1

Reference

https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/

https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/upgrading-linux-nodes/

Building a Kubernetes Cluster using kubeadm

Unni P — Wed, 03 May 2023 12:09:45 +0000

In this article, we will look how we can create a three node Kubernetes cluster using kubeadm

Introduction

kubeadm is a tool used to create Kubernetes clusters
It automates the creation of Kubernetes clusters by bootstrapping the control plane, joining the nodes etc
Follows Kubernetes release cycle
Open-source tool maintained by the Kubernetes community

Prerequisites

Create three Ubuntu 20.04 LTS instances [control-plane, node-1, node-2]
Each instance has a minimum specification of 2 CPU and 2 GB RAM
Networking must be enabled between instances
Required ports must be allowed between instances
Swap must be disabled on instances

Steps

Set up unique hostnames on all instances [control-plane, node-1, node-2] Once the hostnames are set, logout from the current session and log back in to reflect the changes

$ sudo hostnamectl set-hostname control-plane

$ sudo hostnamectl set-hostname node-1

$ sudo hostnamectl set-hostname node-2

Update the hosts file on all instances [control-plane, node-1, node-2] to enable communication via hostnames

$ sudo vi /etc/hosts

172.31.12.122 control-plane
172.31.7.52  node-1
172.31.10.184 node-2

Disable swap on all instances [control-plane, node-1, node-2] and if a swap entry is present in the fstab file then comment out the line

$ sudo swapoff -a

$ sudo vi /etc/fstab
  # comment out swap entry

Setup containerd as our container runtime on all instances [control-plane, node-1, node-2] and to do that, first we need to load some Kernel modules and modify system settings

$ cat << EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

$ sudo modprobe overlay

$ sudo modprobe br_netfilter

$ cat << EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

$ sudo sysctl --system

Once the Kernel modules are loaded and modified the system settings, then can we can install containerd on all instances [control-plane, node-1, node-2]

$ sudo apt update

$ sudo apt install -y containerd

Once installed, generate a default configuration file for contained on all instances [control-plane, node-1, node-2] and restart the service

$ sudo mkdir -p /etc/containerd

$ sudo containerd config default | sudo tee /etc/containerd/config.toml

$ sudo systemctl restart containerd

Install some prerequisite packages on all instances [control-plane, node-1, node-2] for configuring Kubernetes apt repository

$ sudo apt update

$ sudo apt install -y apt-transport-https ca-certificates curl

Download the Google Cloud public signing key and configure Kubernetes apt repository on all instances [control-plane, node-1, node-2]

$ sudo mkdir /etc/apt/keyrings

$ sudo curl -fsSLo /etc/apt/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

$ echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

Install kubeadm, kubelet and kubectl tools and hold their version on all instances [control-plane, node-1, node-2]

$ sudo apt update

$ sudo apt install -y kubeadm=1.27.0-00 kubelet=1.27.0-00 kubectl=1.27.0-00

$ sudo apt-mark hold kubeadm kubelet kubectl

Initialize the cluster by executing the below command on [control-plane] instance

$ sudo kubeadm init --pod-network-cidr 192.168.0.0/16 --kubernetes-version 1.27.0

Once the installation is completed, set up our access to the cluster on [control-plane] instance

$ mkdir -p $HOME/.kube

$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

Verify our cluster by listing the nodes on [control-plane] instance But our nodes are in NotReady state because we haven’t set up networking

$ kubectl get nodes
NAME            STATUS     ROLES           AGE   VERSION
control-plane   NotReady   control-plane   36s   v1.27.0

Install the Calico network addon on the [control-plane] instance and verify the status

$ kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.1/manifests/calico.yaml

$ kubectl get nodes
NAME            STATUS   ROLES           AGE     VERSION
control-plane   Ready    control-plane   2m51s   v1.27.0

Once the networking is enabled, join our workload nodes to the cluster Get the join command from the [control-plane] instance

$ kubeadm token create --print-join-command
kubeadm join 172.31.12.122:6443 --token 6t3r2c.5zzsgtekwwltofwt --discovery-token-ca-cert-hash sha256:2f1c4115125ec62af8ae6ab2648277ace3a625ebc0281e85ca8145e0e9077ee4

Once the join command is copied from the [control-plane] instance, execute them in the [node-1, node-2] instances

$ sudo kubeadm join 172.31.12.122:6443 --token 6t3r2c.5zzsgtekwwltofwt --discovery-token-ca-cert-hash sha256:2f1c4115125ec62af8ae6ab2648277ace3a625ebc0281e85ca8145e0e9077ee4

Verify our cluster from the [control-plane] instance, it should be in a READY state

$ kubectl get nodes
NAME            STATUS   ROLES           AGE    VERSION
control-plane   Ready    control-plane   7m5s   v1.27.0
node-1          Ready    <none>          91s    v1.27.0
node-2          Ready    <none>          17s    v1.27.0

Deploy an Nginx pod, expose it as ClusterIP from the [control-plane] instance and verify its status

$ kubectl run nginx --image=nginx --port=80 --expose
service/nginx created
pod/nginx created

$ kubectl get pod nginx -o wide
NAME    READY   STATUS    RESTARTS   AGE   IP               NODE     NOMINATED NODE   READINESS GATES
nginx   1/1     Running   0          30s   192.168.84.129   node-1   <none>           <none>

$ kubectl get svc nginx
NAME    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
nginx   ClusterIP   10.111.228.20   <none>        80/TCP    39s

Reference

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/

https://acloudguru.com/course/introduction-to-kubernetes

Kubernetes - Merging kubeconfig Files

Unni P — Wed, 03 May 2023 09:13:21 +0000

In this short article, we will look how we can merge a new kubeconfig file to an existing config file

Introduction

Kubernetes is an open-source container orchestration tool to manage containerized applications
An important component of Kubernetes is kubeconfig, a configuration file used to connect to our clusters
A kubeconfig file contains the cluster name and endpoint, user credentials and the context

Steps

Make a copy of your existing kubeconfig file

$ cp ~/.kube/config ~/.kube/config.bak

Merge the old and new kubeconfig files using the below command

$ KUBECONFIG=~/.kube/config:~/new-kubeconfig.yml \
             kubectl config view --flatten > ~/merged-kubeconfig.yml

Replace your old config with new merged config file

$ cp ~/merged-kubeconfig.yml ~/.kube/config

List the contexts, now you can see our new cluster

$ kubectl config get-contexts

Reference

https://jacobtomlinson.dev/posts/2019/how-to-merge-kubernetes-kubectl-config-files/

kind - Setting up CNI using Calico - Part 7

Unni P — Wed, 03 May 2023 07:39:01 +0000

In this article we will look how we can configure our cluster to use Calico as CNI

Introduction

kind ships with a simple networking implementation called kindnetd
Based on standard CNI plugins (ptp, host-local) and simple netlink routes
It also handles IP masquerade
We can disable the default CNI in kind and use Calico as our CNI

Usage

Create a cluster using the below configuration file

$ cat kind.yml 
apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
name: dev
networking:
  disableDefaultCNI: true
nodes:
- role: control-plane
- role: worker
- role: worker

$ kind create cluster --config kind.yml 
Creating cluster "dev" ...
 ✓ Ensuring node image (kindest/node:v1.26.3) 🖼
 ✓ Preparing nodes 📦 📦 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing StorageClass 💾 
 ✓ Joining worker nodes 🚜 
Set kubectl context to "kind-dev"
You can now use your cluster with:

kubectl cluster-info --context kind-dev

Since we don’t have any CNI installed in the cluster and we can see the nodes are in NotReady state and CoreDNS pods are in pending state

$ kubectl get nodes
NAME                STATUS     ROLES           AGE   VERSION
dev-control-plane   NotReady   control-plane   65s   v1.26.3
dev-worker          NotReady   <none>          48s   v1.26.3
dev-worker2         NotReady   <none>          35s   v1.26.3

$ kubectl -n kube-system get pods
NAME                                        READY   STATUS    RESTARTS   AGE
coredns-787d4945fb-c5xfr                    0/1     Pending   0          115s
coredns-787d4945fb-mlphp                    0/1     Pending   0          115s
etcd-dev-control-plane                      1/1     Running   0          2m8s
kube-apiserver-dev-control-plane            1/1     Running   0          2m8s
kube-controller-manager-dev-control-plane   1/1     Running   0          2m7s
kube-proxy-74mrj                            1/1     Running   0          114s
kube-proxy-txj8v                            1/1     Running   0          101s
kube-proxy-xrqnn                            1/1     Running   0          115s
kube-scheduler-dev-control-plane            1/1     Running   0          2m8s

Install Calico CNI using the manifest file available from their documentation

$ kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.1/manifests/calico.yaml
poddisruptionbudget.policy/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
serviceaccount/calico-node created
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
deployment.apps/calico-kube-controllers created

Verify the status of the nodes and pods

$ kubectl get nodes
NAME                STATUS   ROLES           AGE     VERSION
dev-control-plane   Ready    control-plane   6m12s   v1.26.3
dev-worker          Ready    <none>          5m55s   v1.26.3
dev-worker2         Ready    <none>          5m42s   v1.26.3

$ kubectl -n kube-system get pods
NAME                                        READY   STATUS    RESTARTS   AGE
calico-kube-controllers-5857bf8d58-4kcbd    1/1     Running   0          4m14s
calico-node-8qrmb                           1/1     Running   0          4m14s
calico-node-v9mrj                           1/1     Running   0          4m14s
calico-node-vml2c                           1/1     Running   0          4m14s
coredns-787d4945fb-c5xfr                    1/1     Running   0          7m58s
coredns-787d4945fb-mlphp                    1/1     Running   0          7m58s
etcd-dev-control-plane                      1/1     Running   0          8m11s
kube-apiserver-dev-control-plane            1/1     Running   0          8m11s
kube-controller-manager-dev-control-plane   1/1     Running   0          8m10s
kube-proxy-74mrj                            1/1     Running   0          7m57s
kube-proxy-txj8v                            1/1     Running   0          7m44s
kube-proxy-xrqnn                            1/1     Running   0          7m58s
kube-scheduler-dev-control-plane            1/1     Running   0          8m11s

Deploy our Nginx application by creating a pod and exposing it as ClusterIP

$ kubectl run nginx --image=nginx --port=80 --expose
service/nginx created
pod/nginx created

$ kubectl get pods nginx
NAME    READY   STATUS    RESTARTS   AGE
nginx   1/1     Running   0          32s

$ kubectl get svc nginx
NAME    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
nginx   ClusterIP   10.96.106.155   <none>        80/TCP    34s

Verify our Nginx application

$ kubectl run busybox --image=busybox --restart=Never --rm -it -- wget -O- http://nginx
If you don't see a command prompt, try pressing enter.
warning: couldn't attach to pod/busybox, falling back to streaming logs: Internal error occurred: error attaching to container: failed to load task: no running task found: task 9268947ec3741ac1bad25fab9454c9c56e51131e7d65098993a87a96ed7ea7d7 not found: not found
Connecting to nginx (10.96.106.155:80)
writing to stdout
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
-                    100% |********************************|   615  0:00:00 ETA
written to stdout
pod "busybox" deleted

Cleanup

Delete our cluster after use

$ kind delete cluster --name dev
Deleting cluster "dev" ...
Deleted nodes: ["dev-control-plane" "dev-worker" "dev-worker2"]

Reference

https://kind.sigs.k8s.io/

https://docs.tigera.io/calico/latest/getting-started/kubernetes/self-managed-onprem/onpremises

kind - Setting up Load Balancer using MetalLB - Part 6

Unni P — Wed, 03 May 2023 07:37:23 +0000

In this article we will look how we can get service type LoadBalancer in our cluster using MetalLB

Introduction

MetalLB provides a network load balancer implementation in our cluster
Allows you to create Kubernetes services of type LoadBalancer in clusters that don’t run on a cloud provider
Sets up MetalLB using layer2 protocol
We can send traffic directly to the load balancer’s external IP if the IP space is within the Docker IP space

Usage

Create a simple cluster using the below configuration file

$ cat kind.yml 
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: dev
nodes:
- role: control-plane
- role: worker
- role: worker

$ kind create cluster --config kind.yml 
Creating cluster "dev" ...
 ✓ Ensuring node image (kindest/node:v1.26.3) 🖼
 ✓ Preparing nodes 📦 📦 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
 ✓ Joining worker nodes 🚜 
Set kubectl context to "kind-dev"
You can now use your cluster with:

kubectl cluster-info --context kind-dev

$ kubectl get nodes
NAME                STATUS   ROLES           AGE   VERSION
dev-control-plane   Ready    control-plane   68s   v1.26.3
dev-worker          Ready    <none>          37s   v1.26.3
dev-worker2         Ready    <none>          37s   v1.26.3

Deploy MetalLB using the default manifests and verify the components are up and running

$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.7/config/manifests/metallb-native.yaml
namespace/metallb-system created
customresourcedefinition.apiextensions.k8s.io/addresspools.metallb.io created
customresourcedefinition.apiextensions.k8s.io/bfdprofiles.metallb.io created
customresourcedefinition.apiextensions.k8s.io/bgpadvertisements.metallb.io created
customresourcedefinition.apiextensions.k8s.io/bgppeers.metallb.io created
customresourcedefinition.apiextensions.k8s.io/communities.metallb.io created
customresourcedefinition.apiextensions.k8s.io/ipaddresspools.metallb.io created
customresourcedefinition.apiextensions.k8s.io/l2advertisements.metallb.io created
serviceaccount/controller created
serviceaccount/speaker created
role.rbac.authorization.k8s.io/controller created
role.rbac.authorization.k8s.io/pod-lister created
clusterrole.rbac.authorization.k8s.io/metallb-system:controller created
clusterrole.rbac.authorization.k8s.io/metallb-system:speaker created
rolebinding.rbac.authorization.k8s.io/controller created
rolebinding.rbac.authorization.k8s.io/pod-lister created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:controller created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:speaker created
secret/webhook-server-cert created
service/webhook-service created
deployment.apps/controller created
daemonset.apps/speaker created
validatingwebhookconfiguration.admissionregistration.k8s.io/metallb-webhook-configuration created

$ kubectl -n metallb-system get pods
NAME                          READY   STATUS    RESTARTS   AGE
controller-577b5bdfcc-p7sb5   1/1     Running   0          76s
speaker-cgmm4                 1/1     Running   0          76s
speaker-gwfqr                 1/1     Running   0          76s
speaker-jk684                 1/1     Running   0          76s

As we said earlier in the introduction part, we are using the layer2 protocol of MetalLB. For completing the layer2 configuration, we need to provide MetalLB a range of IP addresses it controls. This IP address range needs to be in the Docker kind network.

$ docker network inspect -f '{{.IPAM.Config}}' kind
[{172.18.0.0/16  172.18.0.1 map[]} {fc00:f853:ccd:e793::/64  fc00:f853:ccd:e793::1 map[]}]

Now we want our load balancer IP range to come from this subclass and we can configure MetalLB to use 172.19.255.200 to 172.19.255.250 by creating IPAddressPool and L2Advertisement resources.
Create the necessary MetalLB resources using the below manifest file.

$ cat metallb.yml 
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: kind
  namespace: metallb-system
spec:
  addresses:
  - 172.18.255.200-172.18.255.250

---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: kind
  namespace: metallb-system
spec:
  ipAddressPools:
  - kind

$ kubectl apply -f metallb.yml 
ipaddresspool.metallb.io/kind unchanged
l2advertisement.metallb.io/kind created

$ kubectl -n metallb-system get ipaddresspools
NAME   AUTO ASSIGN   AVOID BUGGY IPS   ADDRESSES
kind   true          false             ["172.18.255.200-172.18.255.250"]

$ kubectl -n metallb-system get l2advertisements
NAME   IPADDRESSPOOLS   IPADDRESSPOOL SELECTORS   INTERFACES
kind   ["kind"]

Deploy our Application

Create an Nginx pod using the below manifest file and verify its status

$ cat nginx.yml 
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: nginx
  name: nginx
spec:
  containers:
  - image: nginx
    name: nginx
    ports:
    - containerPort: 80

$ kubectl apply -f nginx.yml 
pod/nginx created

$ kubectl get pods nginx
NAME    READY   STATUS    RESTARTS   AGE
nginx   1/1     Running   0          23s

Expose the Nginx pod as a LoadBalancer service using the below manifest file

$ cat nginx-loadbalancer.yml 
apiVersion: v1
kind: Service
metadata:
  labels:
    run: nginx
  name: nginx
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

$ kubectl apply -f nginx-loadbalancer.yml 
service/nginx created

Check the created nginx service and we can see an IP address in the EXTERNAL-IP section

$ kubectl get svc nginx 
NAME    TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)        AGE
nginx   LoadBalancer   10.96.43.161   172.18.255.200   80:30433/TCP   30s

Access the application using external IP and port

$ curl http://172.18.255.200:80
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Cleanup

Delete the cluster after use

$ kind delete cluster --name dev
Deleting cluster "dev" ...
Deleted nodes: ["dev-worker2" "dev-control-plane" "dev-worker"]

Reference

https://kind.sigs.k8s.io/

https://metallb.universe.tf/

DEV Community: Unni P

Prometheus - HTTPS & Authentication - Part 4

Prerequisites

Prometheus - Installation on Amazon EC2 (Ubuntu) - Part 3

HTTPS

Node Exporter

Prometheus

Authentication

Node Exporter

Prometheus

Reference

Prometheus - Installation on Amazon EC2 (Ubuntu) - Part 3

Prometheus - Quick Setup on Amazon EC2 (Ubuntu) - Part 2

Prerequisites

Installation

Prometheus

Node Exporter

Configuration

Prometheus

Reference

Infisical - Open Source SecretOps - Kubernetes Setup

Introduction

Features

Prerequisites

Installation

Initial Setup

Configuration

Secrets Operator Setup

Application Deployment

Reference

Uptime Kuma on Amazon EC2 (Ubuntu) - An Open-source Monitoring Tool

Introduction

Prerequisites

Installation

Configuration

Reference

Setting up IBM Db2 Community Edition on Amazon EC2 (Ubuntu)

Introduction

Prerequisites

Installation

Post Installation

Reference

Prometheus - Quick Setup on Amazon EC2 (Ubuntu) - Part 2

Prerequisites

Installation

Prometheus

Node Exporter

Reference

Prometheus - Architecture - Part 1

Introduction

Features

Architecture

Prometheus Server

Exporters

Alertmanager

Pushgateway

Service Discovery

Client Libraries

Reference

Upgrading a Kubernetes Cluster using kubeadm

Prerequisites

Building a Kubernetes Cluster using kubeadm

Unni P ・ May 3 ・ 4 min read

Steps

Control Plane

Nodes

Reference

Building a Kubernetes Cluster using kubeadm

Introduction

Prerequisites

Steps

Reference

Kubernetes - Merging kubeconfig Files

Introduction

Steps

Reference

kind - Setting up CNI using Calico - Part 7

Introduction

Usage

Cleanup