DEV Community: tj_27

Export Data from MySQL

tj_27 — Sat, 07 Sep 2024 09:08:14 +0000

Okay, because I need to remember how would I export data from MySQL, I need to leave it here.

# export
mysql --defaults-file=/data/db/mysql3366/my.cnf -S /data/db/mysql3366/mysql3366.sock -u [user] -p < select.sql | sed 's/\t/", "/g;s/^/"/;s/$"/;s/\n//g' > ./output_file.csv

# translate
sed -i '1s/^/\xEF\xBB\xBF' ./output_file.csv

Installing Grafana for Monitoring 😎

tj_27 — Thu, 05 Sep 2024 06:31:13 +0000

Grafana is a powerful open-source platform for monitoring and observability that allows you to visualize and analyze data from various sources in real time. Here are a few key reasons why you should consider installing Grafana:

1. Centralized Data Visualization

Grafana supports a wide range of data sources, including databases (like MySQL, PostgreSQL), time-series databases (such as Prometheus, InfluxDB), and cloud-based sources. By integrating these sources, you can create unified dashboards to visualize data from multiple systems in one place.

2. Real-Time Monitoring and Alerts

Grafana provides real-time monitoring capabilities with customizable dashboards. You can set up alerts based on certain thresholds or conditions, and Grafana can notify you via email, Slack, or other channels, allowing you to respond to incidents quickly.

3. Customizable and Flexible Dashboards

Grafana offers highly customizable dashboards, allowing you to create and modify graphs, heatmaps, and other visualization tools to suit your specific needs. You can design tailored dashboards to track the metrics that are most important to your organization.

4. Ease of Use

With its intuitive interface and rich set of features, Grafana is relatively easy to set up and use, even for those who are not deeply familiar with data visualization or monitoring tools. It also has a large community and comprehensive documentation to support new users.

5. Performance and Scalability

Grafana is built to handle large-scale data and is optimized for performance. It can be used in environments ranging from small personal projects to large enterprise-level applications, scaling effectively as your needs grow.

6. Open Source and Extensible

Being an open-source tool, Grafana allows you to benefit from community-contributed plugins and integrations. You can also create custom plugins to meet unique visualization needs.

7. Enhance Observability and Decision Making

Grafana helps you gain deep insights into your system’s performance, detect anomalies, and understand trends, which enhances observability and enables better decision-making for IT operations and business strategies.

Installing Grafana is a great step if you aim to improve monitoring capabilities, consolidate data visualization, or simply gain a clearer understanding of your system's performance.

Steps

Configure the proxy settings to enable access, then proceed to download the Grafana latest release from the official source.

sudo yum install -y https:dl.grafana.com/oss/releases/grafana-10.0.0-1.x86_64.rpm

Start and verify the service.

systemctl start grafana-server.service
systemctl status grafana-server.service

Install Prometheus that will serve as the data source.
download the package

wget https://github.com/prometheus/prometheus/releases/download/v2.49.1/prometheus-2.49.1.linux-amd64.tar.gz

extract the file

tar -xzvf prometheus-2.49.1.linux-amd64.tar.gz

make it a soft link

ln -s prometheus-2.49.1.linux-amd64 prometheus

create a prometheus sercive file

vim /usr/lib/systemd/system/prometheus.service

prometheus.service

[Unit]
Description=prometheus
After=network.target

[Service]
Type=simple
PIDFile=/var/run/prometheus.pid
WorkingDirectory=/usr/local/prometheus
ExecStart=/usr/local/prometheus/prometheus --web.listen=09090 --storage.tsdb.path=/data/prometheus --web.enable-admin-api --storage.tsdb.retention=7d
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=true
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

start the service and verify

systemctl start prometheus.service; systemctl status prometheus.service

To log in to your grafana on web, follow the instruction from the official documentation.

https://grafana.com/docs/grafana/latest/setup-grafana/sign-in-to-grafana/

To connect the data source, follow the steps on the official documentation as well.

https://grafana.com/docs/grafana/latest/datasources/prometheus/

References:

https://grafana.com/docs/grafana/latest/setup-grafana/installation/

Generated OTP using python3

tj_27 — Tue, 03 Sep 2024 06:53:13 +0000

Since we were not allowed to have MFA together with the same application we are using, I got an idea from my leader to just have a generated OTPs.

import time
import hmac
import base64
import struct
import hashlib

def generate_otp(secret, interval=30, digits=6):
    # Decode the base32 encoded secret
    key = base64.b32decode(secret)

    # Calculate the number of time intervals since the Unix epoch
    counter = int(time.time()) // interval

    # Convert the counter to bytes
    msg= struct.pack(">Q", counter)

    # Create the HMAC hash using SHA-1
    hmac_hash = hmac.new(key, msg, hashlib.sha1).digest()

    # Get the dynamic offset from the last nibble of the hash
    offset = hmac_hash[-1] & 0x0F

    # Extract a 4-byte dynamic binary code form the hash using the offset
    code = struct.unpack(">I", hmac_hash[offset:offset + 4]) [0] & 0x7FFFFFFF

    # Modulus operation to get the final OTP
    otp = code % (10 ** digits)

    return str(otp).zfill(digits)

# Lists for secret keys and remarks
secret_keys_with_titles = {
    "[name]": "[secret_key]",
    "VAR": "ABDCDEFGHIJKLMNO",
    "VER": "PQRSTUVWXYZ12345",
    "VIR": "6789101112131415"
}

# ANSI escape codes for different color
colors = [
    "\033[91m", # Red
    "\033[92m", # Green
    "\033[94m", # Blue
    "\033[93m", # Yellow
    "\033[96m", # Cyan
]
print()
# Generate the OTPs for all secret keys
otps = {title: generate_otp(secret) for title, secret in secret_keys_with_titles.items()}

# Output the OTPs in horizontal way
output = []
for i, (title, otp) in enumerate(otps.items()):
    color = colors[i % len(colors)] # Rotate through colors
    output.append(f"{color}{title}: {otp}\033[0m") #\033[0m resets the color

print(" | ".join(output))
print()

So, this is how it look on your terminal:

Installing Prometheus Bash Script

tj_27 — Thu, 15 Aug 2024 10:41:00 +0000

Installing monitors like Grafana would be very helpful for us. As one of the most used datasource, I decided to make a script on how to install prometheus.

#!/bin/bash
# Installing Prometheus

if [ -f "/usr/local/prometheus" ]; then
    echo "Prometheus already exists ..."
    exit
fi


# tput commands
CLEAR="tput clear"
DOWN="tput cud1"
BOLD="tput bold"
NORMAL="tput sgr0"
BLACK="tput setaf 0"
RED="tput setaf 1"
GREEN="tput setaf 2"
YELLOW="tput setaf 3"
BLUE="tput setaf 4"


$CLEAR

printf "Installing Prometheus\n\n"
# Curling Google to check if connected to a network
printf "Looling for a network...\n\n"
if curl google.com > /dev/null; then
    $DOWN
    $YELLOW
    printf "Network connected.\n\n"
    $NORMAL
else
    $DOWN
    printf "The server is not connected to the network. Please connect and try again.\n\n";
    exit 1
fi

$DOWN

echo -n "Insert the version you would like to be installed, default is [ 2.49.1 ] : "
$BOLD
$BLUE
read VERSION
$NORMAL
VERSION=${VERSION:-2.49.1}
$DOWN
$NORMAL
if [ ! -f "prometheus-$VERSION.linux-amd64.tar.gz" ]; then
    # Download prometheus
    wget https://github.com/prometheus/prometheus/releases/download/v$VERSION/prometheus-$VERSION.linux-amd64.tar.gz -P /opt
fi

# extract the file
tar -xzvf /opt/prometheus-$VERSION.linux-amd64.tar.gz -C /usr/local
# make it s soft link
ln -s /usr/local/prometheus-$VERSION.linux-amd64 /usr/local/prometheus

# create a prometheus service file
cat >/usr/lib/systemd/system/prometheus.service<<EOF
[Unit]
Description=prometheus
After=network.target

[Service]
Type=simple
PIDFile=/var/run/prometheus.pid
WorkingDirectory=/usr/local/prometheus/prometheus --web.listen-address=0.0.0.0:9090 --storage.tsdb.path=/data/prometheus --web.enable-admin-api --storage.tsdb.retention=7d
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=true
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF
$DOWN
# start the prometheus
sudo systemctl start prometheus.service
# check the status
sudo systemctl status prometheus.service

# see the listening sockets
netstat -tnlp |grep 9090
$DOWN
if sudo systemctl is-active --quiet prometheus.service; then
    $DOWN
    $BOLD
    $YELLOW
    printf "=========================================\n"
    $GREEN
    printf "Prometheus installed successfully!\n"
    $NORMAL
    $BOLD
    $YELLOW
    printf "=========================================\n"
    $NORMAL
    $DOWN
else
    $DOWN
    $RED
    printf "Prometheus installation failed.\n\n"
    $NORMAL
    $DOWN
fi

Installing Elasticsearch exporter Bash Script

tj_27 — Wed, 31 Jul 2024 06:50:06 +0000

Here we go again! Down is a script for installing elasticsearch exporter

#!/bin/bash
# Installing Elasticsearch Exporter

# tput commands
CLEAR="tput clear"
DOWN="tput cud1"
BOLD="tput bold"
NORMAL="tput sgr0"
BLACK="tput setaf 0"
RED="tput setaf 1"
GREEN="tput setaf 2"
YELLOW="tput setaf 3"
BLUE="tput setaf 4"

$CLEAR
$DOWN
# Installation confirmation
printf "You have selected to install elasticsearch_exporter.\n\n"
read -p "Do you want to continue? [ yes / no ] : " USER_INPUT
USER_INPUT=${USER_INPUT:-yes}
$DOWN

# Convert user's choice to lowercase for case-sensitive comparison
USER_INPUT_LOWER=$(echo "$USER_INPUT" | tr '[:upper:]' '[:lower:]')

# Check the user's input
if [ "$USER_INPUT_LOWER" == "yes" ]; then
    $YELLOW
    printf "Elasticsearch exporter installation confirmed.\n\n"
    $NORMAL
else
    printf "Elasticsearch exporter installation cancelled.\n\n"
    exit
fi

# Specify the name of the systemd service
SERVICE_NAME="elasticsearch_exporter"

# Check if the service file exists
if [ -e "/usr/lib/systemd/system/$SERVICE_NAME.service" ]; then
    # Check if the service is active
    if sudo systemctl is-active --quiet "$SERVICE_NAME"; then
        $BOLD
        printf "There is an active $SERVICE_NAME. \n\n"
        $NORMAL
        # Check the version of the active elasticsearch_exporter
        elasticsearch_exporter_PATH="/usr/local/$SERVICE_NAME/$SERVICE_NAME"
        VERSION_INFO="$($elasticsearch_exporter_PATH --version 2>&1 | awk '/elasticsearch_exporter/ {print $3}')"
        $GREEN
        printf "Active Node ExporterVersion: $VERSION_INFO \n\n"
        $NORMAL
        printf "Do you want to remove it and replace with a new one? [ 1 / 2 ]\n\n"
        printf " 1: Remove the active elasticsearch_exporter and replace it with a new one. \n\n"
        printf " 2: Don't do anything and exit.\n\n"
        read -rp "> " ACTION
        # Check the action to do
        if [ -z "$ACTION" ]; then
            printf "Removing all elasticsearch_exporter files... \n\n"
            # Remove elasticsearch_exporter related files
            sudo systemctl stop $SERVICE_NAME
            sudo systemctl disable $SERVICE_NAME
            sudo rm /usr/lib/systemd/system/$SERVICE_NAME.service
            sudo rm -rf /usr/local/elasticsearch_exporter*
            $YELLOW
            printf "Related files removed.\n\n"
            $NORMAL
            printf "Installation will continue...\n\n"
        elif [ "$ACTION" -eq 1 ]; then
            printf "Removing all elasticsearch_exporter files... \n\n"
            # Remove elasticsearch_exporter related files
            sudo systemctl stop $SERVICE_NAME
            sudo systemctl disable $SERVICE_NAME
            sudo rm /usr/lib/systemd/system/$SERVICE_NAME.service
            sudo rm -rf /usr/local/elasticsearch_exporter*
            $YELLOW
            printf "Related files removed.\n\n"
            $NORMAL
            printf "Installation will continue...\n\n"
        elif [ "$ACTION" -eq 2 ]; then
            $DOWN
            printf "No action done.\n\n"
            exit
        else
            printf "Invalid input. Please enter 1 or 2.\n\n"
            exit 1
        fi
    else
        printf "There's a $SERVICE_NAME service that is not active. Removing related files...\n\n"
        sudo systemctl stop $SERVICE_NAME
        sudo systemctl disable $SERVICE_NAME
        sudo rm /usr/lib/systemd/system/$SERVICE_NAME.service
        sudo rm -rf /usr/local/elasticsearch_exporter*
        $YELLOW
        printf "Related files removed.\n\n"
        $NORMAL
        printf "Installation will continue...\n\n"
    fi
else
    printf "No $SERVICE_NAME service file found.\n\n"
    fi

# Curling Google to check if connected to a network
printf "Looking for a network...\n\n"
if curl google.com > /dev/null; then
    $DOWN
    $YELLOW
    printf "Network connected.\n\n"
    $NORMAL
else
    $DOWN
    printf "The server is not connected to the network. Please connect and try again.\n\n";
    exit 1
fi

echo -n "Insert the version you would like to be installed, default is [ 1.3.0 ] : "
$BOLD
$BLUE
read VERSION
$NORMAL
VERSION=${VERSION:-1.3.0}
$DOWN
$NORMAL
# Download the file
wget https://github.com/prometheus-community/elasticsearch_exporter/releases/download/v$VERSION/elasticsearch_exporter-$VERSION.linux-amd64.tar.gz -O /usr/local/elasticsearch_exporter.tgz
# Extract the downloaded tarball in user directory with a new name
tar -xzvf /usr/local/elasticsearch_exporter.tgz -C /usr/local && mv /usr/local/elasticsearch_exporter-$VERSION.linux-amd64 /usr/local/elasticsearch_exporter

# Create a systemd service file for Node Exporter
cat >/usr/lib/systemd/system/elasticsearch_exporter.service<<EOF
[Unit]
Description= Elasticsearch Exporter
Documentation= https://prometheus.io/
After=network.target

[Service]
Type=simple
PIDFile=/usr/local/elasticsearch_exporter/elasticsearch_exporter.pid
ExecStart=/usr/local/elasticsearch_exporter/elasticsearch_exporter --es.all --es.clusterinfo.interval=15m --es.uri=http://elastic:xxpasswordxx@xxipxx:9200 --es.timeout 30s
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=true

[Install]
WantedBy=multi-user.target
EOF

# IP
ip=$(hostname -I |awk '{print $1}')
$DOWN
$YELLOW 
# Password
echo -n "Please insert the password of your elastic : "
$NORMAL
$DOWN
read password
$DOWN

# Edit the service file in-place
sed -i "s#xxpasswordxx#$password#g" /usr/lib/systemd/system/elasticsearch.service
sed -i "s#xxipxx#$ip#g" /usr/lib/systemd/system/elasticsearch.service

# Reload systemd and start Elasticsearch Exporter
sudo systemctl daemon-reload
sudo systemctl start elasticsearch_exporter.service
sudo systemctl status elasticsearch_exporter.service

# Cleanup downloaded file
rm -f /usr/local/$SERVICE_NAME.tgz

$DOWN
if sudo systemctl is-active --quiet "$SERVICE_NAME"; then
    $DOWN
    $BOLD
    $YELLOW
    printf "======================================\n"
    $GREEN
    printf "Elasticsearch Exporter installed successfully!\n"
    $NORMAL
    $BOLD
    printf "Version: $VERSION\n"
    $YELLOW
    printf "======================================\n"
    $NORMAL
    $DOWN
else
    $DOWN
    $RED
    printf "Elasticsearch Exporter installation failed.\n\n"
    $NORMAL
    $DOWN
fi

prometheus.yml file for Consul

tj_27 — Thu, 25 Jul 2024 10:18:13 +0000


global:
    scrape_insterval: 15s
    evaluation_interval: 15s

scrapre_configs:
    - job_name: 'consul'
    consul_sd_configs:
        - server: 'localhost:8500'
          service: []
    relabel_configs:
        - source_labels: [__meta_consul_tags]
          regex: .*,prometheus,.*
          action: keep
        - source_labels: [__meta_consul_service]
          targe_label: job
        - source_labels: [__meta_consul_service_metadaata_project]
          target_lable: project
        - source_labels: [__meta_consul_service_metadata_hostname]
          target_label: hostname
        - source_labels: [__meta_consul_service_metadaata_exporter]
          target_label: exporter
        - source_labels: [__meta_consul_service_metadaata_app]
          target_label: app
        - source_labels: [__meta_consul_service_metadaata_groupname]
          target_label: groupname
        - source_labels: [__meta_consul_service_metadaata_tag]
          target_label: tag
        - source_labels: [__meta_consul_service_address, __meta_consul_service_port]
          separator: ':'
          target_label: instance
          regex: '(.+):(.+)'
          replacement: '$1:+$2'

Installing Consul Bash Script

tj_27 — Tue, 23 Jul 2024 09:16:33 +0000

Hello there! It's been a while! I want to share with you my script to install consul.

As we know that consul is very useful and an important tool in modern software development and operations, primarily known for its role as a service mesh and service discovery solution.

I work for "I don't know how many" days for this, since we will be conducting a new monitoring project. It is a bit challenging for me since this one is a new thing for me.

Anyways, here's the script.

#!/bin/bash
# Installing Consul

# tput commands
CLEAR="tput clear"
DOWN="tput cud1"
BOLD="tput bold"
NORMAL="tput sgr0"
BLACK="tput setaf 0"
RED="tput setaf 1"
GREEN="tput setaf 2"
YELLOW="tput setaf 3"
BLUE="tput setaf 4"

$CLEAR
$DOWN
# Installation confirmation
printf "You have selected to install consul.\n\n"
read -p "Do you want to continue? [ yes / no ] : " USER_INPUT
USER_INPUT=${USER_INPUT:-yes}
$DOWN

# Convert user's choice to lowercase for case-sensitive comparison
USER_INPUT_LOWER=$(echo "$USER_INPUT" | tr '[:upper:]' '[:lower:]')

# Check the user's input
if [ "$USER_INPUT_LOWER" == "yes" ]; then
    $YELLOW
    printf "Consul installation confirmed.\n\n"
    $NORMAL
else
    printf "Consul installation cancelled.\n\n"
    exit
fi


# Specify the name of the systemd service
SERVICE_NAME="consul"

# Check if the service file exists
if [ -e "/usr/lib/systemd/system/$SERVICE_NAME.service" ]; then
    # Check if the service is active
    if sudo systemctl is-active --quiet "$SERVICE_NAME"; then
        $BOLD
        printf "There is an active $SERVICE_NAME. \n\n"
        $NORMAL
        # Check the version of the active node_exporter
        CONSUL_PATH="/usr/bin/$SERVICE_NAME"
        VERSION_INFO="$($CONSUL_PATH --version 2>&1 | awk '/Consul/ {print $2}')"
        $GREEN
        printf "Active Consul Version: $VERSION_INFO \n\n"
        $NORMAL
        printf "Do you want to remove it and replace with a new one? [ 1 / 2 ]\n\n"
        printf " 1: Remove manually the active node_exporter and replace it with a new one. \n\n"
        printf " 2: Don't do anything and exit.\n\n"
        read -rp "> " ACTION
        # Check the action to do
        $DOWN
        if [ -z "$ACTION" ]; then
            printf "Remove the active consul manually. \n\n"
            # Printing the suggested commands to remove the service.
            $YELLOW
            $BOLD
            printf "==========================================\n"
            $NORMAL
            printf "systemctl stop consul.service\n" 
            printf "rm -rf /data/consul\n" # data directory
            printf "sudo yum -y remove consul.x86_64\n"
            printf "rm -f /usr/bin/consul\n" # executable file
            printf "rm -f /usr/lib/systemd/system/consul.*\n" # service file
            $YELLOW
            $BOLD
            printf "==========================================\n\n"
            $NORMAL
            exit
        elif [ "$ACTION" -eq 1 ]; then
            printf "Remove the active consul manually. \n\n"
            # Printing the suggested commands to remove the service.
            $YELLOW
            $BOLD
            printf "==========================================\n"
            $NORMAL
            printf "systemctl stop consul.service\n" 
            printf "rm -rf /data/consul\n" # data directory
            printf "sudo yum -y remove consul.x86_64\n"
            printf "rm -f /usr/bin/consul\n" # executable file
            printf "rm -f /usr/lib/systemd/system/consul.*\n" # service file
            $YELLOW
            $BOLD
            printf "==========================================\n\n"
            $NORMAL
            exit
        elif [ "$ACTION" -eq 2 ]; then
            $DOWN
            printf "No action done.\n\n"
            exit
        else
            printf "Invalid input. Please enter 1 or 2.\n\n"
            exit 1
        fi
    else
        printf "There's a $SERVICE_NAME service that is not active. Remove the files manually.\n\n"
        $YELLOW
        $BOLD
        printf "==========================================\n"
        $NORMAL
        printf "systemctl stop consul.service\n" 
        printf "rm -rf /data/consul\n" # data directory
        printf "sudo yum -y remove consul.x86_64\n"
        printf "rm -f /usr/bin/consul\n" # executable file
        printf "rm -f /usr/lib/systemd/system/consul.*\n" # service file
        $YELLOW
        $BOLD
        printf "==========================================\n\n"
        $NORMAL
        exit
    fi
else
    printf "No $SERVICE_NAME service file found.\n\n"
    fi

# Curling Google to check if connected to a network
printf "Looking for a network...\n\n"
if curl google.com > /dev/null; then
    $DOWN
    $YELLOW
    printf "Network connected.\n\n"
    $NORMAL
else
    $DOWN
    printf "The server is not connected to the network. Please connect and try again.\n\n";
    exit 1
fi

# Download the file
if [ ! -f "/opt/consul_1.19.1_darwin_amd64.zip" ]; then
    wget https://releases.hashicorp.com/consul/1.19.1/consul_1.19.1_darwin_amd64.zip -P /opt
fi

# Extract the file
unzip /opt/consul_1.19.1_darwin_amd64.zip && mv consul /usr/bin

if [ ! -f "/etc/yum.repos.d/hashicorp.repo" ]; then
    cat >/etc/yum.repos.d/hashicorp.repo <<EOF
[hashicorp]
name=HashiCorp Stable - $basearch
baseurl=https://rpm.releases.hashicorp.com/RHEL/$releasever/$basearch/stable
enabled=1
gpgcheck=1
gpgkey=https://rpm.releases.hashicorp.com/gpg
EOF
fi

# Create a data directory
mkdir -p /data/consul

# Install
sudo yum -y install consul

# Modifying the default data directory
sed -i "s/\/opt\/consul/\/data\/consul/g" /etc/consul.d/consul.hcl;

IP=$(hostname -I | awk '{print $1}')

# Creating the configuration file
cat >/etc/consul.d/config.json <<EOF
{
    "datacenter": "dba",
    "data_dir": "/data/consul",
    "log_level": "INFO",
    "node_name": "dba-manager",
    "server": true,
    "bootstrap_expect": 1,
    "ui": true,
    "client_addr": "0.0.0.0",
    "bind_addr": "$IP"
}
EOF

USERNAME="consul"
# Check if the user exists
if id "$USERNAME" &>/dev/null; then
    echo
else
    sudo useradd "$USERNAME"
    printf "User '$USERNAME' has been created.\n\n"
fi
# Change the ownership
chown -R consul. /data/consul
chown -R consul. /etc/consul.d

# Service file content
SERVICE_CONTENT="[Unit]
Description=Consul
Documentation=https://www.consul.io/
Requires=network-online.target
After=network-online.target

[Service]
ExecStart=/usr/bin/consul agent -config-dir=/etc/consul.d/
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/usr/bin/consul leave
KillMode=process
Restart=on-failure

[Install]
WantedBy=multi-user.target"

TARGET_FILE="/usr/lib/systemd/system/consul.service"
# Replacing the default service file
echo "$SERVICE_CONTENT" | sudo tee $TARGET_FILE > /dev/null

# System reload, start, and status check
sudo systemctl daemon-reload
sudo systemctl start consul
sudo systemctl status consul

$DOWN

# Removing the file in /opt directory
if sudo systemctl is-active --quiet consul.service; then
    $DOWN
    $BOLD
    $YELLOW
    printf "======================================\n"
    $GREEN
    printf "Consul installed successfully!\n"
    $NORMAL
    $BOLD
    printf "Version: $VERSION_INFO\n"
    $YELLOW
    printf "======================================\n"
    $NORMAL
    $DOWN
else
    $DOWN
    $RED
    printf "Consul installation failed.\n\n"
    $NORMAL
    $DOWN
fi

Integrating Redis, MySQL, Kafka, Logstash, Elasticsearch, TiDB, and CloudCanal

tj_27 — Thu, 11 Jul 2024 07:58:57 +0000

Here’s how these technologies can work together:

Data Pipeline Architecture:

MySQL: Primary source of structured data.
TiDB: Distributed SQL database compatible with MySQL, used for scalability and high availability.
Kafka: Messaging system for real-time data streaming.
Logstash: Data processing pipeline tool that ingests data from various sources and sends it to various destinations.
Redis: Caching layer for fast access to frequently accessed data.
Elasticsearch: Search and analytics engine for querying large volumes of data.
CloudCanal: Data integration tool used to synchronize data from various sources like MySQL to TiDB, Kafka, Redis, and Elasticsearch.

Workflow Details:

1. Data Ingestion:

Applications save data in MySQL.
CloudCanal is used to sync data from MySQL to TiDB and Kafka.

2. Data Streaming and Processing:

Kafka:

Kafka ingests data from MySQL via CloudCanal and broadcasts it to various topics.
Topics contain streams of data events that can be processed by various consumers.

Logstash:

Logstash acts as a Kafka consumer, processes data from Kafka, and sends it to various outputs such as Elasticsearch and Redis.

3. Data Storage and Retrieval:

TiDB:

TiDB serves as a scalable and highly available database solution that can handle large volumes of data.
TiDB is MySQL-compatible, making integration and migration from MySQL straightforward.

Redis:

Redis is used as a caching layer for frequently accessed data from MySQL or processed events from Kafka.
Applications can query Redis first before querying MySQL to speed up data retrieval.

Elasticsearch:

Logstash can ingest data from Kafka and send it to Elasticsearch.
Elasticsearch indexes the data for fast search and analytics.
Applications can query Elasticsearch for advanced search capabilities and real-time analytics.

Example Data Flow:

Data Entry in MySQL:

A user inserts a new record into the MySQL database.
CloudCanal monitors changes in MySQL and sends events to TiDB and Kafka topics.

Real-Time Processing:

Kafka broadcasts the event to a topic.
Logstash acts as a Kafka consumer, processes the event, and sends the parsed data to Elasticsearch for indexing.
Simultaneously, Redis is updated to cache the new data.

Data Access:

The application checks the Redis cache for the data.
If the data is not in the cache, it queries MySQL or TiDB.
For complex queries and analytics, the application queries Elasticsearch.

This is just for my notes. CTTO

Installing Kafka Exporter Bash Script

tj_27 — Sat, 29 Jun 2024 07:26:32 +0000

Now, this is for kafka exporter. Nothing's new, it is just for kafka.

#!/bin/bash
# Installing Kafka Exporter

# tput commands
CLEAR="tput clear"
DOWN="tput cud1"
BOLD="tput bold"
NORMAL="tput sgr0"
BLACK="tput setaf 0"
RED="tput setaf 1"
GREEN="tput setaf 2"
YELLOW="tput setaf 3"
BLUE="tput setaf 4"

$CLEAR
$NORMAL
$DOWN
# Installation confirmation
printf "You have selected to install kafka_exporter.\n\n"
read -p "Do you want to continue? [ yes / no ] : " USER_INPUT
USER_INPUT=${USER_INPUT:-yes}
$DOWN

# Convert user's choice to lowercase for case-sensitive comparison
USER_INPUT_LOWER=$(echo "$USER_INPUT" | tr '[:upper:]' '[:lower:]')

# Check the user's input
if [ "$USER_INPUT_LOWER" == "yes" ]; then
    $YELLOW
    printf "Kafka exporter installation confirmed.\n\n"
    $NORMAL
else
    printf "Kafka exporter installation cancelled.\n\n"
    exit
fi

# Specify the name of the systemd service
SERVICE_NAME="kafka_exporter"

# Check if the service file exists
if [ -e "/usr/lib/systemd/system/$SERVICE_NAME.service" ]; then
    # Check if the service is active
    if sudo systemctl is-active --quiet "$SERVICE_NAME"; then
        $BOLD
        printf "There is an active $SERVICE_NAME. \n\n"
        $NORMAL
        # Check the version of the active kafka_exporter
        KAFKA_EXPORTER_PATH="/usr/local/$SERVICE_NAME/$SERVICE_NAME"
        VERSION_INFO="$($KAFKA_EXPORTER_PATH --version 2>&1 | awk '/kafka_exporter/ {print $3}')"
        $GREEN
        printf "Active Kafka Exporter Version: $VERSION_INFO \n\n"
        $NORMAL
        printf "Do you want to remove it and replace with a new one? [ 1 / 2 ]\n\n"
        printf " 1: Remove the active kafka_exporter and replace it with a new one. \n\n"
        printf " 2: Don't do anything and exit.\n\n"
        read -rp "> " ACTION
        # Check the action to do
        if [ -z "$ACTION" ]; then
            printf "Removing all kafka_exporter files... \n\n"
            # Remove kafka_exporter related files
            sudo systemctl stop $SERVICE_NAME
            sudo systemctl disable $SERVICE_NAME
            sudo rm /usr/lib/systemd/system/$SERVICE_NAME.service
            $DOWN
            sudo rm -rf /usr/local/kafka_exporter*
            $YELLOW
            printf "Related files removed.\n\n"
            $NORMAL
            printf "Installation will continue...\n\n"
        elif [ "$ACTION" -eq 1 ]; then
            printf "Removing all kafka_exporter files... \n\n"
            # Remove kafka_exporter related files
            sudo systemctl stop $SERVICE_NAME
            sudo systemctl disable $SERVICE_NAME
            sudo rm /usr/lib/systemd/system/$SERVICE_NAME.service
            $DOWN
            sudo rm -rf /usr/local/kafka_exporter*
            $YELLOW
            printf "Related files removed.\n\n"
            $NORMAL
            printf "Installation will continue...\n\n"
        elif [ "$ACTION" -eq 2 ]; then
            $DOWN
            printf "No action done.\n\n"
            exit
        else
            printf "Invalid input. Please enter 1 or 2.\n\n"
            exit 1
        fi
    else
        printf "There's a $SERVICE_NAME service that is not active. Removing related files...\n\n"
        sudo systemctl stop $SERVICE_NAME
        sudo systemctl disable $SERVICE_NAME
        sudo rm /usr/lib/systemd/system/$SERVICE_NAME.service
        sudo rm -rf /usr/local/kafka_exporter*
        $YELLOW
        printf "Related files removed.\n\n"
        $NORMAL
        printf "Installation will continue...\n\n"
    fi
else
    printf "No $SERVICE_NAME service file found.\n\n"
    fi

# Curling Google to check if connected to a network
printf "Looking for a network...\n\n"
if curl google.com > /dev/null; then
    $DOWN
    $YELLOW
    printf "Network connected.\n\n"
    $NORMAL
else
    $DOWN
    printf "The server is not connected to the network. Please connect and try again.\n\n";
    exit 1
fi

echo -n "Insert the version you would like to be installed, default is [ 1.7.0 ] : "
$BOLD
$BLUE
read VERSION
$NORMAL
VERSION=${VERSION:-1.7.0}
$DOWN
$NORMAL
# Download the file
wget https://github.com/danielqsj/kafka_exporter/releases/download/v$VERSION/kafka_exporter-$VERSION.linux-amd64.tar.gz -P /opt
# Extract the downloaded tarball in user directory with a new name
tar -xzvf /opt/kafka_exporter-$VERSION.linux-amd64.tar.gz -C /usr/local && mv /usr/local/kafka_exporter-$VERSION.linux-amd64 /usr/local/kafka_exporter

# IP
IP=$(hostname -I | awk '{print $1}')

# Create a systemd service file for Kafka Exporter
cat >/usr/lib/systemd/system/kafka_exporter.service<<EOF
[Unit]
Description=Kafka_Exporter
Documentation=https://prometheus.io/
After=network.target

[Service]
Type=simple
PIDFile=/usr/local/kafka_exporter/kafka_exporter.pid
ExecStart=/usr/local/kafka_exporter/kafka_exporter --kafka.server=xxIPxx:9092 --log.level=info
ExecReload=/bin/kill -s HUP
ExecStop=/bin/kill -s QUIT
PrivateTmp=true

[Install]
WantedBy=multi-user.target
EOF

# Edit the service file in-place
sed -i "s#xxIPxx#$IP#g" /usr/lib/systemd/system/kafka_exporter.service

# Reload systemd and start Kafka Exporter
sudo systemctl daemon-reload
sudo systemctl start kafka_exporter.service
sudo systemctl enable kafka_exporter.service
sudo systemctl status kafka_exporter.service

# Cleanup downloaded file
rm -f /opt/kafka_exporter-$VERSION.linux-amd64.tar.gz*
$DOWN
if sudo systemctl is-active --quiet "$SERVICE_NAME"; then
    $DOWN
    $BOLD
    $YELLOW
    printf "======================================\n"
    $GREEN
    printf "Kafka Exporter installed successfully!\n"
    $NORMAL
    $BOLD
    printf "Version: $VERSION\n"
    $YELLOW
    printf "======================================\n"
    $NORMAL
    $DOWN
else
    $DOWN
    $RED
    printf "Kafka Exporter installation failed.\n\n"
    $NORMAL
    $DOWN
fi

Installing Redis Exporter Bash Script

tj_27 — Sat, 29 Jun 2024 07:23:31 +0000

Here's my version of writing redis_exporter installtion script.
I just found writing script fun not until my back aches, lol

#!/bin/bash
# Installing Redis Exporter

# tput commands
CLEAR="tput clear"
DOWN="tput cud1"
BOLD="tput bold"
NORMAL="tput sgr0"
BLACK="tput setaf 0"
RED="tput setaf 1"
GREEN="tput setaf 2"
YELLOW="tput setaf 3"
BLUE="tput setaf 4"

$CLEAR
$DOWN
# Installation confirmation
printf "You have selected to install redis_exporter.\n\n"
read -p "Do you want to continue? [ yes / no ] : " USER_INPUT
USER_INPUT=${USER_INPUT:-yes}
$DOWN

# Convert user's choice to lowercase for case-sensitive comparison
USER_INPUT_LOWER=$(echo "$USER_INPUT" | tr '[:upper:]' '[:lower:]')

# Check the user's input
if [ "$USER_INPUT_LOWER" == "yes" ]; then
    $YELLOW
    printf "Redis exporter installation confirmed.\n\n"
    $NORMAL
else
    printf "Redis exporter installation cancelled.\n\n"
    exit
fi

# Specify the name of the systemd service
SERVICE_NAME="redis_exporter"

# Check if the service file exists
if [ -e "/usr/lib/systemd/system/$SERVICE_NAME.service" ]; then
    # Check if the service is active
    if sudo systemctl is-active --quiet "$SERVICE_NAME"; then
        $BOLD
        printf "There is an active $SERVICE_NAME. \n\n"
        $NORMAL
        # Check the version of the active redis_exporter
        REDIS_EXPORTER_PATH="/usr/local/$SERVICE_NAME/$SERVICE_NAME"
        VERSION_INFO="$($REDIS_EXPORTER_PATH --version 2>&1 | awk '{print $6}')"
        $GREEN
        printf "Active Redis ExporterVersion: $VERSION_INFO \n\n"
        $NORMAL
        printf "Do you want to remove it and replace with a new one? [ 1 / 2 ]\n\n"
        printf " 1: Remove the active redis_exporter and replace it with a new one. \n\n"
        printf " 2: Don't do anything and exit.\n\n"
        read -rp "> " ACTION
        # Check the action to do
        if [ -z "$ACTION" ]; then
            printf "Removing all redis_exporter files... \n\n"
            # Remove redis_exporter related files
            sudo systemctl stop $SERVICE_NAME
            sudo systemctl disable $SERVICE_NAME
            sudo rm /usr/lib/systemd/system/$SERVICE_NAME.service
            sudo rm -rf /usr/local/redis_exporter*
            $YELLOW
            printf "Related files removed.\n\n"
            $NORMAL
            printf "Installation will continue...\n\n"
        elif [ "$ACTION" -eq 1 ]; then
            printf "Removing all redis_exporter files... \n\n"
            # Remove redis_exporter related files
            sudo systemctl stop $SERVICE_NAME
            sudo systemctl disable $SERVICE_NAME
            sudo rm /usr/lib/systemd/system/$SERVICE_NAME.service
            sudo rm -rf /usr/local/redis_exporter*
            $YELLOW
            printf "Related files removed.\n\n"
            $NORMAL
            printf "Installation will continue...\n\n"
        elif [ "$ACTION" -eq 2 ]; then
            $DOWN
            printf "No action done.\n\n"
            exit
        else
            printf "Invalid input. Please enter 1 or 2.\n\n"
            exit 1
        fi
    else
        printf "There's a $SERVICE_NAME service that is not active. Removing related files...\n\n"
        sudo systemctl stop $SERVICE_NAME
        sudo systemctl disable $SERVICE_NAME
        sudo rm /usr/lib/systemd/system/$SERVICE_NAME.service
        sudo rm -rf /usr/local/redis_exporter*
        $YELLOW
        printf "Related files removed.\n\n"
        $NORMAL
        printf "Installation will continue...\n\n"
    fi
else
    printf "No $SERVICE_NAME service file found.\n\n"
    fi

# Curling Google to check if connected to a network
printf "Looking for a network...\n\n"
if curl google.com > /dev/null; then
    $DOWN
    $YELLOW
    printf "Network connected.\n\n"
    $NORMAL
else
    $DOWN
    printf "The server is not connected to the network. Please connect and try again.\n\n";
    exit 1
fi

echo -n "Insert the version you would like to be installed, default is [ 1.29.0 ] : "
$BOLD
$BLUE
read VERSION
$NORMAL
VERSION=${VERSION:-1.29.0}
$DOWN
$NORMAL
# Download the file
wget https://github.com/oliver006/redis_exporter/releases/download/v$VERSION/redis_exporter-v$VERSION.linux-amd64.tar.gz -P /opt

# Extract the downloaded tarball in the user directory with new name
tar -xzvf /opt/redis_exporter-v$VERSION.linux-amd64.tar.gz -C /usr/local && mv /usr/local/redis_exporter-v$VERSION.linux-amd64 /usr/local/redis_exporter

# Create a systemd service file for redis exporter
sudo cat >/usr/lib/systemd/system/redis_exporter.service<<EOF
[Unit]
Descrition=redis_exporter
Documentation=https://prometheus.io/
After=network.target

[Service]
Type=simple
PIDFile=/usr/local/redis_exporter/redis_exporter.pid
ExecStart=/usr/local/redis_exporter/redis_exporter -redis.password=xxPASSWORDxx -redis.addr=xxIPxx:xxPORTxx
ExecReload=/bin/kill -s HUP
ExecStop=/bin/kill -s QUIT
PrivateTmp=true

[Install]
WantedBy=multi-user.target
EOF

# Provide the redis information
$DOWN
$YELLOW
printf "Please enter the Redis information (e.g, 10.10.10.10:6379 MyPasswd123) : \n"
$NORMAL
read -r REDIS_INFO
$DOWN
IP=$(echo "$REDIS_INFOR" | awk '{print $1}' FS=":")
PORT=$(echo "$REDIS_INFO" | awk '{print $1}' | awk '{print $2}' FS=":")
PASSWORD=$(echo "$REDIS_INFO" |awk '{print $2}')

sed -i "s#xxPORTxx#$PORT#g" /usr/lib/systemd/system/redis_exporter.service
sed -i "s#xxIPxx#$IP#g" /usr/lib/systemd/system/redis_exporter.service
sed -i "s#xxPASSWORDxx#$PASSWORD#g" /usr/lib/systemd/system/redis_exporter.service

# Reload systemd and start redis exporter
sudo systemctl daemon-reload
sudo systemctl start redis_exporter.service
sudo systemctl enable redis_exporter.service
sudo systemctl status redis_exporter.service

# Cleanup downloaded file
rm -f /opt/redis_exporter*

$DOWN
if sudo systemctl is-active --quiet "$SERVICE_NAME"; then
    $DOWN
    $BOLD
    $YELLOW
    printf "======================================\n"
    $GREEN
    printf "Redis Exporter installed successfully!\n"
    $NORMAL
    $BOLD
    printf "Version: $VERSION_INFO\n"
    $YELLOW
    printf "======================================\n"
    $NORMAL
    $DOWN
else
    $DOWN
    $RED
    printf "Redis Exporter installation failed.\n\n"
    $NORMAL
    $DOWN
fi

Installing Node Exporter Bash Script

tj_27 — Mon, 24 Jun 2024 08:11:54 +0000

Hello, everyone!
I enjoy writing script since I learned it.
But please don't mock me, as I don't know yet the rules. I mean I'm not sure if this is the common and professional way of writing scripts.
I am not actually an IT (my college course is really far from this field, lol).
Anyways, hope this can help.

#!/bin/bash
# Installing Node Exporter

# Output yellow text
echo_yellow() {
    echo - "\e[93m$1\e[0m"
}

# Specify the name of the systemd service
SERVICE_NAME="node_exporter"

# Check if the service file exists
if [ -e "/usr/lib/systemd/system/$SERVICE_NAME.service" ]; then
    # Check if the service is active
    if sudo systemctl is-active --quiet "$SERVICE_NAME"; then
        echo_yellow "There is an active $SERVICE_NAME."
        # Check the version of the active node_exporter
        NODE_EXPORTER_PATH="/usr/local/$SERVICE_NAME/$SERVICE_NAME"
        VERSION_INFO="$($NODE_EXPORTER_PATH --version | awk '/node_exporter/ {print $3}')"
        echo_yellow "Active Node ExporterVersion: $VERSION_INFO"
        echo
        echo "Do you want to remove it and replace with a new one? [ 1 / 2 ]"
        echo
        echo_yellow "1: Remove the active node_exporter and replace it with a new one."
        echo
        echo "2: Don't do anything and exit."
        echo
        read -rp "> " ACTION
        # Check the action to do
        if [ -z "$ACTION" ]; then
            echo
            echo_yellow "Removing all node_exporter files..."
            echo
            # Remove node_exporter related files
            sudo systemctl stop $SERVICE_NAME
            sudo systemctl disable $SERVICE_NAME
            sudo rm /usr/lib/systemd/system/$SERVICE_NAME.service
            sudo rm -rf /usr/local/node_exporter*
            echo
            echo "Related files removed."
            echo
            echo "Installation will continue..."
            echo
        elif [ "$ACTION" -eq 2 ]; then
            echo
            echo "No action done."
            echo
            exit
        else
            echo
            echo "Invalid input. Please enter 1 or 2."
            echo
            exit 1
        fi
    else
        echo "There's a $SERVICE_NAME service that is not active. Removing related files..."
        sudo systemctl stop $SERVICE_NAME
        sudo systemctl disable $SERVICE_NAME
        sudo rm /usr/lib/systemd/system/$SERVICE_NAME.service
        sudo rm -rf /usr/local/node_exporter*
        echo
        echo "Related files removed."
        echo
        echo "Installation will continue..."
        echo
    fi
else
    echo "No $SERVICE_NAME service file found."
    echo
    fi

# Curling Google to check if connected to a network
echo "Looking for a network..."
echo
if curl -sSf https://www.google.com > /dev/null; then
    echo "Network connected."
    echo
else
    echo "The server is not connected to the network. Please connect and try again.";
    echo
    exit 1
fi

echo_yellow -n "Insert the version you would lilke to be installed, default is [ 1.2.2 ] :"
read VERSION
VERSION=${VERSION:-1.2.2}
echo
# Download the file
wget https://github.com/prometheus/node_exporter/releases/download/v$VERSION/node_exporter-$VERSION.linux-amd64.tar.gz -P /opt
# Extract the downloaded tarball in user directory with a new name
tar -xzvf /opt/node_exporter-$VERSION.linux-amd64.tar.gz -C /usr/local && mv /usr/local/node_exporter-$VERSION.linux-amd64 /usr/local_node_exporter

# Create a systemd service file for Node Exporter
cat >/usr/lib/systemd/system/node_exporter.service<<EOF
[Unit]
Description= Node Exporter
Documentation= https://prometheus.io/
Wants=network.target
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/node_exporter/node_exporter
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

# Create the node_exporter user
sudo useradd -M -r -s /bin/false node_exporter

# Set proper permissions
sudo chown -R node_exporter. /usr/local/node_exporter/node_exporter
sudo chmod 755 /usr/local/node_exporter/node_exporter

# Reload systemd and start Node Exporter
sudo systemctl daemon-reload
sudo systemctl start node_exporter.service
sudo systemctl enable node_exporter.service
sudo systemctl status node_exporter.service

# Cleanup downloaded file
rm -f /opt/node_exporter-$VERSION.linux-amd64.tar.gz*
echo 
if sudo systemctl is-active --quiet "$SERVICE_NAME"; then
    echo_yellow "Node Exporter installed successfully!"
    echo
else
    echo "Node Exporter installation failed."
    echo
fi

Node Exporter Service File

tj_27 — Sun, 23 Jun 2024 12:42:04 +0000

[Unit]
Description= Node Exporter
Documentation= https://prometheus.io/
Wants=network.target
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=$PATH/node_exporter
Restart=on-failure

[Install]
WantedBy=multi-user.target