DEV Community: Dimeji Ojewunmi

DORA METRICS EXPORTER

Dimeji Ojewunmi — Thu, 08 Aug 2024 20:42:15 +0000

Introduction to Dora Metrics

DORA Metrics are key performance indicators used to measure the effectiveness of a software delivery process. They help teams understand the efficiency of their DevOps practices by tracking key aspects of the CI/CD pipeline.

Below are the outlined reason stipulate why we need to leverage the need of Dora Metrics in an organization

Deployment Frequency: Measures how often deployments occur. High frequency indicates a fast, iterative development process.
Lead Time for Changes: The time it takes from committing code to deploying it in production. Short lead times suggest an efficient pipeline.
Change Failure Rate: The percentage of deployments that result in a failure in production. Lower rates indicate more stable and reliable releases.
Time to Restore Service: The time it takes to recover from a failure in production. Faster recovery times imply a more resilient system.

Installation Setup

The following git command below are directories and files to be opened in the cause of our Dora metric exporter installation

mkdir dora_metrics_exporter
cd dora_metrics_exporter
python3 -m venv venv
touch requirements.txt .env exporter.py
source venv/bin/activate  # On Windows use `venv\Scripts\activate`
touch requirements.txt .env exporter.py

`requirements.txt`context below

flask
prometheus_client
requests
python-dotenv

Run the script below right after the opening of 'requirements.txt'

pip install -r requirements.txt

`.env` context below

GITHUB_TOKEN=your_github_token
REPO_OWNER=holadmex (GitHub Username)
REPO_NAME=hng_boilerplate_expressjs (Project Repository Name)

`exporter.py` context below

# exporter.py
from flask import Flask
from prometheus_client import start_http_server, Gauge
import requests
import time
import os
import logging
from dotenv import load_dotenv

# Load environment variables from .env file
load_dotenv()

# Configure logging
logging.basicConfig(level=logging.INFO)

app = Flask(__name__)

# Metrics
DEPLOYMENT_FREQUENCY = Gauge('deployment_frequency', 'Frequency of deployments')
LEAD_TIME_FOR_CHANGES = Gauge('lead_time_for_changes', 'Lead time for changes')
CHANGE_FAILURE_RATE = Gauge('change_failure_rate', 'Rate of change failures')
TIME_TO_RESTORE_SERVICE = Gauge('time_to_restore_service', 'Time to restore service')

# GitHub Configuration
GITHUB_TOKEN = os.getenv('GITHUB_TOKEN')
REPO_OWNER = os.getenv('REPO_OWNER')
REPO_NAME = os.getenv('REPO_NAME')

HEADERS = {
    'Authorization': f'token {GITHUB_TOKEN}'
}

# Function to fetch data from GitHub API
def fetch_github_data():
    logging.info("Fetching GitHub data")
    url = f'https://api.github.com/repos/{REPO_OWNER}/{REPO_NAME}/stats/code_frequency'
    response = requests.get(url, headers=HEADERS)
    response.raise_for_status()
    logging.info("Data fetched successfully")
    return response.json()

def calculate_metrics():
    logging.info("Calculating metrics")
    try:
        data = fetch_github_data()
        # Perform calculations for DORA metrics here

        # Dummy values for demonstration
        DEPLOYMENT_FREQUENCY.set(10)
        LEAD_TIME_FOR_CHANGES.set(5)
        CHANGE_FAILURE_RATE.set(0.1)
        TIME_TO_RESTORE_SERVICE.set(2)
        logging.info("Metrics calculated and set")
    except Exception as e:
        logging.error(f"Error calculating metrics: {e}")

@app.route('/metrics')
def metrics():
    return ""

if __name__ == '__main__':
    # Start Prometheus metrics server
    logging.info("Starting Prometheus metrics server")
    start_http_server(8001)
    logging.info("Prometheus metrics server started on port 8001")
    # Schedule metric calculations
    while True:
        calculate_metrics()
        time.sleep(300)  # Fetch data every 5 minutes

The code below execute the Dora Metric Exporter installation

python exporter.py
OR
nohup python exporter.py & (This cmd runs your application in the background without your terminal being held hostage)

NB:

On completion of the installation, Dora metrics job needs to be configured or added to Prometheus config file in order to export the generated metrics from Dora exporter to Prometheus which will be visualized on Grafana dashboard through Grafana data source.

scrape_configs:
  - job_name: 'dora_metrics'
    static_configs:
      - targets: ['localhost:8001']

DORA EXPORTER METRICS QUERY TO BE VISUALISED ON GRAFANA DASHBOARD

deployment_frequency
lead_time_for_changes
change_failure_rate
time_to_restore_service

Thank you for reading to the end,
Happy Hands On Practice!.

Python Messaging Queue App Development

Dimeji Ojewunmi — Tue, 16 Jul 2024 09:46:01 +0000

Hello guys,

Today we'll be developing a Python Messaging Queue Test, which consist of the following 'Tools' as our requirement.

RabbitMQ
Celery
Ngrok
Nginx

Lets talk about the theoretical aspect of the core tools we'll be using in the course of the task, courtesy of (HNG Internship)

What is RabbitMQ?

RabbitMQ is an open-source messaging software that implements the Advanced Message Queuing Protocol (AMQP) and is widely used for managing messaging queues. RabbitMQ here also serves as Amazon Simple Queue Service(SQ) distributed messaging system which is fully managed by AWS, makes it easy to decouple and scale microservices, distributed systems, and serverless applications.

What is Celery?

Celery is an asynchronous task queue/job queue based on distributed message passing. its main purpose is to handle "asynchronous" task queue/job queue that handles the execution of background tasks.

Asynchronous programming is a form of programming that allows a unit of work (like a function or a task) to start processing before the previous unit of work finishes which also means its serves task to function independently.

What is Ngrok?

Ngrok is a tool that creates secure tunnels from a public endpoint (like the internet) to a locally running web service or application. It allows developers to expose their local development stage environment to the internet securely, making it easier to test and share work-in-progress applications with team members without deploying them to a live environment.

What is Nginx?

Nginx is a web server and reverse proxy server that is widely used for serving web content, handling load balancing, and acting as a proxy for email services. It's known for its speed, reliability, and low resource usage mainly in development to production stage.

Lets hop into the business of the day, getting hands on the project task as we progress in the course of the blog post.

Installation Prerequisite

sudo apt-get update

sudo apt-get install rabbitmq-server

sudo apt install nginx -y

Linux | ngrok documentation

Learn how to install ngrok on a remote Linux device to provide secure access and management.

ngrok.com

(Redirection link to installl Ngrok)

On successful completion of installing Ngrok, log on to

ngrok | API Gateway, IoT Device Gateway, Secure Tunnels for Containers, Apps & APIs

ngrok is a secure ingress platform that enables developers to add global server load balancing, reverse proxy, firewall, API gateway and Kubernetes Ingress to applications and APIs.

ngrok.com

and sign up, to have a generate token which will be integrated with your project directory on your Linux terminal environment. Run the command below

ngrok authtoken <your generate token>

Hold still, we'll be writing few couple blocks of code using Python and installing the required Python Framework/Library to execute our Python blocks of code functions.

First we'll open a directory on our Linux Vm (e.g Ubuntu)

mkdir messaging-queue-app

cd messaging-queue-app

touch app.py .env ngnix.conf

Created app.py File context

import os
from flask import Flask, request, Response
from celery import Celery
from dotenv import load_dotenv
from datetime import datetime
import logging
import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart

# Load environment variables
load_dotenv()

# Initialize Flask application
app = Flask(__name__)

# Configure Celery
app.config['CELERY_BROKER_URL'] = 'pyamqp://guest@localhost//'
app.config['CELERY_RESULT_BACKEND'] = 'rpc://'
celery = Celery(app.name, broker=app.config['CELERY_BROKER_URL'])
celery.conf.update(app.config)

# Configure logging
log_path = os.path.join(os.path.expanduser('~'), 'messaging_system.log')
logging.basicConfig(level=logging.INFO, filename=log_path, filemode='a', format='%(asctime)s - %(message)s')

# Celery task to send email
@celery.task
def send_email(to_email):
    from_email = os.getenv('EMAIL_USER')
    from_password = os.getenv('EMAIL_PASSWORD')
    smtp_server = 'smtp.mail.yahoo.com'
    smtp_port = 465
    subject = 'Official Email from Flask Application'
    body = 'This is a test email sent from the Flask application.'
    msg = MIMEMultipart()
    msg['From'] = from_email
    msg['To'] = to_email
    msg['Subject'] = subject
    msg.attach(MIMEText(body, 'plain'))
    try:
        logging.info(f'Connecting to SMTP server {smtp_server}:{smtp_port}')
        with smtplib.SMTP_SSL(smtp_server, smtp_port) as server:
            logging.info('Logging in to SMTP server')
            server.login(from_email, from_password)
            logging.info('Sending email')
            text = msg.as_string()
            server.sendmail(from_email, to_email, text)
            logging.info(f'{datetime.now()} - Email sent to {to_email}')
    except Exception as e:
        logging.error(f'Failed to send email: {e}')

# Flask routes
@app.route('/', methods=['GET'])
def index():
    if 'sendmail' in request.args:
        send_email.apply_async(args=[request.args.get('sendmail')])
        logging.info(f'{datetime.now()} - Email queued for: {request.args.get("sendmail")}')
        return 'Email queued.'
    elif 'talktome' in request.args:
        logging.info(f'{datetime.now()} - Logged the current time.')
        return 'Logged the current time.'
    else:
        return 'Welcome to the messaging system. Use /?sendmail=email or /?talktome.'

@app.route('/logs', methods=['GET'])
def logs():
    if os.path.exists(log_path):
        with open(log_path, 'r') as f:
            log_content = f.readlines()
        filtered_logs = [line for line in log_content if 'Email sent to' in line or 'Email queued for' in line]
        return Response(''.join(filtered_logs), mimetype='text/plain')
    else:
        return "Log file not found.", 404

# Main entry point
if __name__ == '__main__':
    app.run(debug=True, host='0.0.0.0', port=5000)

NB:

The Python Messaging queue application development here was setup to make use of (Yahoomail SMTP Send). You you're to put in your email address, and also generate an app token in your Yahoomail account security path, and make the configuration changes in the (.env)

Created .env File context

EMAIL_USER="ojewumi_dimeji@ymail.com"
EMAIL_PASSWORD="clndcnbjfpaelrj1"
SMTP_SERVER="smtp.mail.yahoo.com"
SMTP_PORT=465

Created nginx.conf File context

server {
    listen 80;
    server_name localhost;
    location / {
        proxy_pass http://127.0.0.1:5000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

On filling each files with their respective blocks of codes, there are few more Python requirement Frameworks/libraries to be installed right in our project directory using the (virtual Linux environment cmd)

Installation

sudo apt-get install python3-venv

Right after the python3-venv installation, execute the following command below on your Linux terminal. This command create a separate virtual environment in your project directory for your block of code function to carry out its functions.

python3 -m venv messaging-queue
source messaging-queue/bin/activate

On carry out the above, we'll commence installation of our Python Frameworks/Libraries right inside our created (venv messaging-queue) which you will be redirected to on executing the (source messaging-queue/bin/activate) cmd

pip install celery
pip install flask
pip install python-dotenv
pip freeze > requirements.txt (This CMD output all the installed Python pip installation for the application to run successfully)

Freight not, lol, we're 95% done to the completion of our project on carrying out the following command to execute our Python Messaging Queue Test.

First step:

sudo rabbitmq-server (This CMD launch our RabbitMQ sever to run for the course of our python messaging queue app test)

sudo rabbitmq-server -detached (This helps to run your RabbitMQ sever in the background without the terminal being held hostage)

Open another sperate terminal on executing each rabbitmq-server and celery cmd

"Optional, but a must on first attempt with the cmd above and below"

Second step:

celery -A app.celery worker --loglevel=info (This opens your Celery monitoring environment on checking for errors and successful ?sendmail execution of your python messaging queue app)

celery -A app.celery worker --loglevel=info --detach (This helps to run your Celery in the background without the terminal being held hostage.)

Third step:

python3 app.py (This CMD lunch your web app development URL to be accessed on your localhost web browser.)

nohup python3 app.py > app.log 2>&1 & (This helps to lunch the app in the background without the terminal being held hostage, and also output the return function to app.log.)

Here are the following path to the application web redirection on sending your queued email locally without Ngrok

localhost:5000
localhost:5000/?sendmail=happy-learning@gmail.com (Email you which to send message to)
localhost:+5000/?talktome
localhost:5000/logs

Finally, on exposing our `Local Development Test` to other team members, carry out the installation below;

sudo apt-get install screen

On completion of the installation, run;

screen -S ngrok

ngrok http 5000 (You should see an outputted endpoint "https://e7c9-44-202-0-101.ngrok-free.app/" which serves as your URL to be publicly accessed.)

Here are the following path to the application web redirection on sending your queued email with Ngrok

https://e7c9-44-202-0-101.ngrok-free.app/
https://e7c9-44-202-0-101.ngrok-free.app:5000/?sendmail=happy-learning@gmail.com
https://e7c9-44-202-0-101.ngrok-free.app:5000/?talktome
https://e7c9-44-202-0-101.ngrok-free.app:5000/logs

We've come to the End of the Task,

Celebrate yourself for not holding back on the success of completing the Python Messaging Queue App Development,

Happy reading,
Thank you.
Dimeji...

Linux Bash Script User Creation

Dimeji Ojewunmi — Wed, 03 Jul 2024 19:06:07 +0000

Hi Enthusiast DevOps Engineer,

Today we'll be having a deep insight on the need to write an automated users bash script in an organization, and also having all the metrics and password auto generated stored in a LOG_FILE.

The main reason for the creation of the users bash scrip is to mitigate repetitive user/group creation task manually, which might be daunting in most cases, and also to have an autonomous control of employees who have assess to the organization Linux workflow environment.

Before i proceed, the user bash script creation is in curtesy of (HNG Internship) program, which has just kicked off for the year 2024 "11th cohort", helping various beginners and intermediate tech enthusiast in having a real world project experience. To learn more about HNG internship program, kindly visit

hng.tech

Now lets delve to the business of the day, seat back, make use of your seat belt, while we journey through this together.

Firstly, we'll create a text file called users filled with the content below, which is dependent on our bash script file to execute its functions.

light; sudo,dev,www-data
 idimma; sudo
 mayowa; dev,www-data
 dmex; devops,sysops
 iosegbo; sysops

Secondly we'll be creating a file called create_users.sh which will run as our bash script file with the code below

#!/bin/bash

# Autogenerate Password and Metrics will be sent here
LOG_FILE="/var/log/user_management.log"
PASSWORD_FILE="/var/secure/user_passwords.txt"


# This will create a /var/secure file, and also assign a <write, read & execution> permission to the created file.
mkdir -p /var/secure
chmod 700 /var/secure

# Create or clear the log and password files
> $LOG_FILE
> $PASSWORD_FILE
chmod 600 $PASSWORD_FILE

# Function to generate random password
generate_password() {
    echo $(openssl rand -base64 12)
}


# Read the input file, which is the <users> file we created to be dependent on our bash script code function execution
INPUT_FILE=$1
# Process each line in the file
while IFS= read -r line; do
  # Ignore characters before the semicolon
  after_semicolon="${line#*;}"

  # Split the line into items separated by commas
  IFS=',' read -ra items <<< "$after_semicolon"

  # Ensure a group exists for each item
  for item in "${items[@]}"; do
    item=$(echo "$item" | xargs)  # Trim whitespace
    if [ ! -z "$item" ]; then
      if ! getent group "$item" > /dev/null; then
        echo "Creating group: $item"
        sudo groupadd "$item"
      else
        echo "Group already exists: $item"
      fi
    fi
  done
done < "$INPUT_FILE"


# Process each line in the input file
while IFS=';' read -r user groups; do
    # Trim whitespace
    user=$(echo "$user" | xargs)
    groups=$(echo "$groups" | xargs)


    # Create user with home directory and primary group
    if ! id "$user" &>/dev/null; then
        useradd -m "$user"
        echo "User $user was created successfully." >> $LOG_FILE
    else
        echo "User $user already exists." >> $LOG_FILE
    fi

    # Set user's groups
    if [ -n "$groups" ]; then
        usermod -aG $groups "$user"
        echo "User $user added to group: $groups." >> $LOG_FILE
    fi

    # Auto Generate and set password
    password=$(generate_password)
    echo "$user:$password" | chpasswd
    echo "$user,$password" >> $PASSWORD_FILE
    echo "Password for user $user set." >> $LOG_FILE


done < "$INPUT_FILE"

Thirdly, we'll have to assign an execution permission to the created bash script file chmod 700 create_users.sh

Lastly, the command to execute our bash script file is

sudo ./create_users.sh

The command above will execute the above bash script function which is written as a code, enabling all the users to be assigned to the specified group and also enabling each users to have an assigned encrypted password which will be redirected or logged to a LOG_FILE encoded in the script.

Here are the list of commands to execute on our Linux VM to check the following output of our bash script block of code

sudo cat /var/log/user_management.log (This cmd checks for the activities which has taken place in the course of the bash script execution i.e "MERTICS").

sudo cat /var/secure/user_passwords.txt (This cmd checks for the auto generated password assigned to each users).

To confirm validity of your user creation, simply execute the following command below

su dmex

Then copy the encrypted password generated for dmex under sudo cat /var/secure/user_passwords.txt

Thank you for taking your time in reading through to the completion of the blog post, do have a wonderful time.