DEV Community: Oluchi Oraekwe

Monitoring and Alerting for Blue/Green Deployment

Oluchi Oraekwe — Tue, 09 Dec 2025 14:44:10 +0000

Introduction

In the previous article, we explored the Blue/Green deployment strategy using Nginx as a reverse proxy to maintain service availability whenever one of the upstream servers fails. You can read that article here: Blue/Green Deployment with Nginx Upstreams

In this follow-up article, we extend the concept by adding monitoring and alerting mechanisms to the deployment.
In DevOps, monitoring and alerting are critical for maintaining system reliability and availability. When a server fails or becomes unstable, the alerting system notifies the responsible team immediately so the issue can be resolved quickly, maintaining high availability.

Here, we introduce a Watcher Service that runs as a sidecar container in the Docker Compose stack. It monitors Nginx logs in real time and sends alerts to Slack whenever:

A failover event occurs whereby the traffic pool switches from Blue to Green
The traffic pool switches back from Green to Blue
There is a high error rate over a given period

1. Formatting Nginx Logs

To enable meaningful monitoring, the Nginx access logs will have to include specific upstream details such as:

Application pool (x-app-pool)
Release version (x-release-id)
Upstream status
Upstream request and response times

These logs are stored in a shared volume so that the watcher application can also access them.

log_format main '$remote_addr - $remote_user [$time_local] '
        '"$request" status=$status body_bytes_sent=$body_bytes_sent '
        'pool=$upstream_http_x_app_pool release=$upstream_http_x_release_id '
        'upstream_status=$upstream_status upstream_addr=$upstream_addr '
        'request_time=$request_time upstream_response_time=$upstream_response_time';

error_log /var/log/nginx/error.log warn;
access_log /var/log/nginx/access.log main;

2. Creating the Log Watcher Application

The watcher script performs these major tasks:

Monitor logs in real time and detect pool switches (Blue to Green and Green back to Blue)
Detect and alert on high failure rates (e.g., Blue server repeatedly failing)
Implement alert cooldown to prevent spamming your Slack channel

Below are the major sections of the watcher.py application.

Imports and Configuration

import time
import os
import re
import requests
import logging
from collections import deque

LOG_FILE = os.getenv("NGINX_LOG_PATH")
SLACK_WEBHOOK_URL = os.getenv("SLACK_WEBHOOK_URL")
ERROR_RATE_THRESHOLD = float(os.getenv("ERROR_RATE_THRESHOLD", "2.0"))
WINDOW_SIZE = int(os.getenv("WINDOW_SIZE", "200"))
CHECK_INTERVAL = int(os.getenv("CHECK_INTERVAL", "10"))
ALERT_COOLDOWN = float(os.getenv("ALERT_COOLDOWN", "300"))

logging.basicConfig(
    level=logging.INFO,
    format="[%(asctime)s] %(levelname)s: %(message)s",
    datefmt="%H:%M:%S",
)

pattern = re.compile(
    r'(?P<ip>\S+) - - \[(?P<time>[^\]]+)\] "(?P<method>\S+) (?P<path>\S+) \S+" '
    r'status=(?P<status>\d+) [^ ]* pool=(?P<pool>\S+) release=(?P<release>\S+) '
    r'upstream_status=(?P<upstream_status>[0-9,\s]+) upstream_addr=(?P<upstream_addr>[0-9\.:,\s]+) '
    r'request_time=(?P<request_time>[\d\.]+) upstream_response_time=(?P<upstream_response_time>[\d\.,\s]+)'
)

recent = deque(maxlen=WINDOW_SIZE)
last_pool = os.getenv("ACTIVE_POOL", "blue")
last_check = 0.0
last_alert_time = {"failover": 0, "switch": 0, "error_rate": 0}

Slack Notification Function

def send_slack_alert(message: str):
    if not SLACK_WEBHOOK_URL:
        logging.warning("SLACK_WEBHOOK_URL not set. Cannot send alert.")
        return
    try:
        response = requests.post(SLACK_WEBHOOK_URL, json={"text": message})
        response.raise_for_status()
        logging.info("Slack alert sent.")
    except requests.exceptions.RequestException as e:
        logging.error(f"Failed to send Slack alert: {e}")

High Error Rate Detection

This function checks the percentage of upstream 5xx errors over a defined window and threshold. Once the rate is higher than the threshold, it will send an alert notification to a Slack channel.

def check_alert():
    now = time.time()

    while recent and now - recent[0][0] > WINDOW_SIZE * CHECK_INTERVAL:
        recent.popleft()

    total = len(recent)
    if total == 0:
        return

    errors = sum(1 for _, _, upstream_status in recent if upstream_status.startswith("5"))
    rate = (errors / total) * 100

    if rate >= ERROR_RATE_THRESHOLD and now - last_alert_time["error_rate"] > ALERT_COOLDOWN:
        last_alert_time["error_rate"] = now
        send_slack_alert(
            f"🚨 *High Error Rate Detected!*\n"
            f"• Error rate: `{rate:.2f}%`\n"
            f"• Threshold: `{ERROR_RATE_THRESHOLD}%`\n"
            f"• Total requests: `{total}`\n"
            f"• Active Pool: `{last_pool}`\n"
            f"• Time: {time.strftime('%Y-%m-%d %H:%M:%S')}`"
        )

Traffic Switch & Failover Monitoring

This function watches Nginx logs in real-time by reading the server logs and detects:

Failover events
Pool changes
Upstream server changes

def monitor_log():
    global last_pool, last_check
    logging.info("Starting log watcher...")
    send_slack_alert("👀 Log watcher started. Monitoring Nginx logs...")

    while not os.path.exists(LOG_FILE):
        time.sleep(5)

    with open(LOG_FILE, "r") as f:
        f.seek(0, 2)
        while True:
            try:
                line = f.readline()
                if not line:
                    time.sleep(CHECK_INTERVAL)
                    continue

                match = pattern.search(line)
                if not match:
                    continue

                data = match.groupdict()
                pool = data["pool"]
                release = data["release"]
                upstream_status = data["upstream_status"]
                upstream = data["upstream_addr"]

                status_list = [s.strip() for s in upstream_status.split(",")]
                addr_list = [a.strip() for a in upstream.split(",")]

                latest_status = status_list[-1]
                previous_status = status_list[0]
                previous_upstream = addr_list[0]
                current_upstream = addr_list[-1]

                recent.append((time.time(), pool, upstream_status))

                # Failover detection
                if upstream_status.startswith("5") and time.time() - last_alert_time["failover"] > ALERT_COOLDOWN:
                    last_alert_time["failover"] = time.time()
                    send_slack_alert(
                        f"⚠️ *Failover Detected!*\n"
                        f"• Previous Pool: `{last_pool}`\n"
                        f"• New Pool: `{pool}`\n"
                        f"• Release: `{release}`\n"
                        f"• Upstream: `{current_upstream}`\n"
                        f"• Status: `{latest_status}`"
                    )

                # Traffic pool switch
                elif pool != last_pool and time.time() - last_alert_time["switch"] > ALERT_COOLDOWN:
                    last_alert_time["switch"] = time.time()
                    send_slack_alert(
                        f"🔄 *Traffic Switch Detected!*\n"
                        f"• `{last_pool}` → `{pool}`\n"
                        f"• Release: `{release}`\n"
                        f"• Upstream: `{current_upstream}`\n"
                        f"• Status: `{latest_status}`"
                    )

                last_pool = pool

                if time.time() - last_check >= CHECK_INTERVAL:
                    check_alert()
                    last_check = time.time()

            except Exception as e:
                logging.error(f"Error processing log line: {e}")
                time.sleep(2)

3. Main Application Entry Point

if __name__ == "__main__":
    try:
        monitor_log()
    except KeyboardInterrupt:
        send_slack_alert("Log monitor stopped manually by user.")
        logging.info("Log watcher stopped manually.")
    except Exception as e:
        send_slack_alert(f"Log monitor stopped due to error: `{e}`")
        logging.error(f"Log watcher stopped due to error: {e}")

Add this requests to your requirements.txt file.

4. Running the Watcher in a Sidecar Container

The watcher runs alongside Nginx using a sidecar pattern. A sidecar container is a secondary container that runs alongside a main application container. The shared log directory allows the watcher application to read Nginx logs in real time, as shown in the part of the Docker Compose file below

monitor:
    image: python:3.11-slim
    container_name: monitor
    volumes:
      - ./logs:/var/log/nginx
      - ./log_watcher/watcher.py:/app/watcher.py
      - ./log_watcher/requirements.txt:/app/requirements.txt
    environment:
      - SLACK_WEBHOOK_URL=${SLACK_WEBHOOK_URL}
      - ACTIVE_POOL=${ACTIVE_POOL}
      - ERROR_RATE_THRESHOLD=${ERROR_RATE_THRESHOLD}
      - WINDOW_SIZE=${WINDOW_SIZE}
      - ALERT_COOLDOWN=${ALERT_COOLDOWN}
      - MAINTENANCE_MODE=${MAINTENANCE_MODE}
      - CHECK_INTERVAL=${CHECK_INTERVAL}
      - NGINX_LOG_PATH=/var/log/nginx/access.log
    working_dir: /app
    command: >
      /bin/sh -c "pip install -r requirements.txt && python watcher.py"

networks:
  default:
    driver: bridge
volumes:
  nginx_logs:

5. Run the Docker Compose file

After adding the monitor service in the Docker Compose file, see the previous article to see the existing Docker Compose file. Start up the application by running docker compose up -d --build.

6. Simulate Monitoring

To simulate the monitoring and alerting, we will use a bash script below that will send requests to the http://localhost:8080/version endpoint 120 times and after every 3 requests, and it will introduce chaos in the system by triggering the chaos endpoint POST http://localhost:8081/chaos/start?mode=error. This will ensure that there is enough number of default server failures that will be above the threshold in order to trigger a high failure alert message.

#!/bin/bash

BASE_NGINX="http://localhost:8080/version"
CHAOS_ON="http://localhost:8081/chaos/start?mode=error"
CHAOS_OFF="http://localhost:8081/chaos/stop"
TOTAL_REQUESTS=120
TOGGLE_INTERVAL=3  # Toggle error every 3 requests
IN_ERROR_MODE=false

echo "Starting Chaos Simulation..."
echo "Sending $TOTAL_REQUESTS requests to $BASE_NGINX with chaos toggled every $TOGGLE_INTERVAL requests"
echo ""

for ((i=1; i<=TOTAL_REQUESTS; i++)); do
  # Toggle chaos mode every N requests
  if (( i % TOGGLE_INTERVAL == 0 )); then
    if [ "$IN_ERROR_MODE" = true ]; then
      echo -e "\n[$i] Turning OFF error mode on Blue..."
      if curl -s -X POST "$CHAOS_OFF" > /dev/null; then
        IN_ERROR_MODE=false
      else
        echo "Failed to stop chaos."
      fi
    else
      echo -e "\n[$i] Turning ON error mode on Blue..."
      if curl -s -X POST "$CHAOS_ON" > /dev/null; then
        IN_ERROR_MODE=true
      else
        echo "Failed to start chaos."
      fi
    fi
  fi

  # Send request to Nginx (load balancer)
  HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" "$BASE_NGINX")
  if [[ "$HTTP_STATUS" == "200" ]]; then
    echo "[$i] Status: $HTTP_STATUS"
  else
    echo "[$i] Error: $HTTP_STATUS"
  fi

  # Sleep 0.1 second
  sleep 0.1
done

# Ensure chaos mode is turned off at the end
echo -e "\n Stopping any remaining chaos mode..."
curl -s -X POST "$CHAOS_OFF" > /dev/null || echo "Cleanup failed."

echo -e "\n Simulation complete!"

The screenshots for the high error rate, failover and server switch when the blue server is healthy are shown below

Conclusion

This monitoring and alerting system enhances Blue/Green deployments by providing real-time insights into application stability. It ensures that:

Failovers are detected immediately
Traffic switches are tracked
High error rates trigger alerts
Your team can respond quickly to maintain maximum uptime

This is just a simplified monitoring and alerting system. Advanced observability tools like Grafana Loki, Prometheus, etc, can be used for more complex systems.

Blue/Green Deployment with Nginx Upstream

Oluchi Oraekwe — Tue, 09 Dec 2025 11:46:57 +0000

Introduction

Ensuring minimal downtime and maintaining continuous service availability are crucial aspects of delivering software applications to users. Users expect a seamless experience at all times, even when failures occur behind the scenes. Servers or applications may fail due to high traffic, malfunctioning code, or unexpected system errors. When these situations arise, it is essential to keep services running with as little interruption as possible.

In this article, we will explore how to maintain application availability using the Blue/Green Deployment strategy. As the name suggests, this method relies on two environments: Blue and Green. These environments may be two separate servers or two instances of the same application running on a single server.

To demonstrate this concept, we will run two versions of our application, blue and green, inside two separate Docker containers. The Blue server will act as the primary server, while the Green server will function as the backup. The following sections provide a walkthrough of the setup.

Nginx Server

Nginx is a high-performance, open source web server commonly used for serving static content, reverse proxying, load balancing, and caching. In this setup, Nginx will function as a reverse proxy and a load balancer, receiving all user requests and forwarding them to either the Blue or Green application server.

In the configuration file (nginx.conf.template), the upstream servers are defined using Nginx’s upstream block. There are two upstream configurations:

blue_primary: Blue is active, Green is backup
green_primary: Green is active, Blue is backup
The ACTIVE_POOL environment variable controls which upstream group is selected.

user  nginx;
worker_processes  auto;
pid /var/run/nginx.pid;

events {
    worker_connections 1024;
}

http {

    log_format main '$remote_addr - $remote_user [$time_local] '
        '"$request" status=$status body_bytes_sent=$body_bytes_sent '
        'pool=$upstream_http_x_app_pool release=$upstream_http_x_release_id '
        'upstream_status=$upstream_status upstream_addr=$upstream_addr '
        'request_time=$request_time upstream_response_time=$upstream_response_time';

    error_log /var/log/nginx/error.log warn;
    access_log /var/log/nginx/access.log main;

    upstream blue_primary {
        server app_blue:4000 max_fails=1 fail_timeout=5s;
        server app_green:4000 backup;
    }

    upstream green_primary {
        server app_green:4000 max_fails=1 fail_timeout=5s;
        server app_blue:4000 backup;
    }

    map "${ACTIVE_POOL}" $active_backend {
        default     blue_primary;
        blue        blue_primary;
        green       green_primary;
    }

    server {
        listen 80;

        location / {
            proxy_pass http://$active_backend;
            proxy_pass_header X-App-Pool;
            proxy_pass_header X-Release-Id;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            proxy_connect_timeout 1s;
            proxy_send_timeout 1s;
            proxy_read_timeout 1s;
            proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
            proxy_next_upstream_tries 2;
        }
    }
}

For simulation purposes, the proxy timeouts were set to very low values to trigger fast failover during testing.

Upstream Servers

In Nginx terminology, upstream servers are the backend servers that Nginx forwards client requests to. Nginx does not execute application logic; rather, it simply accepts incoming traffic and routes it to one or more upstream servers defined in its configuration.

An upstream group may contain:

One or multiple application servers
A load-balancing strategy
Failover rules
Backup servers that only activate when the primary becomes unavailable

In a Blue/Green deployment, the upstream servers represent the two versions of the application. Nginx sends traffic to the active server (Blue or Green), based on the ACTIVE_POOL variable, while the other server remains on standby. This guarantees seamless switching and minimises downtime during failures or deployments.

The upstream servers in our setup are two identical Docker containers created from the same Docker image. This ensures consistency and prevents users from experiencing different behaviours during failover.

Below is the Docker Compose configuration managing Nginx, Blue, and Green containers:

services:
  nginx:
    image: nginx:latest
    container_name: nginx
    volumes:
      - ./nginx.conf.template:/etc/nginx/nginx.conf.template:ro
      - ./logs:/var/log/nginx
    ports:
      - "8080:80"
    environment:
      - ACTIVE_POOL=${ACTIVE_POOL}
    command: >
       /bin/sh -c "envsubst '$$ACTIVE_POOL' < /etc/nginx/nginx.conf.template > /etc/nginx/nginx.conf && nginx -g 'daemon off;'"
    depends_on:
      - app_blue
      - app_green

  app_blue:
    image: ${BLUE_IMAGE}
    container_name: app_blue
    ports:
      - "8081:${PORT}"
    environment:
      - PORT=${PORT}
      - APP_POOL=${ACTIVE_POOL}
      - RELEASE_ID=${RELEASE_ID_BLUE}

  app_green:
    image: ${GREEN_IMAGE}
    container_name: app_green
    ports:
      - "8082:${PORT}"
    environment:
      - PORT=${PORT}
      - APP_POOL=green
      - RELEASE_ID=${RELEASE_ID_GREEN}

For this demonstration, the headers app_pool and release_idhelp differentiate between the two servers.

Step by Step Guide

1. Set up Docker Compose

Create your Docker services as shown in the Docker Compose file above: Blue (active) and Green (backup), both running identical applications.

2. Define environment variables

These ensure unique identifiers for each server:

BLUE_IMAGE=yimikaade/wonderful:devops-stage-two
GREEN_IMAGE=yimikaade/wonderful:devops-stage-two
ACTIVE_POOL=blue
RELEASE_ID_BLUE=blue:v1.0.0
RELEASE_ID_GREEN=green:v1.0.0
PORT=4000

3. Configure Nginx

Use the provided template to dynamically generate the active configuration using envsubst.

4. Start the services

Ensure Docker and Docker Compose are running, then start the system:
docker compose up -d --build

5. Test the Nginx endpoint

Visit: http://<IP>:8080. If you are running on your local machine, the IP is localhost or 127.0.0.1

6. Check the version headers

Access: http://<IP>:8080/version to check the version and the server headers. The response is shown below;

7. Introduce failure on the Blue server

Trigger chaos mode: POST http://<IP>:8081/chaos/start?mode=timeout or POST http://<IP>:8081/chaos/start?mode=error to render the blue server not to be reached by nginx
Example response:

{
    "message": "Simulation mode 'error' activated"
}

8. Verify automatic failover

Check the version endpoint again; you should now see the Green server is responding to requests.

9. Stop the chaos simulation

POST http://<IP>:8081/chaos/stop
The Blue server becomes active again to serve requests.

10. Shut down your environment

Shut down your environment

Conclusion

This walkthrough demonstrates how to implement a Blue/Green deployment strategy with automatic failover using Nginx as a reverse proxy and load balancer. For failover to work correctly, ensure that the backup server is marked as backup in the Nginx configuration.

In the next article, I will explain how to monitor which server is active at any given time.

Before concluding, I want to briefly introduce backend.im, a developer-friendly platform for deploying applications using the Claude Code CLI directly from your desktop. It integrates seamlessly with Claude CLI, allowing you to provision infrastructure and deploy code with just a few commands. I will cover this in more detail soon.

Creating Bash Scripts for User Management

Oluchi Oraekwe — Mon, 01 Jul 2024 19:47:55 +0000

Introduction

A Bash script is a file containing a sequence of commands that are executed on a Bash shell. It allows performing a series of actions or automating tasks. This article will examine how to use a Bash script to create users dynamically by reading a CSV file. A CSV (Comma Separated Values) file contains data separated by commas or other delimiters.

In Linux systems, multiple users can access the same machine, necessitating the creation of various users with access to the machine. This article demonstrates a Bash script for creating users from a CSV file, assigning them to groups, and generating random passwords.

Objective

The main purpose of this article is to show how to create users in a Linux system using a simple bash script.

Bash Script Overview

We will follow a sequence of steps to create the Bash script. You can clone the full code from this GitHub repository.

Reading the CSV File

The first step is to read the CSV file. The Bash script will take the CSV file as an input argument. The script will throw an error and exit if no input is provided. Below is the code snippet for checking the file:

CSV_FILE="$1"

# Check if the file exists
if [[ ! -f "$CSV_FILE" ]]; then
    echo "File not found!"
    exit 1
fi

Creating a Directory and File to Store Users

After reading the file, we need to create a directory and file to store the usernames and passwords of any new users created which will be accessible only to the file owner. To avoid creating a directory that already exists, we first check for its existence before creating a new one:

PASSWD_DIR="/var/secure"
PASSWD_FILE="user_passwords.csv"

if [ ! -d "$PASSWD_DIR" ]; then
    mkdir -p "$PASSWD_DIR"
    touch "$PASSWD_DIR/$PASSWD_FILE"
    chmod 600 "$PASSWD_DIR/$PASSWD_FILE"
fi

Reading the Usernames and Group Names

After checking the text file and creating the file for storing the users, we will iterate through the rows to create users and assign them to groups. While creating the users, a random password is generated for each user. The code snippet below shows how users are created:

LOG_PATH="/var/secure/user_management.txt"

# Split the user and group by ";"
while IFS=';' read -r user groups; do
    user=$(echo $user | tr -d ' ')
    groups=$(echo $groups | tr -d ' ')

    # Check if the user exists
    if id "$user" &>/dev/null; then
        echo "$(date "+%Y-%m-%d %H:%M:%S") user $user already exists." >> "$LOG_PATH"
    else
        # Generate random password
        echo "$(date "+%Y-%m-%d %H:%M:%S") user with username $user already exist." >> $logPath

        # Create user and assign password
        useradd -m "$user"
        echo "$user:$password" | chpasswd

        # Store the created user and password
        echo "$user,$password" >> "$PASSWD_DIR/$PASSWD_FILE"
        echo "$(date "+%Y-%m-%d %H:%M:%S") user with username: $user cretaed by user $(whoami)" >> $logPath
    fi

    # Split the groups by comma and add user to each group
    IFS=',' read -ra GROUP_ARRAY <<< "$groups"
    for group in "${GROUP_ARRAY[@]}"; do
        # check for the existense of group before creating group
         if [ $(getent group $group) ]; then
            echo "$(date "+%Y-%m-%d %H:%M:%S") group: $group already exists." >> $logPath
         else
            groupadd $group
            echo "$(date "+%Y-%m-%d %H:%M:%S") group: $group created by user $(whoami)."  >> $logPath
        fi

        # check for the existense of user in a group before addding the user
        if getent group "$group" | grep -qw "$user"; then
            echo "$(date "+%Y-%m-%d %H:%M:%S") user: $user is already in group: $group"  >> $logPath
        else
            adduser $user $group
            echo "$(date "+%Y-%m-%d %H:%M:%S") user with username: $user was added to group :$group by user $(whoami)" >> $logPath
        fi


done < "$CSV_FILE"

Important Notes

Separators: The CSV file uses ; to separate usernames and groups, and , to separate multiple groups.
Logging: All actions taken concerning user creation are logged to a file with timestamps.

Running the Script

To run the script, follow these steps:

Ensure you are running on a Linux system with root privileges or use the sudo command.
Clone the repository and navigate to the directory.

Create a sample CSV file users.csv as shown below.

mary; developer,sys-admin
paul; sys-admin
peter; operations

Execute the script as shown add sudo if you are not a root user:
```
bash create_users.sh users.csv
```

After running the script, new users will be created, and their details will be stored in /var/secure/user_passwords.csv. All actions will be logged in /var/secure/user_management.txt.

Summary

The steps outlined in this article are not exhaustive other sources of information such as documentation can be used to complement your article.
To learn more about bash scripting join us at HNG Internship or subscribe and become a premium member by joining HNG Premium