DEV Community: Nonso Echendu

From Code to Production: A DevOps Journey with Docker, Traefik, and Modern Monitoring 🚀

Nonso Echendu — Fri, 11 Apr 2025 18:40:14 +0000

Introduction

Hey builders! 👋

So I recently took on an exciting challenge: transforming a full-stack FastAPI and React application into a production-ready system with robust monitoring. While the application itself was already well-structured, my role was to bring it to production standards through proper containerization, orchestration, and monitoring/observability. Let me walk you through this journey.

Want to see the complete code? Check out the GitHub repository.

📋 Prerequisites

Some things you'll need to have for this project:

Docker and Docker Compose installed. You can check the official documentation on how to install them on ubuntu.
A domain name (for SSL/TLS setup). I recommend getting from Hostinger
Basic understanding of:

Docker containers and images

Reverse proxies

Monitoring concepts

Linux command line

Sufficient system resources:

At least 4GB RAM

2 CPU cores

20GB storage

Cloned the github repo containing the frontend, backend, db .
Firewall configured to allow necessary ports (22, 3000, 9090, etc.)

The Challenge 🎯

When I first looked at the project, I saw a typical full-stack application with:

A FastAPI backend
A React frontend
PostgreSQL database
Basic authentication

My mission? Transform this into a production-grade system with:

Proper containerization
Automated SSL/TLS
Comprehensive monitoring
Efficient log management
Zero-downtime deployments

The Solution Architecture

I designed a modern DevOps architecture that looks like this (if you've been following my articles recently, you'll know i like this types of diagrams now 😅):

┌─────────────────────────────────────────────────────────────────────────┐
│                           Client Requests                               │
└───────────────────────────────┬─────────────────────────────────────────┘
                                │
                        ┌───────▼───────┐
                        │    Traefik    │
                        │  Reverse Proxy│
                        │   (SSL/TLS)   │
                        └───────┬───────┘
                                │
        ┌───────────────────────┼───────────────────────┐
        │                       │                       │
┌───────▼───────┐       ┌───────▼───────┐       ┌───────▼───────┐
│    Frontend   │       │    Backend    │       │    Adminer    │
│  (React/Nginx)│◄─────►│   (FastAPI)   │◄─────►│  (DB Admin)   │
└───────┬───────┘       └───────┬───────┘       └───────┬───────┘
        │                       │                       │
        │                       │                       │
        │                ┌──────▼───────┐               │
        └───────────────►│  PostgreSQL  │◄──────────────┘
                         │   Database   │
                         └──────┬───────┘
                                │
                         ┌──────▼───────┐
                         │  Monitoring  │
                         │    Stack     │
                         └──────┬───────┘
                                │
    ┌─────────────┬─────────────┼─────────────┬────────────┐
    │             │             │             │            │
┌───▼─────┐   ┌───▼─────┐   ┌───▼───┐    ┌────▼─────┐  ┌───▼──────┐
│cAdvisor │   │Promtail │   │Loki   │    │Prometheus│  │Grafana   │
│Container│   │Logs     │   │Log    │    │Metrics   │  │Dashboards│
│Metrics  │   │Collector│   │Storage│    │Database  │  │& Alerts  │
└─────────┘   └─────────┘   └───────┘    └──────────┘  └──────────┘

Application in Action

Containerization Strategy: Building Efficient Images 🐳

Multi-stage Builds: Optimizing Image Size

So let's look at how I achieved proper containerization.

One of the first challenges I faced was keeping the Docker images small in size and efficient. That's where multi-stage builds came in. Let me show you how I implemented this for both frontend and backend:

Frontend Container

# Build stage
FROM node:18-alpine as build
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
RUN npm run build

# Production stage
FROM nginx:alpine
COPY --from=build /app/dist /usr/share/nginx/html
COPY nginx.conf /etc/nginx/conf.d/default.conf
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

Why did I adopt using multi-stage?

It reduces final image size by excluding build tools.
For the frontend imagine the image size reducing from a whooping ~590mb with a single-stage build to a 50.1mb image size from the multi-stage build.
Using multi-stage also separates build dependencies from runtime dependencies
Improves security by reducing attack surface
Makes for faster deployments due to smaller images

Single-stage build...

vs Multi-stage build...

Backend Container

I also applied multi-stage build here:

# Build stage
FROM python:3.11-slim AS builder

WORKDIR /app

# Install only necessary build dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    libpq-dev \
    && rm -rf /var/lib/apt/lists/*

# Install specific version of poetry with export support
RUN pip install --no-cache-dir poetry==1.5.1

# Copy only dependency files first for better caching
COPY pyproject.toml poetry.lock* ./

# Generate requirements.txt - using the correct syntax for poetry export
RUN poetry export --without-hashes --format=requirements.txt > requirements.txt

# Copy the rest of the application
COPY . .

# Production stage
FROM python:3.11-slim
WORKDIR /app

# Install runtime dependencies only
RUN apt-get update && apt-get install -y --no-install-recommends \
    libpq5 \
    && rm -rf /var/lib/apt/lists/*

# Copy requirements and install directly with pip
COPY --from=builder /app/requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy application code
COPY --from=builder /app .

# Make startup script executable
RUN chmod +x /app/prestart.sh

# Set correct PYTHONPATH to ensure app imports work properly
ENV PYTHONPATH="${PYTHONPATH}:/app"

EXPOSE 8000
CMD ["bash", "-c", "cd /app && ./prestart.sh && uvicorn app.main:app --host 0.0.0.0 --port 8000"]

Database Management with Adminer

I added Adminer in my application stack in order to monitor the database. I also configured it to have secure access through Traefik.
Did this by simply added that Traefik label in the adminer service section in the docker-compose file.

adminer:
  image: adminer
  labels:
    - "traefik.enable=true"
    - "traefik.http.routers.adminer.rule=Host(`michaeloxo.tech`) && PathPrefix(`/adminer`)"
    - "traefik.http.routers.adminer.entrypoints=websecure"
    - "traefik.http.routers.adminer.tls=true"

Adminer also supports multiple database types so you can try using it too in your various application stacks.

Volume Management

For data persistence for grafana and prometheus, i implemented named volumes:

volumes:
  postgres_data:
    driver: local
  prometheus_data:
    driver: local
  grafana_data:
    driver: local

This helps persist data across container restarts.

Network Isolation

I implemented proper network isolation using Docker networks:

networks:
  app-network:
    driver: bridge

This will allow easy communication among the containers.

Health Checks

Every service includes health checks. The database, for example, has this health check:

healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB}"]
      interval: 5s
      timeout: 5s
      retries: 5

These health checks are important as they ensure that all the services are available, and we can detect problems early.

Detailed Component Breakdown: The Monitoring Stack 🔍

Now let's talk about observability and monitoring.

When I first approached this project, I knew I needed a monitoring solution that would be both powerful and maintainable.

After careful consideration, I settled on a modern stack that combines the best tools for container monitoring, metrics collection, and log aggregation. Let me walk you through each component and why I chose them.

The Foundation: cAdvisor and Container Metrics

The first piece of the puzzle was finding a way to monitor the containers effectively. That's where cAdvisor came in. What makes cAdvisor special is its zero-configuration approach - just mount the right volumes, and it starts collecting metrics automatically. In my setup, it watches over every container, tracking CPU usage, memory consumption, network I/O, and disk usage in real-time.

cadvisor:
  image: gcr.io/cadvisor/cadvisor:v0.47.0
  volumes:
    - /:/rootfs:ro
    - /var/run:/var/run:rw
    - /sys:/sys:ro
    - /var/lib/docker/:/var/lib/docker:ro

The beauty of cAdvisor lies in its simplicity. It exposes metrics in Prometheus format out of the box, making it a perfect fit for my monitoring stack. Every container's performance is now visible at a glance, helping us identify potential issues before they become problems.

Prometheus and Metrics Storage

For storing and querying our metrics, I chose Prometheus. Yeah I know, Prometheus is quite everyone's go-to when it come storing and collecting metrics. This is because of its pull-based architecture, which is more reliable than push-based systems, especially in containerized environments. My Prometheus configuration is quite clean and straightforward:

global:
  scrape_interval: 15s
  evaluation_interval: 15s     

scrape_configs:
  - job_name: 'prometheus'
    metrics_path: '/prometheus/metrics'
    static_configs:
      - targets: ['prometheus:9090']

  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8080']

The Log Aggregation Duo: Loki and Promtail

For log management, I implemented a combination of Loki and Promtail. This choice was driven by the need for a lightweight yet powerful logging solution. Unlike traditional ELK stacks that can be resource-intensive, Loki and Promtail provide efficient log aggregation with minimal overhead.

loki:
  image: grafana/loki:latest
  volumes:
    - ./loki/loki-config.yml:/etc/loki/config.yml

promtail:
  image: grafana/promtail:latest
  volumes:
    - ./promtail/promtail-config.yml:/etc/promtail/config.yml
    - /var/log:/var/log:ro
    - /var/lib/docker/containers:/var/lib/docker/containers:ro

The synergy between these tools is impressive. Promtail collects logs from both the system and containers, while Loki stores them efficiently. What's particularly useful is how they use the same labeling system as Prometheus, making it easy to correlate logs with metrics.

The Visualization Layer: Grafana

To bring all this data to life, I chose and implemented Grafana. I mean, Grafana just wonderfully ties everything together, providing beautiful dashboards and powerful querying capabilities. My Grafana setup is configured to work seamlessly with both Prometheus and Loki:

grafana:
  image: grafana/grafana:latest
  volumes:
    - grafana_data:/var/lib/grafana
  environment:
    - GF_SERVER_ROOT_URL=http://grafana:3000/grafana
    - GF_SERVER_SERVE_FROM_SUB_PATH=true

Here's how the grafana dashboards looks like in production:

The Traffic Manager: Traefik

Finally, to tie everything together, I implemented Traefik as my reverse proxy. This modern reverse proxy stands out for its automatic service discovery and dynamic configuration capabilities. My Traefik setup ensures secure access to all my monitoring tools. Just add these labels to the desired service in the docker-compose file:

labels:
  - "traefik.enable=true"
  - "traefik.http.routers.[service].rule=Host(`michaeloxo.tech`) && PathPrefix(`/[service]`)"
  - "traefik.http.routers.[service].entrypoints=websecure"
  - "traefik.http.routers.[service].tls=true"
  - "traefik.http.routers.[service].middlewares=global-middleware@file"

What makes this setup particularly effective is how all components work together.

cAdvisor collects metrics, Prometheus stores them, Loki and Promtail handle logs, Grafana visualizes everything, and Traefik ensures secure access. It's a well-oiled machine where each part plays its role perfectly.

The result? A comprehensive monitoring solution that provides real-time insights into our application's performance, helps us identify and troubleshoot issues quickly, and ensures we have the data we need to make informed decisions about our infrastructure.

(Btw, I recently implemented a similar but more complex monitoring stack on a product we're building, and ofc i wrote an article about it. Read it here )

Lessons Learned 📚

Throughout this DevOps implementation journey, I've gathered some valuable insights that are worth noting:

Traefik was like magic!

Automatic SSL Was a Game-Changer. Before Traefik, I spent hours manually configuring and renewing SSL certificates. I also never have to worry about certificate renewals again.
Middleware Chains Simplified Security. Creating reusable security configurations with middleware chains was a marvel. I could easily apply consistent security headers across all services with a single reference.

For containerization best practices

Multi-Stage Builds Transformed My Images

The satisfaction of seeing frontend image sizes drop from 590MB to 50MB (0ver 90%) was incredible
Eliminated unnecessary build tools from production images
Significantly improved deployment speed and reduced bandwidth usage

Implementing Proper Service Dependencies Using Healthchecks

One thing that could make your setup even better is implementing proper service dependencies with healthchecks for startup order.

This ensures services start in the right order and only when their dependencies are truly ready, not just running. It's eliminated those annoying "connection refused" errors during startup and makes the system much more resilient to restarts.

The biggest takeaway from this project was how the right tooling can transform complex tasks into manageable ones. Traefik turned what would have been several hours of reverse proxy configuration into minutes, while the monitoring stack gave me insights I didn't even know I needed until I had them.

Conclusion 🎉

Phew! What a rewarding journey this has been! Taking a regular app and turning it into a containerized, monitored production system was quite the adventure.

Is it perfect? Nah, nothing ever is. But that's the beauty of DevOps - it's all about continuous improvement. I'm still tinkering with container sizes, playing with alert thresholds, and learning new security tricks,. And tbh, that's the fun part!

The biggest win for me wasn't just getting everything up and running (though that felt amazing!), but seeing how all these pieces work together. Watching container metrics flow into Prometheus, visualizing them in Grafana, and catching issues before they become problems - it's like having superpowers lol.

I still have a list of improvements I want to make. Just thought of adding a ci/cd pipeline and integrating terraform & ansible to automate deployment, I know i'll still think of more.

But for now, I'm pretty good with this setup. It's running smoothly, it's secure, and most importantly - it's giving us the insights we need to keep getting better.

I do hope sharing my experience helps you in your own containerization and monitoring journey. Again don't forget to check out my article exclusively on monitoring here.

If you'd like to explore the complete implementation, the entire project is available on GitHub. I welcome your feedback, issues, and pull requests!

Till the next, happy building!

Building a Container System from Scratch: Understanding the Magic Behind Docker

Nonso Echendu — Fri, 28 Mar 2025 22:49:14 +0000

Hi builders!

I think this has been the shortest time intervals between any of my post (lol).

Well, today, I'm excited to share a project I've been working on (HNG thingy): building a container system from scratch!

Maybe when you hear of "container", Docker comes right to mind, yeah? Well, if you've ever wondered what's happening under the hood when you run docker run, this post is for you. I'll demystify containers by creating my own lightweight implementation that captures the core functionality of Docker.

🔍 Introduction: Why Build Your Own Container System?

Containers have transformed how we deploy and run applications, but they can seem like magic. By building our own container system, we can:

Gain a deep understanding of the core Linux technologies that power containers
Learn about isolation, resource control, and namespace concepts firsthand
Appreciate the engineering decisions behind production container systems
Build a foundation for more advanced container orchestration concepts

By the end of this guide, you'll have a functional container system capable of running processes in isolation with resource limits, networking, and other essential features.

📋 Prerequisites

Before diving in, ensure you have:

A Linux system (Ubuntu 20.04 or similar)
Root or sudo access
Basic knowledge of Python and Bash
Understanding of Linux processes and networking concepts
Necessary packages installed: python3, cgroups-tools, iptables

Btw, you can access all the scripts written for this project in my git GitHub repo here

🏗️ The Architecture: Understanding Our Container System

Our container implementation relies on these core components:

Python CLI Manager: Handles user commands and orchestrates container lifecycle

2. Bash Container Script: Implements the low-level container functionality

3. Linux Namespaces: For process, network, and filesystem isolation

4. Cgroups: To implement resource limits (CPU, memory)

5. Chroot: For filesystem isolation

┌─────────────────────────────────────────────────────────────┐
│                       User Commands                         │
│                                                             │
│  simple_container.py start|stop|list|logs                   │
└────────────────────────────┬────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────┐
│                 Python Container Manager                    │
│                                                             │
│  • Parses command line arguments                            │
│  • Manages container lifecycle                              │
│  • Tracks running containers                                │
│  • Sets up resource limits (cgroups)                        │
└────────────────────────────┬────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────┐
│                    container.sh Script                      │
│                                                             │
│  • Creates namespaces (process, network, mount)             │
│  • Sets up filesystem isolation (chroot)                    │
│  • Configures networking & port forwarding                  │
│  • Implements volume mounts                                 │
│  • Handles user isolation                                   │
└────────────────────────────┬────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────┐
│                      Linux Kernel                           │
│                                                             │
│  • Namespaces   • Cgroups     • Network Stack               │
│  • Filesystem   • Devices     • Process Management          │
└─────────────────────────────────────────────────────────────┘

🧠 Understanding Container Technologies: What Powers Our System

Let's demystify the key technologies that make containers possible:

(I'll be talking and elaborating on some of my code snippets in those scripts. so you can open the repo by the side, so you'll get the most of my explanations)

Namespaces: The Isolation Foundation

Linux namespaces are the cornerstone of container isolation, providing separate views of system resources:

PID Namespace: Gives containers their own process IDs, starting with PID 1
Network Namespace: Creates isolated network stacks with separate interfaces and routing tables
Mount Namespace: Isolates filesystem mount points between containers and host
UTS Namespace: Allows containers to have their own hostname and domain name
IPC Namespace: Isolates inter-process communication mechanisms
User Namespace: Maps user IDs between container and host, improving security

Our implementation uses unshare to create these namespaces, achieving process isolation:

unshare --mount --uts --ipc --pid --fork chroot "$ROOT_FS" bash -c "$WRAPPED_CMD"

Cgroups: Resource Control Made Simple

Control groups (cgroups) limit and account for resource usage:

CPU Limits: Prevent containers from hogging CPU resources
Memory Constraints: Protect the host from memory-hungry containers
Disk I/O Controls: Limit disk activity for fair resource sharing

Our implementation sets these limits using the cgroup filesystem:

def set_cpu_limit(container_name, cpu_limit):
    """Limit CPU usage for a container (percentage)"""
    # Create cgroup and set CPU limit
    # ...

Chroot: Filesystem Isolation

The chroot command changes the root directory for a process, creating filesystem isolation:

chroot "$ROOT_FS" bash -c "$WRAPPED_CMD"

This simple yet powerful mechanism ensures containers can't access files outside their designated root filesystem.

🚀 Part 1: Building the Container Manager (Python CLI)

Let's start by creating our Python CLI for managing containers:

#!/usr/bin/env python3
import argparse
import os
import subprocess
import sys
import json
import signal
import time

class SimpleContainer:
    def __init__(self):
        self.container_dir = "/var/run/simple-container"
        os.makedirs(self.container_dir, exist_ok=True)

    def start(self, name, command, cpu_limit=None, memory_limit=None, 
              network=True, volume=None, port=None, detach=True, use_userns=False):
        """Start a new container"""
        # Implementation details...

Our CLI supports these commands:

start: Launch a new container with specified resources
stop: Gracefully terminate a running container
list: Show all running containers and their details
logs: Display container logs for troubleshooting

🌐 Part 2: Creating Network Isolation

Networking is crucial for container functionality. Our implementation creates:

A network namespace for the container

2. Virtual Ethernet (veth) pairs to connect container and host

3. NAT rules for internet access

4. Port forwarding for service exposure

# Setup network namespace
ip netns add "$CONTAINER_NETNS"

# Create veth pair
ip link add "$VETH_HOST" type veth peer name "$VETH_CONTAINER"

# Move container end to namespace
ip link set "$VETH_CONTAINER" netns "$CONTAINER_NETNS"

# Configure interfaces and routing
# ...

This gives each container its own isolated network stack while maintaining connectivity to the outside world.

📦 Part 3: Implementing Filesystem Isolation and Volumes

Our container system supports both filesystem isolation and volume mounts:

Base filesystem: Using chroot with a minimal root filesystem

2. Overlay filesystem: For non-destructive modifications

3. Volume mounts: For sharing directories between host and container

# Mount essential filesystems
mount -t proc proc "$ROOT_FS/proc"
mount -t sysfs sysfs "$ROOT_FS/sys"

# Setup volume mounts
for volume in "${VOLUMES[@]}"; do
    host_path=$(echo "$volume" | cut -d: -f1)
    container_path=$(echo "$volume" | cut -d: -f2)
    mount -o bind "$host_path" "$ROOT_FS$container_path"
done

🔒 Part 4: User Isolation and Security

Security is essential for containers. Our implementation:

Creates a non-root container user (UID 1000)

2. Runs commands as this user inside the container

3. Sets up a minimal /dev environment

4. Manages permissions for mounted volumes

# Setup user isolation
cat > "$ROOT_FS/etc/passwd" <<EOF
root:x:0:0:root:/root:/bin/bash
container:x:1000:1000:container:/home/container:/bin/bash
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
EOF

# Run command as container user
su - container -c "$COMMAND"

⚙️ Part 5: Resource Limiting with Cgroups

To prevent containers from consuming excessive resources, we implement cgroup-based limits:

def set_memory_limit(container_name, memory_limit):
    """Limit memory usage for a container (in bytes)"""
    cgroup_path = f"/sys/fs/cgroup/memory/simple-container-{container_name}"
    os.makedirs(cgroup_path, exist_ok=True)

    # Set memory limit
    with open(f"{cgroup_path}/memory.limit_in_bytes", "w") as f:
        f.write(str(memory_limit))

    return cgroup_path

This prevents "noisy neighbor" problems and protects your host system from container resource abuse.

🔍 Part 6: Testing Container Isolation

Let's verify our container implementation works correctly:

Process Isolation: Container processes can't see host processes

./simple_container.py start --name test1 --command "ps aux"

2. Network Isolation: Container has its own network stack

./simple_container.py start --name test2 --command "ip addr && ping -c 1 8.8.8.8"

3. Filesystem Isolation: Container can't access host files

./simple_container.py start --name test3 --command "ls -la / && cat /etc/hostname"

4. Resource Limits: Container respects CPU and memory constraints

./simple_container.py start --name test4 --cpu 50 --memory 256M --command "stress --cpu 4"

🚀 Part 7: Real-World Application - Deploying a Todo App in Our Container

Let's put our container system to the test with a real-world application! We'll deploy a simple Flask Todo application inside our custom container, demonstrating how the concepts we've explored can be applied to practical use cases.

Setting Up a Fresh Environment

Starting with a fresh Ubuntu server, here's how to deploy a simple web application in our container system:

# First, update the system
sudo apt update && sudo apt upgrade -y

# Install required dependencies
sudo apt install -y python3 python3-pip python3-venv git

# Clone our container system repository
git clone https://github.com/NonsoEchendu/container-system
cd container-system

# Make the scripts executable
sudo chmod +x container.sh simple_container.py cgroups.sh

Preparing the Container Root Filesystem

Our container system needs a proper root filesystem. Let's prepare one.

But before then, we need to install a very important tool, debootstrap. Debootstrap is what will install a Debian base system into our root filesystem's directory.

sudo apt install debootstrap

Now we can continue with setting up our root filesystem.

# Create a directory for our container's root filesystem
sudo mkdir -p /home/ubuntu/todo-rootfs

# Prepare the root filesystem
sudo ./simple_container.py prepare-rootfs --target /home/ubuntu/todo-rootfs

Setting Up the Container Root Filesystem Repository Sources

We need to ensure the container has proper repository sources configured:

# Configure proper repository sources in the container
sudo bash -c 'cat > /home/ubuntu/odo-rootfs/etc/apt/sources.list << EOF
deb http://archive.ubuntu.com/ubuntu focal main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu focal-updates main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu focal-security main restricted universe multiverse
EOF'

# Update package lists with the new repositories
sudo chroot /home/ubuntu/todo-rootfs/ apt-get update

# Install Python and Flask in the existing root filesystem
sudo chroot /home/ubuntu/todo-rootfs apt-get install -y python3 python3-pip
sudo chroot /home/ubuntu/todo-rootfs pip3 install flask

Getting the Todo Application

Now let's get the Todo application:

# Clone the Todo app repository
git clone https://github.com/NonsoEchendu/simple-flask-todo

Creating and Starting the Container

With our root filesystem and application ready, let's create and start the container:

# Start the container with the Todo app
cd container-system

sudo ./simple_container.py create --name todo-container --rootfs /home/ubuntu/todo-rootfs 

sudo ./simple_container.py start \
    --name todo-container \
    --volume $(pwd)/../simple-flask-todo:/app \
    --port 8080:8080 \
    --command "cd /app && python3 app.py" \
    --cpu 50 \
    --memory 256M

A successful run should look like this on your terminal:

Accessing the Todo Application

Once the container is running, you can access the Todo app in your browser by going to http://your-server-ip:8080.

It should look like this:

Managing the Container

After successfully deploying the application, you can manage the container:

# Check the container status
sudo ./simple_container.py list

# View the application logs
sudo ./simple_container.py logs --name todo-container

# Stop the container when done
sudo ./simple_container.py stop --name todo-container

# Remove the container
sudo ./simple_container.py remove --name todo-container

🔥 Part 8: Challenges Faced and Overcome

I'll be very honest, building a container system from scratch wasn't without challenges. Let me tell you some of them:

Problem: Permission Problems with User Namespaces

One of the trickiest issues was handling permissions correctly when combining user namespaces with chroot:

unshare --user --map-root-user --mount chroot "$ROOT_FS" /bin/bash

(The above command will fail with "Permission denied")

Solution: I separated the concerns - using chroot as root but then switching to the container user afterward using su:

chroot "$ROOT_FS" su - container -c "$COMMAND"

Problem: Network Namespace Communication

Setting up proper communication between host and container network namespaces was challenging. My initial approach that caused connectivity issues was:

ip netns exec "$CONTAINER_NETNS" ip route add default via "$GATEWAY_IP"

Solution: I implemented a complete solution with:

Proper veth pair setup
Correct IP and routing configuration
NAT rules for outbound connections
Careful DNS configuration

Problem: Volume Mount Permission

Volume mounts created complex permission issues, particularly with nested directories:

mount -o bind "$HOST_PATH" "$ROOT_FS$CONTAINER_PATH"

Using the above command, volume permissions wouldn't match user expectations. Files that were created would have root ownership, not container user

Solution: I implemented special handling for volume permissions, ensuring volumes have appropriate permissions for container user:

if [ "$USE_USER_NS" == "true" ]; then
    # Make mount point accessible to container user
    chown -R 1000:1000 "$ROOT_FS$container_path"
    # Mount with specific options
    mount -o "$mount_opts" "$host_path" "$ROOT_FS$container_path"
fi

Problem: DNS Resolution Failures

DNS resolution inside containers was initially broken, preventing network connections. Container couldn't resolve external hostnames.

A command like, ping google.com would return "Unknown host".

Solution: I properly configured DNS by copying host resolver settings and ensuring proper access, like this:

setup_dns() {
    # Get host's DNS servers
    HOST_DNS=$(grep nameserver /etc/resolv.conf | awk '{print $2}' | head -n 1)
    if [ -z "$HOST_DNS" ]; then
        HOST_DNS="8.8.8.8"
    fi

    # Create resolv.conf with host's DNS
    cat > "$ROOT_FS/etc/resolv.conf" <<EOF
nameserver $HOST_DNS
nameserver 8.8.8.8
nameserver 8.8.4.4
EOF

    # Add DNS server IP to container's routing table
    ip netns exec "$CONTAINER_NETNS" ip route add $HOST_DNS via $(echo $HOST_IP | cut -d'/' -f1)
}

And some other problems i can't remember atm.

💡 Part 9: Lessons Learned

Building this container system taught me some valuable lessons:

Lesson 1: The power of of Linux fundamentals and building blocks

Lesson 2: Importance of Security Layering

Namespaces for isolation
Capability restrictions
User separation
Resource limits
Filesystem restrictions

Lesson 3: Abstractions Have Real Value

After implementing containers from scratch, I have newfound appreciation for the abstractions Docker provides. What seems like "magic" is actually careful engineering to hide complexity.

Lesson 4: Resource Management is Crucial

Containers without resource limits can easily disrupt host systems. Proper cgroup configuration is not optional but essential for production use.

🔮 Conclusion: From Understanding to Innovation

Congratulations! You've successfully built a container system that implements all the key features of production container runtimes:

✅ Process isolation with namespaces
✅ Network isolation and port forwarding
✅ Filesystem isolation and volume mounts
✅ Resource limits with cgroups
✅ User isolation for security

This journey has given you deep insights into how containers actually work, demystifying what often seems like magic.

Now you can:

Better understand Docker and Kubernetes internals
Debug container issues more effectively
Make informed decisions about container deployment
Potentially extend your implementation with more advanced features

Remember, while Docker and other production container systems are much more sophisticated, they're built on these same fundamental Linux primitives we've explored. By building our own implementation, we've peeled back the layers of abstraction to reveal the elegant simplicity at the core of container technology.

Till next time, happy building!

Modern Monitoring Approach: A Comprehensive Guide to Grafana, Prometheus, and DORA Metrics

Nonso Echendu — Fri, 21 Mar 2025 10:26:46 +0000

Hi builders.

Been a while i posted an article here. I've been so consumed with an internship program (HNG tech), where we're actually building a real-world product. Sneak peak: it's an AI grading system that schools and educational organisations can use. When it's out, i'll tell y'all about it :)

But today, i'll be talking about monitoring your web applications, which will save you time and money (application downtime will make you lose money).

This guide will walk you through building a powerful monitoring ecosystem that not only alerts you when things break but helps you understand why they break and how to improve your development practices.

🔍 Introduction: Why Monitoring Matters

A robust monitoring system is very important as it aids in:

Providing real-time visibility into system health
Alerting teams before small issues become major outages
Offering actionable insights for continuous improvement
Enabling data-driven decisions about infrastructure and development practices

By the end of this guide, you'll have implemented a comprehensive monitoring solution leveraging industry-standard tools like Prometheus, Grafana, and cutting-edge DORA metrics to transform how you understand and improve your applications.

📋 Prerequisites

Before diving in, ensure you have:

A Linux-based cloud server (I used Ubuntu 20.04, but any other linux distro works too)
Root or sudo access
Basic command line knowledge
GitHub Actions configured for your repositories
Firewall configured to allow necessary ports (22, 3000, 9090, etc.)
A Slack workspace

🏗️ The Architecture: Understanding Our Monitoring Stack

Our monitoring system consists of these tools:

Prometheus: To collect and store all our metrics
Grafana: Makes pretty, customizable dashboards that'll help you visualize your metrics
Node Exporter: Collects hardware server metriks (like CPU, memory, disk)
Blackbox Exporter: Checks if endpoints are actually responding. Will also check for SSL validity
DORA Metrics Exporter: Tracks development performance on our github repos using industry-standard metrics
AlertManager: Manages alert routing and delivery to Slack

┌───────────────────────────────────────────────────────────────┐
│                    Monitored Infrastructure                   │
│                                                               │
│  ┌────────────────┐    ┌────────────────┐   ┌──────────────┐  │
│  │ Node Exporter  │    │ Blackbox Probe │   │    GitHub    │  │
│  │(Server Metrics)│    │ (Website Tests)│   │(DORA Metrics)│  │
│  └───────┬────────┘    └───────┬────────┘   └───────┬──────┘  │
└──────────┼─────────────────────┼────────────────────┼─────────┘
           │                     │                    │
           └─────────┬───────────┴────────────────────┘
                     │
                     ▼
            ┌─────────────────┐
            │    Exporters    │
            │(Data Collectors)│
            └────────┬────────┘
                     │
                     ▼  
           ┌───────────────────┐
           │    Prometheus     │
           │(Metrics Database) │────────────────┐
           └─────┬─────────────┘                │
                 │                              │
        ┌────────▼─────┐                        │
        │ AlertManager │                        │
        │  (Alerts)    │                        │
        └────────┬─────┘                        │
                 │                              │
                 │                              │
        ┌────────▼──────┐                       │
        │   Slack       │                       │
        │(Notifications)│                       │
        └───────────────┘                       │
                                                │
                                                ▼
            ┌────────────────────────────────────────────────────┐
            │                 Grafana Dashboards                 │
            │  ┌───────────┐  ┌───────────┐  ┌───────────────┐   │
            │  │  System   │  │  Website  │  │ DORA Metrics  │   │
            │  │ Dashboard │  │ Dashboard │  │   Dashboard   │   │
            │  └───────────┘  └───────────┘  └───────────────┘   │
            └────────────────────────┬───────────────────────────┘
                                     │
                                     ▼
                            ┌─────────────────┐
                            │   Developers    │
                            │  & Operations   │
                            └─────────────────┘

🧠 Why I Chose These Specific Monitoring Tools

Our monitoring stack wasn't assembled randomly; each component was carefully selected to fulfil specific requirements in a modern infrastructure. Let's talk a bit about each of them.

Prometheus: The Foundation of Our Stack

I chose Prometheus as our core monitoring tool for several practical reasons:

Pull-based approach: Prometheus collects metrics by reaching out to our systems, which simplifies setup and improves reliability.
Simple yet powerful queries: PromQL lets us create meaningful alerts and visualizations without complex coding.
Built-in alerting: Prometheus detects issues based on our defined thresholds, sending these to AlertManager for notification.
Easy metric filtering: I can quickly analyze metrics by server, application, or any other label we've defined.
Perfect fit for our infrastructure: Works seamlessly with both our servers and web applications.

Grafana: Visualization That Drives Insights

I chose Grafana for visualization because it excels at:

Multi-source dashboards: Can combine Prometheus data with other sources like logs or traces.
Rich visualization options: From time-series graphs to heatmaps and histograms.
Alerting capabilities: Though I primarily use AlertManager, Grafana's alerting can provide an additional layer.
Annotation support: Allows marking deployments, incidents, and other events directly on dashboards.
User-friendly interface: Makes it accessible for both technical and non-technical stakeholders.

Node Exporter: Server-Level Insights

I implemented Node Exporter to:

Monitor system resources: Track CPU, memory, disk, and network usage on our servers.
Identify bottlenecks: Quickly pinpoint which resources are constraining performance.
Track system health: Get early warnings about server issues before they affect users.
Collect detailed metrics: Access hundreds of system-level data points for thorough analysis.

Blackbox Exporter: External Endpoint Monitoring

Blackbox Exporter was specifically chosen to monitor external endpoints and provides:

Uptime monitoring: Detects when websites and APIs are unavailable
Response time tracking: Measures latency that could affect user experience
SSL certificate validation: Alerts before certificates expire to prevent security warnings
HTTP status verification: Ensures services are returning proper response codes
Basic content validation: Can verify that responses contain expected content

AlertManager: Intelligent Alert Routing

AlertManager wasn't chosen merely for sending notifications, but for its sophisticated handling of alerts:

Grouping and deduplication: Prevents alert storms during major outages.
Silencing and inhibition: Reduces noise by temporarily muting alerts during maintenance or when a higher-priority alert is already firing.
Multiple receivers: Routes different alerts to appropriate teams or channels.
Templating: Creates rich, context-aware notifications that speed up troubleshooting.

DORA Metrics Exporter: Connecting Technical and Business Performance

This custom component bridges the gap between operational metrics and development performance, allowing teams to make data-driven decisions about process improvements.

📊 The Importance of Tracking DORA Metrics

DORA (DevOps Research and Assessment) metrics identifies four key metrics:

Deployment Frequency (DF): How often code is successfully deployed to production
Lead Time for Changes (LTC): The time it takes for a commit to reach production
Mean Time to Restore (MTTR): How quickly service can be restored after an incident
Change Failure Rate (CFR): The percentage of deployments that cause a failure

Our setup doesn't just passively collect these metrics — it makes them actionable:

Real-time visibility: Dashboards show current performance on all four metrics
Historical trends: Track improvements over time with historical data
Proactive alerts: Get notified when metrics fall outside healthy ranges
Correlation with system metrics: Understand how infrastructure affects delivery performance

For example, if your Change Failure Rate rises above a threshold, you'll be alerted before it becomes a serious problem. Similarly, if Mean Time to Restore grows too long, you can focus efforts on improving incident response.

By integrating DORA metrics into your monitoring system, you create a feedback loop that continuously improves both technical operations and business outcomes.

🚨 How Alerts Transform System Reliability

Now let's talk about alerting.

Alerts are far more than just notifications — they are a critical component of a reliability strategy that shifts teams from reactive to proactive operations. A well-configured alerting system fundamentally changes how teams approach reliability.

Here's what I mean. We'll compare the traditional approach (proactive alerts) and modern approach (with our system)

The Traditional Approach (Without Proactive Alerts)

System fails completely
Users report problems
Engineers scramble to diagnose unfamiliar issues
Teams work under pressure to restore service
Business impact accumulates during downtime

The Modern Approach (With Our Alert System)

System begins showing early warning signs
Alerts trigger before users are affected
Engineers investigate with detailed context already provided
Problems are resolved during their early stages
Many outages are prevented entirely

The alerts we've configured in this guide do more than just notify—they transform how teams interact with systems:

1. Context-Rich Notifications

Our alerts include:

Precise timing information
System relationships
Impact assessments
Historical context
Actionable next steps

This means engineers can begin addressing issues immediately rather than spending critical time gathering basic information.

2. Progressive Severity Levels

By distinguishing between warnings and critical alerts, the system creates a natural progression:

Warnings: Address during normal working hours to prevent future problems
Critical alerts: Require immediate attention to restore service

This tiered approach ensures appropriate response without creating false urgency.

Enough talk. Now let's roll up our sleeves and get these tools live and up and running on our servers (was that just tautology lol)

🔧 Part 1: Building the Foundation

Let's start by creating the directory structure and installing our components.

Directory Setup

sudo mkdir -p /etc/prometheus /etc/prometheus/rules /etc/alertmanager /etc/blackbox_exporter
sudo mkdir -p /var/lib/prometheus /var/lib/grafana /var/lib/alertmanager
sudo mkdir -p /opt/dora-exporter

Component Installation

Prometheus: Your Metrics Database

Install Prometheus (I'm using v2.47.0, but use whatever's current):

wget https://github.com/prometheus/prometheus/releases/download/v2.47.0/prometheus-2.47.0.linux-amd64.tar.gz
tar xvfz prometheus-*.tar.gz
sudo mv prometheus-2.47.0.linux-amd64/prometheus /usr/local/bin/
sudo mv prometheus-2.47.0.linux-amd64/promtool /usr/local/bin/

AlertManager: Your Alert Orchestrator

wget https://github.com/prometheus/alertmanager/releases/download/v0.26.0/alertmanager-0.26.0.linux-amd64.tar.gz
tar xvfz alertmanager-*.tar.gz
sudo mv alertmanager-0.26.0.linux-amd64/alertmanager /usr/local/bin/

Node Exporter: Hardware & OS Metrics

Install Node Exporter for server metrics:

wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz
tar xvfz node_exporter-*.tar.gz
sudo mv node_exporter-1.6.1.linux-amd64/node_exporter /usr/local/bin/

Blackbox Exporter: External Monitoring

Run these commands:

wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.24.0/blackbox_exporter-0.24.0.linux-amd64.tar.gz
tar xvfz blackbox_exporter-*.tar.gz
sudo mv blackbox_exporter-0.24.0.linux-amd64/blackbox_exporter /usr/local/bin/

DORA Metrics Exporter: Engineering Performance

For the DORA metrics exporter, we'll be using some custom scripts from this repo github.com/NonsoEchendu/dora-metrics (please follow me on github ;) ) that collects key performance metrics from GitHub repositories.

Clone the repo into your server:

git clone https://github.com/NonsoEchendu/dora-metrics

Copy all its contents to the /opt/dora-exporter directory I created.

sudo cp dora-metrics/* /opt/dora-exporter
cp .env.example .env
sudo cp .env /etc/default/dora-metrics

Create and edit the .env file:

cp .env.example .env
sudo cp .env /etc/default/dora-metrics
sudo nano /etc/default/dora-metrics

Create and prepare virtual environment where the dora-metrics python script will run:

sudo apt-get update
sudo apt-get install python3-venv
sudo python3 -m venv /opt/dora-exporter/venv 
sudo chown -R dora:dora /opt/dora-exporter/venv
sudo -u dora /opt/dora-exporter/venv/bin/pip install -r requirements.txt

Grafana: Your Visualization Platform

Let's install grafana now. Run these commands:

sudo apt-get install -y apt-transport-https software-properties-common wget
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install grafana

User Creation & Permissions

Security best practice: each component runs as its own user with limited permissions.

sudo useradd --no-create-home --shell /bin/false prometheus
sudo useradd --no-create-home --shell /bin/false alertmanager
sudo useradd --no-create-home --shell /bin/false node_exporter
sudo useradd --no-create-home --shell /bin/false blackbox
sudo useradd --no-create-home --shell /bin/false dora
sudo useradd --no-create-home --shell /bin/false grafana

sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus
sudo chown -R alertmanager:alertmanager /etc/alertmanager /var/lib/alertmanager
sudo chown -R blackbox:blackbox /etc/blackbox_exporter
sudo chown -R grafana:grafana /var/lib/grafana
sudo chown -R dora:dora /opt/dora-exporter

🚀 Part 2: Service Configuration

Now let's configure each service to run automatically at startup.

Prometheus Service

Create (/etc/systemd/system/prometheus.service) and write into it with this:

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --web.console.libraries=/usr/share/prometheus/console_libraries \
  --web.console.templates=/usr/share/prometheus/consoles \
  --web.enable-lifecycle

Restart=always

[Install]
WantedBy=multi-user.target

AlertManager Service

Create and write into this /etc/systemd/system/alertmanager.service

[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
ExecStart=/usr/local/bin/alertmanager \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --storage.path=/var/lib/alertmanager \
  --log.level=debug

Restart=always

[Install]
WantedBy=multi-user.target

Node Exporter Service

Create and write into this /etc/systemd/system/node-exporter.service:

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
ExecStart=/usr/local/bin/node_exporter \
  --collector.filesystem.ignored-mount-points="^/(sys|proc|dev|host|etc)($$|/)"

Restart=always

[Install]
WantedBy=multi-user.target

Blackbox Exporter Service

Create /etc/systemd/system/blackbox-exporter.service and write inti it with this:

[Unit]
Description=Blackbox Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=blackbox
Group=blackbox
ExecStart=/usr/local/bin/blackbox_exporter \
  --config.file=/etc/blackbox_exporter/blackbox.yml

Restart=always

[Install]
WantedBy=multi-user.target

DORA Metrics Service

Create /etc/systemd/system/dora-metrics.service and write inti it with this:

[Unit]
Description=DORA Metrics Exporter
After=network.target

[Service]
User=dora
Group=dora
EnvironmentFile=/etc/default/dora-metrics
WorkingDirectory=/opt/dora-exporter
ExecStart=/opt/dora-exporter/venv/bin/python3 /opt/dora-exporter/main.py
Restart=always

[Install]
WantedBy=multi-user.target

📝 Part 3: Configuration Files

Time to set up the actual monitoring configuration. This is where I tell our stack what to monitor and how to alert.

Prometheus Configuration

Create /etc/prometheus/prometheus.yml and write into it with this:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
      - targets: ['localhost:9093']        

# Load rules once and periodically evaluate them
rule_files:
  - "rules/node_exporter_alerts.yml"
  - "rules/blackbox_alerts.yml" 
  - "rules/dora_alerts.yml"

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node_exporter'
    scrape_interval: 10s
    static_configs:
      - targets: ['localhost:9100']

  # PM2 metrics from host (for NextJS apps)
  - job_name: 'pm2'
    scrape_interval: 10s
    static_configs:
      - targets: ['localhost:9209']

  # Blackbox exporter for HTTP/HTTPS uptime and SSL monitoring
  - job_name: 'blackbox_http'
    metrics_path: /probe
    params:
      module: [http_2xx]  # Look for a HTTP 200 response
    static_configs:
      - targets:
        - https://website-url
        # Add more URLs as needed
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115  # Blackbox exporter's address

  # Blackbox exporter for SSL monitoring
  - job_name: 'blackbox_ssl'
    metrics_path: /probe
    params:
      module: [tls]  # Use the TLS probe
    static_configs:
      - targets:
        - website-url:443
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115

  - job_name: 'dora-metrics'
    static_configs:
      - targets: ['localhost:8000']

Alert Rules

Server Health Alerts

Create /etc/prometheus/rules/node_exporter_alerts.yml and write into it with this:

groups:
- name: node-exporter
  rules:
  - alert: HighCPULoad
    expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: "High CPU load (instance {{ $labels.instance }})"
      description: "CPU load is > 80%\n  VALUE = {{ $value }}%\n  LABELS: {{ $labels }}"

  - alert: HighMemoryLoad
    expr: (node_memory_MemTotal_bytes - (node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes)) / node_memory_MemTotal_bytes * 100 > 80
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: "High memory load (instance {{ $labels.instance }})"
      description: "Memory load is > 80%\n  VALUE = {{ $value }}%\n  LABELS: {{ $labels }}"

  - alert: HighDiskUsage
    expr: (node_filesystem_size_bytes{fstype!~"tmpfs|fuse.lxcfs|squashfs|vfat"} - node_filesystem_free_bytes{fstype!~"tmpfs|fuse.lxcfs|squashfs|vfat"}) / node_filesystem_size_bytes{fstype!~"tmpfs|fuse.lxcfs|squashfs|vfat"} * 100 > 85
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High disk usage (instance {{ $labels.instance }})"
      description: "Disk usage is > 85%\n  VALUE = {{ $value }}%\n  LABELS: {{ $labels }}"

Website Availability Alerts

Create /etc/prometheus/rules/blackbox_alerts.yml and write into it with this:

groups:
- name: blackbox-exporter
  rules:
  - alert: EndpointDown
    expr: probe_success == 0
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "Endpoint down (instance {{ $labels.instance }})" 
      description: "Endpoint {{ $labels.instance }} is down\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"

  - alert: SlowResponseTime
    expr: probe_duration_seconds > 1
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Slow response time (instance {{ $labels.instance }})"
      description: "Response time is > 1s\n  VALUE = {{ $value }}s\n  LABELS: {{ $labels }}"

  - alert: SSLCertExpiringSoon
    expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 30
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "SSL certificate expiring soon (instance {{ $labels.instance }})"
      description: "SSL certificate expires in less than 30 days\n  VALUE = {{ $value }} days\n  LABELS: {{ $labels }}"

  - alert: SSLCertExpired
    expr: probe_ssl_earliest_cert_expiry - time() <= 0
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "SSL certificate expired (instance {{ $labels.instance }})"
      description: "SSL certificate has expired\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"

DORA Performance Metrics

Create /etc/prometheus/rules/dora_alerts.yml and write into it with this:

groups:
- name: dora-metrics
  rules:
  - alert: HighChangeFailureRate
    expr: (sum(increase(github_deployment_failed_total[7d])) / sum(increase(github_deployment_total[7d]))) * 100 > 15
    for: 1h
    labels:
      severity: warning
    annotations:
      summary: "High change failure rate"
      description: "Change failure rate is > 15% over the last 7 days\n  VALUE = {{ $value }}%"

  - alert: LongMeanTimeToRestore
    expr: avg(github_incident_mttr_seconds) / 60 > 60
    for: 1h
    labels:
      severity: warning
    annotations:
      summary: "Long mean time to restore"
      description: "Mean time to restore is > 60 minutes\n  VALUE = {{ $value }} minutes"

  - alert: LowDeploymentFrequency
    expr: sum(increase(github_deployment_total[7d])) < 3
    for: 1d
    labels:
      severity: warning
    annotations:
      summary: "Low deployment frequency"
      description: "Less than 3 deployments in the last 7 days\n  VALUE = {{ $value }} deployments"

AlertManager Configuration

In creating the alertmanager file, we need to have the slack webhook url. Now we can actually pass the url directly into the config file, but that's a huge security risk; and Slack will deactivate it after a while.

So what we'll do instead is to save the webhook url into a file, give it needed permissions and pass the file into our alertmanager config file.

Create /etc/alertmanager/slack_api_url and write ONLY your webhook url into it.

Next give the file only read permissions for users:

chmod 644 /etc/alertmanager/slack_api_url

Next, create /etc/alertmanager/alertmanager.yml and write into it with this:

global:
  resolve_timeout: 5m
  slack_api_url_file: '/etc/alertmanager/slack_api_url'

route:
  group_by: ['alertname', 'instance', 'job']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'slack-notifications'
  routes:
    - match:
        severity: critical
      receiver: 'slack-critical'
    - match:
        severity: warning
      receiver: 'slack-warning'

receivers:
  - name: 'slack-notifications'
    slack_configs:
      - send_resolved: true
        channel: '#devops-alerts'
        title: '{{ if eq .Status "firing" }}🔴 ALERT{{ else }}🟢 RESOLVED{{ end }}: {{ .CommonLabels.alertname }}'
        text: |
          {{ if eq .Status "firing" }}*SYSTEM ALERT*{{ else }}*SYSTEM RECOVERED*{{ end }}

          {{ range .Alerts }}
          *{{ .Annotations.summary }}*
          {{ .Annotations.description }}

          *⏰ Incident Details:*
          • Started: {{ .StartsAt }}
          • Status: {{ .Status | toUpper }}

          *🔍 Technical Information:*
          • System: {{ .Labels.instance }}
          • Job: {{ .Labels.job }}
          • Severity: {{ .Labels.severity }}

          *👥 Impact Assessment:*
          • Users affected: {{ if eq .Labels.job "blackbox_http" }}Website visitors{{ else }}Service users{{ end }}

          *👥 Team to Notify:* @devops-team
          {{ end }}
        icon_emoji: '{{ if eq .Status "firing" }}:red_circle:{{ else }}:green_circle:{{ end }}'

  - name: 'slack-critical'
    slack_configs:
      - send_resolved: true
        channel: '#devops-alerts'
        title: '{{ if eq .Status "firing" }}🔴 CRITICAL{{ else }}🟢 RESOLVED{{ end }}: {{ .CommonLabels.alertname }}'
        text: |
          {{ if eq .Status "firing" }}*CRITICAL SYSTEM ALERT*{{ else }}*SYSTEM RECOVERED*{{ end }}

          {{ range .Alerts }}
          *{{ .Annotations.summary }}*
          {{ .Annotations.description }}

          *⏰ Incident Details:*
          • Started: {{ .StartsAt }}
          • Status: {{ .Status | toUpper }}

          *🔍 Technical Information:*
          • System: {{ .Labels.instance }}
          • Job: {{ .Labels.job }}
          {{ if eq .Labels.job "blackbox_http" }}• Error: Connection failed
          • HTTP Status: No response{{ end }}

          *👥 Impact Assessment:*
          • Severity: Critical
          • User Impact: {{ if eq .Labels.job "blackbox_http" }}All website users affected{{ else }}Service degradation{{ end }}

          *🚨 Attention:* <@U089ZLRDV1N> <@U08BD88M87J> <@U08AN9DLDMH> <@U08AQNQVC8L> <@U08B8JT5RAN> <@U08ANPWRP9Q> <@U08A81WGZHV> <@U08A8QAN2P9> <@U08AP0AGQ9Z> <@U08AMPHCPSN>
          {{ end }}
        icon_emoji: '{{ if eq .Status "firing" }}:fire:{{ else }}:white_check_mark:{{ end }}'
        link_names: true

  - name: 'slack-warning'
    slack_configs:
      - send_resolved: true
        channel: '#devops-alerts'
        title: '{{ if eq .Status "firing" }}⚠️ WARNING{{ else }}🟢 RESOLVED{{ end }}: {{ .CommonLabels.alertname }}'
        text: |
          {{ if eq .Status "firing" }}*WARNING ALERT*{{ else }}*WARNING RESOLVED*{{ end }}

          {{ range .Alerts }}
          *{{ .Annotations.summary }}*
          {{ .Annotations.description }}

          *⏰ Incident Details:*
          • Started: {{ .StartsAt }}
          • Status: {{ .Status | toUpper }}

          *🔍 Technical Information:*
          • System: {{ .Labels.instance }}
          • Job: {{ .Labels.job }}
          {{ if eq .Labels.alertname "SlowResponseTime" }}• Response Time: {{ if eq .Labels.job "blackbox_http" }}Slow{{ end }}{{ end }}
          {{ if eq .Labels.alertname "SSLCertExpiringSoon" }}• Certificate Expires: Soon{{ end }}
          {{ if eq .Labels.alertname "HighCPULoad" }}• CPU Load: High{{ end }}
          {{ if eq .Labels.alertname "HighMemoryLoad" }}• Memory Use: High{{ end }}
          {{ if eq .Labels.alertname "HighDiskUsage" }}• Disk Usage: High{{ end }}

          *👥 Impact Assessment:*
          • Severity: Warning
          • User Impact: Potential performance degradation

          *💡 Recommended Actions:*
          {{ if eq .Labels.alertname "SlowResponseTime" }}Check database queries or high backend resource usage.{{ else if eq .Labels.alertname "SSLCertExpiringSoon" }}Renew SSL certificate before expiration.{{ else if eq .Labels.alertname "HighCPULoad" }}Identify CPU-intensive processes and optimize.{{ else if eq .Labels.alertname "HighMemoryLoad" }}Check for memory leaks or increase available memory.{{ else if eq .Labels.alertname "HighDiskUsage" }}Clean up disk space or expand storage.{{ end }}

          *🚨 Attention:* <@U089ZLRDV1N> <@U08BD88M87J> <@U08AN9DLDMH> <@U08AQNQVC8L> <@U08B8JT5RAN> <@U08ANPWRP9Q> <@U08A81WGZHV> <@U08A8QAN2P9> <@U08AP0AGQ9Z> <@U08AMPHCPSN>
          {{ end }}
        icon_emoji: '{{ if eq .Status "firing" }}:warning:{{ else }}:white_check_mark:{{ end }}'
        link_names: true

Blackbox Exporter Configuration

Create /etc/blackbox_exporter/blackbox.yml and write into it with this:

modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: [200]
      method: GET
      follow_redirects: true
      preferred_ip_protocol: "ip4"
      tls_config:
        insecure_skip_verify: false

  http_post_2xx:
    prober: http
    timeout: 5s
    http:
      method: POST
      preferred_ip_protocol: "ip4"

  tcp_connect:
    prober: tcp
    timeout: 5s

  tls:
    prober: tcp
    timeout: 5s
    tcp:
      tls: true
      preferred_ip_protocol: "ip4"
      tls_config:
        insecure_skip_verify: false

🔔 Part 4: Integrating with Slack

Real-time notifications are crucial for fast incident response. Let's set up Slack integration:

Create a Slack App:

Go to Slack API Apps
Click "Create New App" → "From scratch"
Name your app (e.g., "DeploymentMonitor")
Select your workspace

2. Enable Incoming Webhooks:

Navigate to "Incoming Webhooks" in the sidebar
Toggle "Activate Incoming Webhooks" to ON
Click "Add New Webhook to Workspace"
Select the channel for receiving alerts
Copy the Webhook URL and replace the placeholder in your alertmanager.yml

3. Configure Alert Formatting:

Our configuration creates informative, well-formatted alerts
Critical alerts use 🔴 while resolved alerts use 🟢
Alerts include relevant links and contextual information

An example of a slack alert

🖥️ Part 5: Launching Your Monitoring Stack

Time to bring everything online:

# Enable services to start at boot
sudo systemctl daemon-reload
sudo systemctl enable prometheus alertmanager node-exporter blackbox-exporter grafana-server dora-metrics

# Start all services
sudo systemctl start prometheus alertmanager node-exporter blackbox-exporter grafana-server dora-metrics

# Verify everything is running
sudo systemctl status prometheus alertmanager node-exporter blackbox-exporter grafana-server dora-metrics

If all was setup well, it should show success messages like this:

📊 Part 6: Setting Up Grafana Dashboards

Now for the exciting part - visualizing all this data!

1. Access Grafana:

Open your browser to http://:3000
Default credentials are admin/admin
You'll be prompted to set a new password

2. Add Prometheus as a Data Source:

Click the gear icon (Configuration) → Data Sources
Click "Add data source" and select "Prometheus"
Set the URL to http://localhost:9090
Click "Save & Test"

3. Import Pre-built Dashboards:

Click "+" → "Import"
Enter dashboard ID:
Node Exporter: 1860
Blackbox Exporter: 7587
Select Prometheus as the data source
Click "Import"

Node Exporter dashboard

Blackbox Dashboard

4. Import DORA Metrics Dashboard:

Use the JSON file from NonsoEchendu/dora-metrics
This provides detailed visualization of your software delivery performance

📈 Part 7: Understanding DORA Metrics

Our monitoring setup includes tracking of DORA metrics, which are industry-standard measures of development team performance:

Deployment Frequency (DF): How often you successfully release to production
Lead Time for Changes (LTC): Time from code commit to production deployment
Change Failure Rate (CFR): Percentage of deployments causing a failure in production
Mean Time to Restore (MTTR): How quickly service is restored after an incident

Dora Metrics dashboard

These metrics help you understand:

How responsive your development process is
How stable your changes are
How quickly you can recover from failures

The alerts we've configured will notify you when these metrics fall outside healthy ranges, enabling continuous improvement of your development practices.

🔮 Conclusion: Building a Data-Driven Culture

Yipeee! You've successfully implemented a comprehensive monitoring solution that provides:

Real-time visibility into system performance and availability
Early warnings for potential issues
Insights into your development practices
Clear metrics for measuring improvements

By combining system monitoring with DORA metrics, you've created a foundation for a truly data-driven engineering culture. Use these insights to fuel your next improvements, and watch your team's effectiveness and your application's reliability grow together.

Till the next, happy building!

Building a FastAPI Book API with CI/CD Pipelines (Using Github Actions) and Docker Deployment

Nonso Echendu — Thu, 13 Feb 2025 15:33:10 +0000

In this article, I’ll walk you through the development of a FastAPI-based book API, complete with a Continuous Integration (CI) and Continuous Deployment (CD) pipeline. This project was part of a Devops challenge with HNG, and is aimed to retrieve book details by ID, and deployed to an AWS ec2 instance using Docker.

Project Overview

The repository provided already included a predefined structure, and we had to ensure that all endpoints were correctly implemented and accessible via Nginx.

These were the goals to be achieved in this challenge:

Add an endpoint to retrieve a book by its ID, so that it is accessible via this url path /api/v1/books/{book_id}.
Set up a CI pipeline that automates tests whenever a pull request is made to the main branch.
Set up a CD pipeline that automates the deployment process whenever changes are pushed to the main branch.
Serve the application over NGINX

Project Structure

fastapi-book-project/
├── api/
│   ├── db/
│   │   ├── __init__.py
│   │   └── schemas.py      # Data models and in-memory database
│   ├── routes/
│   │   ├── __init__.py
│   │   └── books.py        # Book route handlers
│   └── router.py           # API router configuration
├── core/
│   ├── __init__.py
│   └── config.py           # Application settings
├── tests/
│   ├── __init__.py
│   └── test_books.py       # API endpoint tests
├── main.py                 # Application entry point
├── requirements.txt        # Project dependencies
└── README.md

Prerequisites

Docker. To install docker, check their official documentation.

Setting Up

The initial step was cloning the provided git repo, which contained the basic project structure. Here's a link to the project's git repo, https://github.com/NonsoEchendu/fastapi-book-project.

To clone:

git clone https://github.com/NonsoEchendu/fastapi-book-project.git

Challenge Task #1

So starting with the first task in the challenge: Add an endpoint to retrieve a book by its ID.

To do this, I need to go to the books.py file found in the routes/ folder, and add the /api/v1/books/{book_id} endpoint.

I'm also going to add a get_book function to retrieve the book details by ID. And if the book is not found, the function throws a 404 Not Found error.

Endpoint definition:

@router.get(
    "/{book_id}", response_model=Book, status_code=status.HTTP_200_OK
)
async def get_book(book_id: int) -> Book:
    book = db.books.get(book_id)
    if not book:
        raise HTTPException(status_code=404, detail="Book not found")
    return book

Next, i tested if the endpoint I just added is working.

To test, i'll be starting the application using Uvicorn:

uvicorn app.main:app --reload

Then, access the endpoint on my browser url at http://127.0.0.1:8000/api/v1/books/1.

It works!

Dockerizing the Application

Before moving to the next tasks for the challenge - creating CI/CD pipelines, i'll want to containerize the app using docker.

So first, i'll be creating a Dockerfile in the project root's directory. The Dockerfile will be structured to install dependencies, set up a non-root user, and start the app using Uvicorn.

Here's the Dockerfile:

FROM python:3.9-slim

WORKDIR /app

COPY requirements.txt .

RUN pip install --no-cache-dir -r requirements.txt

COPY . .

EXPOSE 8000

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

Now, we can build the application into a docker image and run it inside a container while maintaining a clean and isolated environment.

To build the image, while in the project's root directory, run:

docker build -t fastapi-book .

To run the app in a container:

docker run -d -p 8000:8000 fastapi-book

You should be able to also access it on your browser url at http://127.0.0.1:8000/api/v1/books/1.

Challenge Task #2

Moving to the next task: Setting up a CI (Continuous Integration) pipeline.

This CI pipeline will simply automate a pytest whenever a pull request is made to the main branch. These tests are important as they ensure that any changes to the codebase do not break existing functionality.

The CI pipeline:

name: CI Pipeline

on:
  pull_request:
    branches:
      - main

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout Repository
        uses: actions/checkout@v3

      - name: Set Up Python
        uses: actions/setup-python@v4
        with:
          python-version: "3.9"

      - name: Install Dependencies
        run: |
          python -m venv venv
          source venv/bin/activate
          pip install --upgrade pip
          pip install -r requirements.txt

      - name: Run Tests
        run: |
          source venv/bin/activate
          pytest --maxfail=1 --disable-warnings

Challenge Task #3

Setting up a CD (Continuous Deployment) pipeline

This CD pipeline will automate the deployment process whenever changes are pushed to the main branch.

The CD pipeline:

name: Deployment Pipeline

on:
  push:
    branches:
      - main

jobs:
  deploy:
    runs-on: ubuntu-latest

    env:
          SSH_PRIVATE_KEY: ${{ secrets.SSH_PRIVATE_KEY }}
          SERVER_IP: ${{ secrets.SERVER_IP }}
          USER: ${{ secrets.SERVER_USER }}

    steps:
      - name: Checkout Repository
        uses: actions/checkout@v3

      - name: Debug Environment Variables
        run: |
          echo "Server IP: $SERVER_IP"
          echo "User: $USER"

      - name: Set Up SSH
        run: |
          # Setup SSH
          mkdir -p ~/.ssh 
          echo "$SSH_PRIVATE_KEY" | tr -d '\r' > ~/.ssh/id_rsa
          chmod 600 ~/.ssh/id_rsa
          ssh-keyscan -H $SERVER_IP >> ~/.ssh/known_hosts

      - name: Prepare EC2 Instance & Pull Latest Code
        run: |
          ssh $USER@$SERVER_IP << 'EOF'
            set -e  # Stop script if any command fails

            # Ensure project directory exists
            if [ ! -d "/home/$USER/fastapi-book-project" ]; then
              git clone https://github.com/NonsoEchendu/fastapi-book-project.git /home/$USER/fastapi-book-project
            fi

            # Go to project folder
            cd /home/$USER/fastapi-book-project

            # Fetch latest changes
            git reset --hard  # Ensure a clean state
            git pull origin main
          EOF

      - name: Install Docker
        run: |
          ssh $USER@$SERVER_IP << 'EOF'
            sudo apt-get update
            sudo apt-get install -y docker.io 
            sudo usermod -aG docker $USER
          EOF

          ssh $USER@$SERVER_IP "docker ps"

      - name: Deploy With Docker 
        run: |
          ssh $USER@$SERVER_IP << 'EOF'
            cd /home/$USER/fastapi-book-project
            docker stop fastapi-app || true
            docker rm fastapi-app || true
            docker build -t fastapi-app .
            docker run -d --name fastapi-app -p 8000:8000 fastapi-app
          EOF

This setup ensures that changes pushed to the main branch were automatically pulled into the server. It also builds the image and runs the application in a Docker container.

The use of secrets also ensures secure authentication without exposing sensitive credentials.

To setup the repository secrets in your Github Actions, go to the github repository settings. Then go to **Secrets and variables**, then click on **Actions**. Next, click on **New repository secret** and add the 3 repo secrets we used in the CD pipeline, namely, SERVER_IP, SERVER_USER, SSH_PRIVATE_KEY.

Before creating these secrets, please ensure to already create an AWS ec2 instance. The values assigned to these secrets will be from the running ec2 instance.

Challenge Task #4

And now the last task - Serving the application over NGINX.

I want to use NGINX as a reverse proxy to route traffic from my ec2 instance's public ip address to my FastAPI application running on port 8000.

Inside my ec2 instance ubuntu server, i'll first install and enable Nginx using:

sudo apt-get update

sudo apt-get nginx

sudo systemctl start enable nginx

Then i add the Nginx configuration file:

sudo nano etc/nginx/sites-available/fastapi-app

And add this:

server {
    listen 80;
    server_name yourdomain.com;  # Replace with your domain or server IP

    location / {
        proxy_pass http://127.0.0.1:8000/;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Next, i enable the configuration by creating a symlink, and remove the default site:

sudo ln -s /etc/nginx/sites-available/fastapi /etc/nginx/sites-enabled/

sudo rm /etc/nginx/sites-enabled/default

Then i can restart NGINX:

sudo systemctl restart nginx

Now i can access the endpoint directly without adding 8000 to the url, like this: http://ec2-public-ip/api/v1/books/{book_id}.
{book_id} here being a positive integer.

Conclusion

In summary, what was I able to achieve in this project?

I built a FastAPI-based book API that retrieves book details by ID. I also made the application production-ready by containerizing it using Docker and used Nginx as a reverse proxy.

I also implemented CI/CD pipelines using GitHub Actions ensuring that the application is thoroughly tested and automatically deployed whenever changes are pushed to the main branch.

If you’re interested in exploring the code, check out the GitHub repository. Feel free to fork it, experiment, and adapt it to your needs!

Till my next project, happy building! ✌🏽

Setting up Nginx Server

Nonso Echendu — Wed, 29 Jan 2025 11:32:10 +0000

Objective

Hello builders! Today we'll be looking at something basic, i.e setting up and configuring an nginx web server.

We'll be using it to host a very simple webpage.

Prerequisites

An Ubuntu Server

I'll be using AWS to quickly spin up an Ubuntu server using an ec2 instance.

You can check out the official AWS documentation on how to create an Ubuntu ec2 instance here.

Because Nginx listens on port 80 (HTTP), i'll also be creating an inbound rule to allow traffic to port 80.

Also allow traffic to port 22, so that we can SSH into the ubuntu server to configure Nginx.

Installing Nginx

Now that we've got our Ubuntu server running, SSH into the just the ubuntu server, using:

ssh -i <your-pem-key-file> ubuntu:<your-ec2-public-ip>

Next, let's install Nginx.

First, we'll update Ubuntu's package lists:

sudo apt update && sudo apt upgrade -y

The -y command-line option is to auto-approve and skip the prompt.

Install Nginx:

sudo apt install nginx -y

Next we'll enable and start the installed Nginx service

sudo systemctl enable nginx
sudo systemctl start nginx

Let's verify our Nginx configuration and check if Nginx is running:

sudo systemctl status nginx

If successfully installed and running, you should see an output like this:

Setting Up Nginx

Now that we've installed Nginx successfully, let's set up a custom webpage that'll be shown by default when we visit our public ip address on any browser.

We'll be creating a custom index.html file in /var/www/html/index.html:

nano /var/www/html/index.html

Then we create a basic html template. Paste the following line of code into the index.html file:

<!DOCTYPE html>
<html>
<head>
    <title>A Custom Nginx Page</title>
</head>
<body>
    <h1>Welcome to DevOps Stage 0 - [Nonso Echendu]/Michaelo_0x]</h1>
</body>
</html>

Next, we want to configure our nginx server, such that it listens for HTTP requests on port 80, and will serve files from /var/www/html.

Let's edit the Nginx default site configuration file:

sudo nano /etc/nginx/sites-available/default

Replace everything in it with this:

server {
    listen 80 default_server;

    root /var/www/html;
    index index.html;

    server_name _;

    location / {
        try_files $uri $uri/ =404;
    }
}

Let me explain some key lines in the above configuration.

listen 80 default_server tells Nginx to listen for incoming connections on port 80.

The default_server flag means that if a request comes in and it doesn't match any other server block, this server block will respond by default.

root /var/www/html; defines the root directory that Nginx should go to look for files to serve. Remember, that we have our custom index .html file in that directory - /var/www/html.

index index.html specifies when anyone goes to your webpage http://<your-public-ip>, it should serve the default index.html file which we created earlier.

Now let's restart our Nginx server for all these changes to take effect.

Restart Nginx with this command:

sudo systemctl restart nginx

Re-confirm the Nginx server is running by using:

sudo systemctl status nginx

The output should be something like this:

And that's it! We have our Nginx server configured to serve a custom webpage.

Let's now confirm all that we've done visually.

Get your ec2 instance public ip, go to a web browser, and visit http://<your-public-ip>. You should see something like this:

Conclusion

And so there we have it. I was able to show how to install and setup an Nginx on a fresh Ubuntu server.

I also showed how to a create a custom HTML web page and configure Nginx to serve that webpage by default.

P.s.
This mini project is a task given to me by HNG Intenship - an internship program that will be running for a couple of months.

If you're looking to hire Devops engineers (and related fields) do check these out:

DevOps Engineers

Cloud Engineers

Site Reliability Engineers

Platform Engineers

Infrastructure Engineers

Kubernetes Specialists

AWS Solutions Architects

Azure DevOps Engineers

Google Cloud Engineers

CI/CD Pipeline Engineers

Monitoring/Observability Engineers

Automation Engineers

Docker Specialists

Linux Developers

PostgreSQL Developers

Setting up a VPC Infrastructure For Jenkins, Artifactory, Sonarqube on AWS using Terraform

Nonso Echendu — Thu, 23 Jan 2025 22:37:36 +0000

Hey builders! So this is an article very much related to a previous one, where I wrote about setting up a VPC infrastructure on AWS but using the AWS console UI.

In this article, however, we'll be using Terraform, an IAC tool, to automate the setup and configuration of the vpc infrastructure and instances.

Here's a link to this terraform project's github repo - https://github.com/NonsoEchendu/terraform-for-aws-instances

You see, as Devops engineers, our job is to automate things, and make them work faster and seamlessly. And that's what makes using Terraform efficient. Because with just one command we can create and configure the whole infrastructure, as well as delete them all at once.

Alright, let's dive in...

Prerequisites

Before we get into the terraform script though, here are some prerequisites you must have:

Install terraform
Install AWS CLI
Set up the AWS CLI with an IAM user having sufficient permissions (e.g., AdministratorAccess), using this command
```
aws configure
```

Then fill in your AWS credentials.

Objective

We want to create EC2 instances for a Jenkins, Artifactory and Sonarqube servers. All resources will be in one VPC. The Jenkins server will be in a public subnet and the Artifactory and Sonarqube will both be in a private subnet.

By placing Artifactory and Sonarqube in a private subnet, we are not exposing them directly to the public or the internet, thus reducing the risk of unauthorized access or attacks.

Repository Structure

AWS Architecture Diagram

This is an architectural diagram of what we want to do or achieve.

Terraform Script

Now let's take a look at the terraform scripts that'll be doing all the work for us.

1. The providers.tf file.

This script defines and configures the AWS provider for Terraform.

provider "aws" {
  region = "us-east-1"
}

The provider "aws" declares that terraform will interact with AWS services by using the AWS provider plugin. It allows Terraform to create and manage resources in AWS, such as EC2 instances, VPCs, etc.

The region = "us-east-1" specifies that all AWS resources in this Terraform configuration will be created in the us-east-1 region.

2. The variables.tf file.

variable "cidr" {
  default = "10.0.0.0/16"
}

This creates a variable cidr, with a default value of 10.0.0.0/16. We'll be calling this variable in the main terraform script.

3. The main.tf file.

Now this is a lengthy script, 300+ lines, but we'll be taking it bit by bit.

Again, you can find the whole script in the project repo here.

Alright, let's start with the first resource, which creates a VPC.

resource "aws_vpc" "main_vpc" {
  cidr_block = var.cidr
  instance_tenancy = "default"
  tags = {
    Name = "javaVPC"
  }
}
We're giving it the name main_vpc, and that's what we'll be using to reference the VPC throughout the script when attaching subnets, instances and the likes.

cidr_block = var.cidr. We're assigning the value of cidr_block using the variable we defined earlier in the variables.tf file.

The CIDR block defines the range of private IP addresses available for use within the VPC. For a /16 block, you get 65,536 IP addresses.

Then, we also give it a tag with the name javaVPC (the 'java' there is cause the script was originally written to run a java app). This tag will help you identify the VPC in the AWS Console.

Next, let's create an internet gateway. This is an important component because that's what will allow our resources like ec2 instances within our VPC to access the internet.

resource "aws_internet_gateway" "main_igw" {
  vpc_id = aws_vpc.main_vpc.id
  tags = {
    Name = "javaVpcInternetGateway"
  }
}
We're giving it the name main_igw and attaching it to the VPC we created earlier.

Without an Internet Gateway, the VPC would remain isolated, and no outbound or inbound traffic to/from the internet would be possible.

Next, we'll be creating route tables - a public and a private one. First, the public route table.

resource "aws_route_table" "public_rt" {
  vpc_id = aws_vpc.main_vpc.id
  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.main_igw.id
  }
  tags = {
    Name = "javaPublicRouteTable"
  }
}
So we're creating a public route table and attaching it to the vpc. While there's no terraform resource explicitly for public route table, it is how we configure the route table that makes it public or private.

The cidr_block is set to 0.0.0.0/0 meaning we're creating a route entry for all internet-bound traffic. Then we're also directing the traffic to the internet gateway we created earlier. With this, subnets associated with this route table can communicate with the internet.

Next, let's create a public subnet. This will be what will host our Jenkins server and Bastion host (yeah :) i know we never mentioned Bastion, we'll get to it), that need direct internet access.

resource "aws_subnet" "public_subnet" {
  vpc_id                  = aws_vpc.main_vpc.id
  cidr_block              = "10.0.1.0/24"
  availability_zone       = "us-east-1a"
  map_public_ip_on_launch = true
  tags = {
    Name = "javaPublicSubnet"
  }
}
Again, we're attaching this subnet to our vpc (all resources created in this project are being attached to this vpc).

For the cidr_block, IP address range, 10.0.1.0/24, allocates 256 IP addresses (from 10.0.1.0 to 10.0.1.255).

Then we're also placing this subnet in the us-east-1a availability zone for fault tolerance and high availability.

map_public_ip_on_launch set to true automatically assigns a public IP address to instances launched in this subnet (Jenkins, Bastion host), as both instances need direct internet access.

We still haven't attached or associated this public subnet to the public route table we created earlier. There's another terraform resource for that.

So we create a route_table_association Terraform resource, that will simply associate our public subnet to the public route table.

resource "aws_route_table_association" "public_rta" {
  subnet_id      = aws_subnet.public_subnet.id
  route_table_id = aws_route_table.public_rt.id
}

Let's move to creating a private route table and subnet.

First, though, we need to create 2 things - an elastic IP and a NAT gateway. Why though? I'll explain.

For security reasons, we don't want our Artifactory and Sonarqube instances to have direct public internet access. But they still need to access the internet to download updates, plugins, or dependencies.

So we'll be creating an Elastic IP which we will assign to a NAT gateway.

The Elastic IP address is used specifically for the NAT Gateway, not for the private instances like Artifactory and Sonarqube.

The purpose of the Elastic IP for the NAT Gateway is to provide a static, public IP address that the NAT Gateway can use to enable internet access for the resources in the private subnet.

Without an Elastic IP, the NAT Gateway would be assigned a dynamically allocated public IP address, which could change over time. And so by associating an Elastic IP with the NAT Gateway, the public IP address remains constant, which is important for maintaining reliable internet connectivity for the resources in the private subnet.

The private instances, like Artifactory and Sonarqube, do not need public IP addresses assigned to them directly. They reside in the private subnet and will then access the internet through the NAT Gateway, using the Elastic IP.

The NAT gateway will then be placed in the public subnet which has access to the internet.

If it's a bit confusing, just please look at the architecture diagram again.

Here's the terraform configuration for this:
# Elastic IP for NAT Gateway
resource "aws_eip" "nat_eip" {
  domain = "vpc"
}
# NAT GAteway
resource "aws_nat_gateway" "main_nat" {
  allocation_id = aws_eip.nat_eip.id
  subnet_id     = aws_subnet.public_subnet.id
  tags = {
    Name = "NatGateway"
  }
}

Next, we can create our private route table and attach to our vpc

resource "aws_route_table" "private_rt" {
  vpc_id = aws_vpc.main_vpc.id
  tags = {
    Name = "javaPrivateRouteTable"
  }
}

Then, we need to create a route in the private route table that directs all internet-bound traffic through the NAT gateway.

Basically, this is to allow resources in the private subnet, such as the Artifactory and Sonarqube servers, to access the internet indirectly through the NAT Gateway, without having a direct public IP address.

Aws route configuration:
resource "aws_route" "private_nat" {
  route_table_id         = aws_route_table.private_rt.id
  destination_cidr_block = "0.0.0.0/0"
  nat_gateway_id         = aws_nat_gateway.main_nat.id
}
destination_cidr_block = "0.0.0.0/0" sets the destination CIDR block for the route to 0.0.0.0/0, which represents all internet traffic (i.e., any destination outside the VPC).

Next, we create a private subnet and associate it with the private route table.

# Private Subnet
resource "aws_subnet" "private_subnet" {
  vpc_id            = aws_vpc.main_vpc.id
  cidr_block        = "10.0.2.0/24"
  availability_zone = "us-east-1a"
  tags = {
    Name = "javaPrivateSubnet"
  }
}
# Private Route Table Association with Private Subnet
resource "aws_route_table_association" "private_rta" {
  subnet_id      = aws_subnet.private_subnet.id
  route_table_id = aws_route_table.private_rt.id
}

Now we're moving to creating the instances.

We start with the Jenkins server

But first, let's create a Security Group for the Jenkins instance.

resource "aws_security_group" "jenkins_sg" {
  name   = "jenkins_sg"
  vpc_id = aws_vpc.main_vpc.id
  tags = {
    Name = "jenkins_sg"
  }
}

Then, we're going to add inbound and outbound rules to this security group.

# Jenkins Security Group Inbound Rule 1
resource "aws_vpc_security_group_ingress_rule" "jenkins_sg_inbound_rule1" {
  security_group_id = aws_security_group.jenkins_sg.id
  cidr_ipv4         = "0.0.0.0/0"
  from_port         = 22
  ip_protocol       = "tcp"
  to_port           = 22
}
# Jenkins Security Group Inbound Rule 2
resource "aws_vpc_security_group_ingress_rule" "jenkins_sg_inbound_rule2" {
  security_group_id = aws_security_group.jenkins_sg.id
  cidr_ipv4         = "0.0.0.0/0"
  from_port         = 8080
  ip_protocol       = "tcp"
  to_port           = 8080
}
# Jenkins Security Group Outbound Rule 1
resource "aws_vpc_security_group_egress_rule" "outbound_rule1" {
  security_group_id = aws_security_group.jenkins_sg.id
  cidr_ipv4         = "0.0.0.0/0"
  ip_protocol       = "-1"
}

Basically, we're setting inbound rules to allow access to ports 22 (for SSH) and 8080 (Jenkins HTTP UI).

cidr_ipv4 is set to 0.0.0.0/0 meaning any ip address is allowed to access these ports.

Next, let's create the ec2 instance for Jenkins itself.

resource "aws_instance" "jenkins_server" {
  ami                    = "ami-04b4f1a9cf54c11d0"
  instance_type          = "t2.micro"
  key_name = "new-test-key-pair"
  vpc_security_group_ids = [aws_security_group.jenkins_sg.id]
  subnet_id              = aws_subnet.public_subnet.id
  user_data              = filebase64("./jenkins_user_data.sh")
  tags = {
    Name = "JenkinsServer"
  }
}
You can change your own AMI id to one of your choice. The one stated above is for Ubuntu 24.04 x86 architecture.

key_name argument requires that you assign an already created key pair. In this case, i have already created a key pair from my AWS console named new-test-key-pair.

We also attach this instance to the Jenkins security group we created earlier. Also attaching the instance to the public subnet.

We're also making use of user_data arg. This will help us to pass a script to our Jenkins EC2 instance during its launch. The jenkins_user_data.sh script installs docker and docker compose, and spins up a Jenkins container from a jenkins docker image.

To get the user-data scripts check the project repository under the user-data-scripts directory.

And so we're done with the Jenkins instance.

Let's move to the next ec2 instances - Artifactory and Sonarqube.

But before that, we have to create one instance before them.

2. The Bastion Host. The one mentioned earlier.

But why do we need to create this Bastion host?

Well, the Bastion host will serve as a secure gateway in the public subnet that provides SSH access to instances in private subnets. It will act as the single entry point to SSH into our private Artifactory and Sonarqube instances.

Let's create first, the Bastion Security Group, and add inbound and outbound rules.

resource "aws_security_group" "bastion_sg" {
  name   = "bastion_sg"
  vpc_id = aws_vpc.main_vpc.id
  tags = {
    Name = "bastion_sg"
  }
}
# Bastion Security Group Inbound Rule 1
resource "aws_vpc_security_group_ingress_rule" "bastion_sg_inbound_rule1" {
  security_group_id = aws_security_group.bastion_sg.id
  cidr_ipv4         = "0.0.0.0/0"
  from_port         = 22
  ip_protocol       = "tcp"
  to_port           = 22
}
# Bastion Security Group Outbound Rule 1
resource "aws_vpc_security_group_egress_rule" "bastion_outbound_rule1" {
  security_group_id = aws_security_group.bastion_sg.id
  cidr_ipv4         = "0.0.0.0/0"
  ip_protocol       = "tcp"
  from_port         = 22
  to_port           = 22
}
# Bastion Security Group HTTPS Outbound Rule 
resource "aws_vpc_security_group_egress_rule" "bastion_outbound_https" {
  security_group_id = aws_security_group.bastion_sg.id
  cidr_ipv4         = "0.0.0.0/0"
  ip_protocol       = "tcp"
  from_port         = 443
  to_port           = 443
}
With the ingress or inbound rule, we're simply allowing access to port 22 (for SSH).

cidr_ipv4 is set to 0.0.0.0/0 meaning any ip address is allowed to access this port.

Note: Replace "0.0.0.0/0" with a more restrictive CIDR block that only includes trusted IP addresses. For example, your trusted IP range. Allowing 0.0.0.0/0 means any device connected to the internet can attempt to connect to the bastion host using SSH.

We also set some egress or outbound rules. Both rules allow Bastion Security Group to send outbound traffic on ports 22 (SSH) and 443 (HTTPS) to any IP address (0.0.0.0/0).

Next, we create the Bastion instance:

resource "aws_instance" "bastion_host" {
  ami                    = "ami-04b4f1a9cf54c11d0"
  instance_type          = "t2.micro"
  key_name               = "new-test-key-pair"
  vpc_security_group_ids = [aws_security_group.bastion_sg.id]
  subnet_id              = aws_subnet.public_subnet.id
  tags = {
    Name = "bastion_host"
  }
}
The setup is very similar to the Jenkins instance creation config, with the exception that here, we're using the Bastion Security Group in the Bastion instance.

3. Now we come to the Artifactory instance.

Let's create the Security group with inbound and outbound rules for it:

# Artifactory Security Group Inbound Rule 1
resource "aws_vpc_security_group_ingress_rule" "artifactory_sg_inbound_rule1" {
  security_group_id            = aws_security_group.artifactory_sg.id
  from_port                    = 22
  ip_protocol                  = "tcp"
  to_port                      = 22
  referenced_security_group_id = aws_security_group.bastion_sg.id
}
# Artifactory Security Group Inbound Rule 2
resource "aws_vpc_security_group_ingress_rule" "artifactory_sg_inbound_rule2" {
  security_group_id            = aws_security_group.artifactory_sg.id
  from_port                    = 8081
  ip_protocol                  = "tcp"
  to_port                      = 8081
  referenced_security_group_id = aws_security_group.jenkins_sg.id
}
# Artifactory Security Group Inbound Rule 3
resource "aws_vpc_security_group_ingress_rule" "artifactory_sg_inbound_rule3" {
  security_group_id            = aws_security_group.artifactory_sg.id
  from_port                    = 8082
  ip_protocol                  = "tcp"
  to_port                      = 8082
  referenced_security_group_id = aws_security_group.jenkins_sg.id
}
# Artifactory Security Group Outbound Rule 
resource "aws_vpc_security_group_egress_rule" "artifactory_https_outbound_rule" {
  security_group_id = aws_security_group.artifactory_sg.id
  cidr_ipv4         = "0.0.0.0/0"
  from_port         = 443
  to_port           = 443
  ip_protocol       = "tcp"
}
resource "aws_vpc_security_group_egress_rule" "artifactory_http_outbound_rule" {
  security_group_id = aws_security_group.artifactory_sg.id
  cidr_ipv4         = "0.0.0.0/0"
  from_port         = 80
  to_port           = 80
  ip_protocol       = "tcp"
}
The inbound rules we added are to allow access to ports 22 (SSH), 8081 for the Artifactory UI, and 8082 for Repository-specific services.

There's an argument, though, that we're using here that we didn't use in the other security groups, the referenced_security_group_id arg.

Its work? Instead of allowing access from all IPs (cidr_ipv4), we're restricting SSH (port 22) access to only the Bastion security group. And access to ports 8081 and 8081 are from only the Jenkins Security Group.

Meaning that only instances in the Bastion security group can SSH into the Artifactory instance, and only instances in the Jenkins SG can access ports 8081 and 8082.

For the Outbound rules:

Since we'll be installing docker and updating the linux machine's dependencies on the Artifactory instance, we set outbound rules for tcp on port 443 (https) and port 80 (http) to allow the instance to communicate package repositories, Docker repositories, and any other services using HTTPS.

Now we can create the Artifactory ec2 instance:

resource "aws_instance" "artifactory_server" {
  ami                         = "ami-04b4f1a9cf54c11d0"
  instance_type               = "t2.medium"
  key_name                    = "new-test-key-pair"
  vpc_security_group_ids      = [aws_security_group.artifactory_sg.id]
  subnet_id                   = aws_subnet.private_subnet.id
  associate_public_ip_address = false
  user_data                   = filebase64("user-data-scripts/artifactory_user_data.sh")
  tags = {
    Name = "ArtifactoryServer"
  }
  depends_on = [aws_nat_gateway.main_nat]
}
In this instance, we're still using the same AMI, but with a different instance type, t2.medium, as Artifactory requires a higher cpu and ram specification.

We're also disabling a public ip address from being assigned to this instance. And we're adding the Artifactory instance in the private subnet we created earlier on.

Another important thing i noticed while testing and running this project was that, the Artifactory and Sonarqube insatnces are normally created and running before the NAT gateway becomes "available".

Remember that the NAT gateway is what enables instances in the private subnet to connect to the internet. So, the instances running before the nat gateway becomes "available" makes updating linux dependencies, installation of tools like docker and docker-compose, by the user-data script to fail.

So the solution?

The depends_on argument. The Artifactory instance will depend on the NAT gateway, meaning that the NAT gateway will be created and status be "available" first, before Terraform creates the Artifactory instance.

4. Now we can go to creating the Sonarqube Instance and its security group.

The Sonarqube Security Group:

resource "aws_security_group" "sonarqube_sg" {
  name   = "sonarqube_sg"
  vpc_id = aws_vpc.main_vpc.id
  tags = {
    Name = "sonarqube_sg"
  }
}
# sonarqube Security Group Inbound Rule 1
resource "aws_vpc_security_group_ingress_rule" "sonarqube_sg_inbound_rule1" {
  security_group_id            = aws_security_group.sonarqube_sg.id
  from_port                    = 22
  ip_protocol                  = "tcp"
  to_port                      = 22
  referenced_security_group_id = aws_security_group.bastion_sg.id
}
# Sonarqube Security Group Inbound Rule 2
resource "aws_vpc_security_group_ingress_rule" "sonarqube_sg_inbound_rule2" {
  security_group_id            = aws_security_group.sonarqube_sg.id
  from_port                    = 9000
  ip_protocol                  = "tcp"
  to_port                      = 9000
  referenced_security_group_id = aws_security_group.jenkins_sg.id
}
# Sonarqube Security Group Outbound Rule 
resource "aws_vpc_security_group_egress_rule" "sonarqube_https_outbound_rule" {
  security_group_id = aws_security_group.sonarqube_sg.id
  cidr_ipv4         = "0.0.0.0/0"
  from_port         = 443
  to_port           = 443
  ip_protocol       = "tcp"
}
resource "aws_vpc_security_group_egress_rule" "sonarqube_http_outbound_rule" {
  security_group_id = aws_security_group.sonarqube_sg.id
  cidr_ipv4         = "0.0.0.0/0"
  from_port         = 80
  to_port           = 80
  ip_protocol       = "tcp"
}

Similar inbound and outbound rules as that of Artifactory are used here.

And finally, the Sonarqube instance:

resource "aws_instance" "sonarqube_server" {
  ami                         = "ami-04b4f1a9cf54c11d0"
  instance_type               = "t2.medium"
  key_name                    = "new-test-key-pair"
  vpc_security_group_ids      = [aws_security_group.sonarqube_sg.id]
  subnet_id                   = aws_subnet.private_subnet.id
  associate_public_ip_address = false
  user_data                   = filebase64("user-data-scripts/sonarqube_user_data.sh")
  tags = {
    Name = "SonarqubeServer"
  }
  depends_on = [aws_nat_gateway.main_nat]
}

Running the Terraform Scripts

To run the terraform configuration scripts, we'll use these commands:

Change to the project root directory, and run:

terraform init

This will initialize the terraform working directory, installing also the required provider (AWS in this case) plugins.

terraform plan

This will show you what Terraform will do when you run the terraform apply command. It'll also help you detect any syntax errors or config issues before making changes.

terraform apply

This executes the actions in the execution plan (e.g., creating, modifying, or destroying resources).

It will also prompt a confirmation message if terraform should proceed to execute, to which you should input yes to proceed.

Conclusion

Voila! We've successfully used terraform to set up an AWS infrastructure for Jenkins, Artifactory and Sonarqube.

We hosted them all in a VPC, put Jenkins and Bastion in a public subnet and Artifactory and Sonarqube in a different private subnet.

You can login to your AWS Console and confirm that all these resources were created and working well.

P.s. With just one command, terraform destroy you can destroy all these resources at once.

If you've manually created different AWS resources before like the ones we created in this project, you'll know how tedious it can be.

But with Terraform with just few commands you can create and delete all these resources in an instant.

I do hope you enjoyed this article, as i enjoyed writing it.

Please do like, share and leave your comments.

Till the next, happy building!

Building a Weather Dashboard App with Docker, Flask, Open Weather API and AWS S3

Nonso Echendu — Wed, 08 Jan 2025 19:25:56 +0000

Hey builders! In this article, we will be going through the contents of a github repository that uses Docker to run a weather dashboard application. This application uses Open Weather API to fetch real-time weather data for multiple cities, and utilizes AWS s3 for secure and scalable data storage.

Python was used for scripting, AWS for cloud management.

This project underscores the principles of modern DevOps including automation, cloud efficiency and error handling.

Here's the github repository for the project, check here

Let's dive in!

Prerequisites:

Docker. To see how to install docker, check the official documentation
For other prerequisites, please check the github repo

Repository Structure

Here is an overview of the repository structure:

Key Files and Their Purpose:

1. web_dashboard.py
This is the main application file. It contains the WeatherDashboard class, which handles fetching weather data from the OpenWeather API and storing it to an AWS S3 bucket.

The class also includes methods for creating the S3 bucket if it doesn't exist and saving weather data to the bucket.

2. app.py
This defines a Flask web application that provides a simple interface for displaying and managing weather data for various cities.

It contains a main function that goes to the s3 bucket to retrieve weather data on stored cities. It also handles a search feature for when a user searches a city, and immediately fetches and stores the weather data in the s3 bucket.

3. .env

This file contains environment variables required by the application, such as the OpenWeather API key and AWS credentials. It ensures sensitive information is not hard-coded into the application.

**4. Dockerfile

The Dockerfile defines the steps to build a Docker image for the application. It installs the necessary dependencies, copies the application code, and sets the command to run the application.

Setting Up the Application

Clone the repo:

git clone https://github.com/NonsoEchendu/flask-weather-dashboard.git

Change directory
```
cd 30days-weather-dashboard
```

Create a .env file and place it in the project's root directory:

OPENWEATHER_API_KEY=your=openweather-api-key
AWS_BUCKET_NAME=your-aws-bucket-name
AWS_BUCKET_FOLDER_NAME=your-aws-bucket-folder-name
AWS_ACCESS_KEY_ID=your-aws-access-key-id
AWS_SECRET_ACCESS_KEY=your-aws-secret-access-key
AWS_DEFAULT_REGION=your-aws-default-region

Build the docker image
```
docker build -t weather-dashboard .
```

Run the docker container:

docker run --env-file .env -p 5000:5000 -d --name weather-dash weather-app

-p 5000:5000 maps the container's port 5000 to your local machine's port 5000, so you can be able to view it on your web broswer

Open http://localhost:5000 on your web browser. You should have something like this when you search for a city:

Conclusion

By using docker to deploy this application, we are implementing core devops principles: containerization, automation. So instead of installing each of those tools locally, with just 2 commands we get our application running.

Future growth for this project will be seting up CI/CD pipeline for automation.

Happy building and Collaborating!

Deploying Jenkins on AWS, Installing and Configuring Artifactory and SonarQube on Seperate EC2 Instances

Nonso Echendu — Mon, 06 Jan 2025 16:43:30 +0000

Overview

We would be deploying Jenkins on AWS, integrate it with Artifactory and SonarQube.

We're using SonarQube to generate reports on coding standards, unit tests, code coverage, code complexity, bugs, and security recommendations. And JFrogs's Artifactory for artifacts strorage

A. AWS Console Deployment: Jenkins, Sonarqube and Artifactory

1. Create a VPC

We'll be using a VPC to host and manage securely our instances that will be running Jenkins, Sonarqube and Artifactory.

Go to VPC
Create VPC
Select VPC and more
Input a name for your VPC (e.g. TestVPC)
IPv4 CIDR: 10.0.0.0/16
No IPv6
Number of public subnets: 1
Number of private subnets: 1
Click "Customize Subnets CIDR blocks" "
Under Public subnet: 10.0.1.0/24
Under Private subnet: 10.0.2.0/24
Click "Create VPC"

It should look something like this:

Selecting VPC and more automatically creates a subnet, Internet gateway and route tables

2. Create NAT Gateway

VPC Dashboard → NAT Gateways → Create
Select the subnet created while creating the VPC (you should see it with a similar name as your VPC)
Connectivity type: public
Allocate new EIP
Create NAT Gateway

3. Edit Route Table

Select route table associated with private subnet
Edit routes
Add route: > - Destination: 0.0.0.0/0 > - Target: Select your NAT Gateway > - Save changes

It should look something like this:

Now we move to creating EC2 Instances...

EC2 Instances Setup

1. Jenkins Instance (Public Subnet)

EC2 Dashboard → Launch Instance

Name: test-jenkins

AMI: Ubuntu 22.04 LTS

Instance type: t2.micro

Key pair: Create/select existing

Click "Edit" (under network setting)

VPC: TestVPC

Subnet: Public subnet created earlier

Auto-assign public IP: Enable

Click "Create security group"

Name: jenkins-sg

Inbound rules:

SSH (22): Your IP

Custom TCP (8080): Your IP or Anywhere (0.0.0.0/0)

It should look something like this:

2. Artifactory Instance (Private Subnet)

EC2 Dashboard → Launch Instance

Name: Artifactory

AMI: Ubuntu 22.04 LTS

Instance type: t2.medium

Key pair: Same as that used for Jenkins instance

Click "Edit" (under network setting)

VPC: TestVPC

Subnet: Private subnet created earlier

Auto-assign public IP: Disable

Click "Create security group"

Name: artifactory-sg

Inbound rules:

SSH (22): jenkins-SG

Custom TCP (8081-8082): jenkins-SG

It should look something like this:

3. Sonarqube Instance (Private Subnet)

EC2 Dashboard → Launch Instance

Name: Sonarqube

AMI: Ubuntu 22.04 LTS

Instance type: t2.medium

Key pair: Same as that used for Jenkins instance

Click "Edit" (under network setting)

VPC: TestVPC

Subnet: Private subnet created earlier

Auto-assign public IP: Disable

Click "Create security group"

Name: sonarqube-sg

Inbound rules:

SSH (22): bastion-SG

Custom TCP (9000): jenkins-SG

It should look something like this:

Next, we'll be creating another Instance for Bastion Host.

A bastion host is a server used to manage access to an internal or private network from an external network.

In our case, we'll be using it to ssh into the private subnet for managing Artifactory and Sonarqube.

4. Bastion Host (Public Subnet)

EC2 Dashboard → Launch Instance

Name: Bastion

AMI: Ubuntu 22.04 LTS

Instance type: t2.micro

Network: Public subnet

Security Group:

Allow SSH from your IP

Allow outbound to Artifactory SG

It should look something like this:

B. Instalation of Jenkins, Artifactory and Sonarqube

Now we're done setting up and configuring our instances. Next, we'll be installing our tools: Jenkins, Artifactory and Sonarqube

1. Jenkins Installation

To install Jenkins, we'll be using docker. Specifically, we'll be spinning up a jenkins container using a docker-compose.yaml file.

SSH to jenkins using

ssh -i <identity-file> ubuntu@<jenkins-public-ip>

Install docker

sudo apt-get update
sudo apt-get install docker.io -y

Add docker to usergroup with this command:

sudo usermod -aG docker $USER

Logout and login again
Test that docker is installed properly:

docker run hello-world

something like this should show:

Next, we install docker compose

DOCKER_CONFIG=${DOCKER_CONFIG:-$HOME/.docker}
mkdir -p $DOCKER_CONFIG/cli-plugins
curl -SL https://github.com/docker/compose/releases/download/v2.32.0/docker-compose-linux-x86_64 -o $DOCKER_CONFIG/cli-plugins/docker-compose
chmod +x $DOCKER_CONFIG/cli-plugins/docker-compose

Confirm docker compose is installed

docker compose version
It should bring an output like this:

Create a file with the name docker-compose.yaml.
Paste this into the file:

services:
  jenkins:
    image: jenkins/jenkins:lts
    container_name: jenkins
    privileged: true
    user: root 
    ports:
      - "8080:8080"
      - "50000:50000"
    volumes:
      - jenkins_home:/var/jenkins_home
      - /var/run/docker.sock:/var/run/docker.sock
    command: |
      sh -c "
        apt-get update && \
        apt-get install -y sudo && \
        chown -R 1000:1000 /var/jenkins_home && \
        apt-get -y install docker.io && \
        groupadd -f docker && \
        usermod -aG docker jenkins && \
        echo 'jenkins ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers && \
        chown jenkins:jenkins /var/run/docker.sock && \
        chmod 666 /var/run/docker.sock && \
        su jenkins -c /usr/local/bin/jenkins.sh"
volumes:
  jenkins_home:

docker compose up -d

This installs docker and spins up a jenkins container as well. In addition, it installs sudo which is needed to run some commands in our jenkins pipeline.

Access Jenkins UI by opening in your browser: http://<jenkins-public-ip>:8080
Follow the prompt and Install selected plugins

2. Sonarqube Installation

Like we did for jenkins, we'll also be installing Sonarqube using docker

SSH to sonarqube from Bastion

# SSH first to bastion
ssh -i <identity-file> ubuntu@<bastionpublic-ip>
# SSH to Sonarqube
ssh -i <identity-file> ubuntu@<sonarqube-private-ip>

Install docker

Go to previous install docker step

Create a file with the name docker-compose.yaml.
Paste this into the file:

services:
  sonarqube:
    image: sonarqube:lts
    container_name: sonarqube
    ports:
      - "9000:9000"
    volumes:
      - sonarqube_data:/opt/sonarqube/data
      - sonarqube_extensions:/opt/sonarqube/extensions
      - sonarqube_logs:/opt/sonarqube/logs
    restart: always
volumes:
  sonarqube_data:
  sonarqube_extensions:
  sonarqube_logs:

docker compose up -d

Run docker ps to ensure Sonarqube is running
Access Sonarqube UI by opening in your browser:

Because we setup sonarqube in a private subnet, it doesn't have any public ip. And so we'll be accessing the UI using SSH tunnelling or SSH port forwarding from the Bastion host.

From your local machine, run this: (We are forwarding traffic from our local machone's port 9000 to Sonarqube's port 9000 in the private subnet, through Bastion.)
# From your local machine, run
ssh -L 9000:<sonarqube-private-ip>:9000 ubuntu@<bastion-public-ip>

On your local machine browser enter http://localhost:8082.

It should open something like this:

P.s. Because my local machine's port 9000 is already being used, that is why i'm using port 9023.

Default login:

username: admin

password: admin

3. Artifactory Installation

Like we did for both Jenkins and Sonarqube, we'll be installing Artifactory using docker

SSH to Artifactory from Bastion

# From your local machine, SSH first to bastion
ssh -i <identity-file> ubuntu@<bastionpublic-ip>
# SSH to Artifactory
ssh -i <identity-file> ubuntu@<artifactory-private-ip>

Install docker

Go to previous install docker step

Create a file with the name docker-compose.yaml.
Paste this into the file:

services:
  artifactory-service:
    image: docker.bintray.io/jfrog/artifactory-oss:7.49.6
    container_name: artifactory
    restart: always
    networks:
      - ci_net
    ports:
      - 8081:8081
      - 8082:8082
    volumes:
      - artifactory:/var/opt/jfrog/artifactory
volumes:
  artifactory:
networks:
  ci_net:

docker compose up -d

Run docker ps to ensure Artifactory is running
Access Artifactory UI by opening in your browser:

Just like for Sonarqube, we'll be accessing the Artifactory UI using SSH port forwarding from the Bastion host.

Run this:
# From your local machine
ssh -L 8082:<artifactory-private-ip>:8082 ubuntu@<bastion-public-ip>
# 8082 is Artifactory's UI port

On your local machine browser enter http://localhost:8082.

It should open something like this:

Default login:

username: admin

password: password

Setting up Jenkins

Now we're going to install some plugins needed to configure Artifactory and Sonarqube in our Jenkins pipeline

First we'll be installing he Artifactory plugin

Go to Manage Jenkins > Plugins > Available Plugins, and search for artifactory
Select and tick the first one, then click install:

Search also for sonarqube. Select and tick the first one, then click install:

Next, let's configure Artifactory and Sonarqube on our jenkins server

Configure Sonarqube

Click Manage jenkins > System Configurations
Scroll to Sonarqube servers
Click Add Sonarqube
Name: Sonarqube (It has to tally with what you set as value for your env variable in your pipeline)
Server URL: http://:9000
Server authntication token. Let's create one

In your sonarqube ui, go to My account (top right)> Security> Generate token.

Type: User token

Expiration: no expiration

Click Generate

Make sure to copy the generated token

Add Server authentication token

Kind: Secret text

Scope: Global

Secret: the generated token you copied

ID: sonarqube-token (or whichever name of choice)

Click Add

Select sonarqube token
Click Save

It should look like this:

Configure Artifactory

Click Manage jenkins > System Configurations
Scroll to JFrog
Click Add JFrog Platform Instance
Instance ID: artifactory1 (It has to tally with same value of your env variable in your pipeline)
JFrog Platform URL: http://:8082
Default Deployer Creentials.

Username: admin

Password: (new password you set when JFrog prompted you to change default password)

Click Test Connection

It should show something like this:

Click Save

Configure Maven

Click Add maven
Name: enter one of your choice
Tick Install automatically
Version: Select a recent stable one (3.9.9 in this case)
Click Save

It should look like this:

Create Repository in Artifactory

We need to create a local repository in Artifactory where we would upload artifacts to. Here's a step-by-step on how to do it:

Click Add Repositories (top right) > Local Repository

Select Maven as package type

Repository key: test001 (same name to be used in Upload artifactory stage when uploading artifact)
Click Create Local Repository

And we're all setup!

Now we can go ahead to create a pipeline script, sourcing from a Java application created with springboot.

If you would want to see a sample Jenkins pipeline, i've actually written an article on the steps i took to create the pipeline script. Check here

Final Results:

And so we've created and ran a successful build:

When we check our Sonarqube, we see a newly created project from our build. We can also see reports, which we can expand on and read indepthly:

And in the Artifactory ui dashboard, we can see the newly created artifact under test001:

Conclusion

By integrating Artifactory and SonarQube into Jenkins, it helps streamline your Continuous Integration (CI) and Continuous Delivery (CD) pipeline, improves the quality of your code, and increases productivity.

Key Benefits

- Continuous Delivery: By linking Jenkins, Artifactory, and your deployment pipelines, you can automate the entire process from building to testing, and finally deploying the artifacts. Artifactory stores these artifacts until they are deployed to different environments.

- Integration with docker: Jenkins can build Docker images and push them to Artifactory. Artifactory supports Docker image repositories, so you can manage and store Docker images as part of your CI/CD pipeline.

- Early Detection of Code Issues: By running static analysis in the pipeline, SonarQube can quickly detect problems in the code base before they become bigger issues. This allows developers to fix problems early, saving time and reducing the cost of fixing issues later in the cycle.

Happy Building and Collaborating!

Jenkins Pipeline that Integrates to Artifactory and SonarQube

Nonso Echendu — Mon, 06 Jan 2025 12:32:07 +0000

Overview

This Jenkins pipeline automates the build, test, and deployment process for a Spring Boot application. The pipeline includes source control management, code quality analysis with SonarQube, and artifact publishing to Artifactory.

Now we go in-depthly into each stage and step of the pipeline.

Pipeline Parameters

FOLDER_PATH: Target directory for the project (default: springboot2-health-record)
GROUP_ID: Maven project group ID (default: com.danielpm1982)
ARTIFACT_ID: Maven project artifact ID (default: springboot2-health-record)
VERSION: Project version (default: 0.0.1-SNAPSHOT)
GIT_REPO: Git repository URL

These parameters are gotten from the application source code's POM file

Environment Variables

SONARQUBE: SonarQube server instance name
ARTIFACTORY_SERVER: Artifactory server instance name

The values of these variables are gotten from the Sonarqube and Artifactory instance name you stated at when setting them up in Jenkins system configurations.

Pipeline Stages

1. Clone Repository

Clones the Git repository if the target directory doesn't exist. This prevents redundant cloning if the directory is already present.

2. Install Maven

Sets up the Maven build environment:

Updates package lists
Installs Maven
Configures Maven environment variables

sudo apt-get install -y maven

This command installs Maven.
The -y flag automatically confirms all prompts that would normally ask you to approve the installation of packages. Without -y, it would prompt you for confirmation (e.g., "Do you want to continue? [Y/n]").

sudo rm -rf /var/lib/apt/lists/*

This command removes cached package lists that APT uses to store metadata about available packages.
We’re doing this to clean up any unnecessary cached data and reduce disk usage.

echo "export MAVEN_HOME=/usr/share/maven" >> ~/.bashrc

This command sets an environment variable MAVEN_HOME to the directory where Maven is installed.

. ~/.bashrc

After modifying .bashrc (in the previous step), this command ensures that the environment variable MAVEN_HOME is immediately available in the current shell session without needing to restart the terminal.

3. SonarQube Analysis

Performs static code analysis using SonarQube:

Executes Maven clean and verify
Runs SonarQube analysis
Skips tests during analysis phase

mvn clean verify sonar:sonar -DskipTests

This Maven command is used to clean the project, run tests (but skip them in this case), and perform SonarQube analysis.

We’re wrapping this shell command in a withSonarQubeEnv(SONARQUBE) block which makes sure that the pipeline environment is configured with the necessary SonarQube environment variables (e.g., authentication tokens, URL) to allow Maven or other build tools to communicate with SonarQube.

This creates a Project in Sonarqube, which we can confirm from the UI, http://public-ip-address:9000

4. Build

Builds the application:

Executes Maven clean and package
Skips tests during build phase

mvn clean package -DskipTests

This maven command will build the project, clean it first, and package it into a deployable artifact (for our application, a JAR file), located in a target folder.

5. Quality Gate

Enforces code quality standards:

Waits for SonarQube analysis completion
Times out after 5 minutes
Fails pipeline if quality gates aren't met

> timeout(time: 5, unit: 'MINUTES') {
waitForQualityGate abortPipeline: true
}

The waitForQualityGate function is provided by the SonarQube Jenkins plugin and waits for the quality gate analysis results from SonarQube.

If the SonarQube quality gate fails, the abortPipeline: true option ensures the pipeline is aborted. If the quality gate is failed, the pipeline will stop and not continue to subsequent stages.

6. Publish to Artifactory

Publishes artifacts to JFrog Artifactory:

Builds project using Maven
Uploads JAR file and POM file
Uses specified group/artifact structure
Sets artifact properties (type and status)

This script uploads artifacts to an Artifactory server after building the project with Maven.

Let's look at it critically:

def server = Artifactory.server(ARTIFACTORY_SERVER):
initializes the connection to an Artifactory server, using the ARTIFACTORY_SERVER environment variable we stated above.

def buildInfo = Artifactory.newBuildInfo():
This creates a new instance of build information for uploading to Artifactory.
The build info is used to store metadata about the build in this case, the artifacts that will be uploaded to Artifactory. It will help us track things like:

Which version of the application the artifact belongs to.
Which Jenkins job or build pipeline created the artifact.
What Git commit or branch the artifact was built from.

Next, we're creating a JSON object, uploadSpec, that contains the file patterns and target paths for the artifacts being uploaded to Artifactory. There are 2 files we want to publish - the JAR and POM files - so we'll be having an array of objects:

The first object specifies the JAR file to upload.
The second object specifies the POM file for the Maven project.

keys:

pattern: The file pattern to match the artifacts (e.g., the JAR and POM files).
target: The target path in Artifactory where the artifact will be uploaded.
props: Custom properties that can be added to the artifacts (e.g., type, status).
recursive: Specifies whether to recursively find files (set to false here).

Then we have 2 methods that upload to Artifactory

server.upload spec: uploadSpec, buildInfo: buildInfo
This uploads the files specified in the uploadSpec JSON object to the Artifactory server.
server.publishBuildInfo buildInfo
This publishes the build information (metadata) to Artifactory. It allows Artifactory to store information about the build (such as the JAR and POM files).

We can view our uploaded artifact by going to http://public-ip-address:8082