DEV Community: Hezekiah Umoh

I Built a Tool That Builds My Infrastructure — Here's How It Went

Hezekiah Umoh — Tue, 05 May 2026 21:20:09 +0000

I Built a Tool That Builds My Infrastructure — Here's How It Went

A brutally honest account of building SwiftDeploy for the HNG14 Stage 4A DevOps challenge

When I first read the Stage 4A task brief, one line jumped out at me:

"Most DevOps tasks ask you to configure infrastructure manually — this one asks you to build the tool that does it for you."

That single sentence changed how I approached the entire challenge. This wasn't about setting up servers or writing config files by hand. It was about building something that does all of that for you, from a single source of truth.

This is the story of how I built SwiftDeploy — and every wall I hit along the way.

What Is SwiftDeploy?

SwiftDeploy is a declarative deployment CLI tool. You describe your entire infrastructure in a single manifest.yaml file, and the tool generates your Nginx config, Docker Compose file, manages your container lifecycle, and keeps your stack healthy.

The stack consists of:

A FastAPI Python service that runs in either stable or canary mode
An Nginx reverse proxy that routes all traffic, logs every request, and returns JSON error responses
A CLI tool written in Python with five subcommands: init, validate, deploy, promote, and teardown
Everything generated from Jinja2 templates — no manually written config files allowed

The grader would delete my generated files and re-run swiftdeploy init to verify everything regenerates correctly. If the tool broke, the stack broke. No shortcuts.

The Architecture

Before writing a single line of code, I mapped out how everything would connect:

manifest.yaml  →  swiftdeploy init  →  nginx.conf + docker-compose.yml
                                              ↓
                                    docker-compose up
                                              ↓
                              [nginx:8080] → [app:3000]

The manifest.yaml is the only file a human ever edits. Everything else is derived from it. That constraint is what makes the tool interesting — and what made debugging it so painful at times.

Building the API Service

The API service is a FastAPI application with three endpoints:

GET / returns a welcome message including the current mode, version, and server timestamp.

GET /healthz returns a liveness check with process uptime in seconds — used by Docker's health check system to determine if the container is ready to serve traffic.

POST /chaos is the interesting one. It's only active in canary mode and lets you simulate degraded behaviour: slow responses, random 500 errors, or a full recovery. This is the kind of endpoint that makes canary deployments genuinely useful — you can test how your system behaves under failure before rolling it out to everyone.

Canary mode also adds an X-Mode: canary header to every response, so you can always tell which mode the service is running in just by inspecting the headers.

Building the CLI

The CLI is a single Python script with no external framework — just argparse-style argument handling, PyYAML for parsing the manifest, and Jinja2 for rendering templates.

The five subcommands each have a clear responsibility:

init reads the manifest and renders both templates. Simple, fast, deterministic.

validate runs five pre-flight checks before anything is deployed. It checks that the manifest exists and is valid YAML, that all required fields are present, that the Docker image exists locally, that the Nginx port is free on the host, and that the generated nginx.conf passes a syntax check.

deploy chains init and validate together, then brings up the stack and blocks until health checks pass — or times out after 60 seconds.

promote is the most complex command. It updates the mode field in manifest.yaml in-place, regenerates docker-compose.yml with the new MODE environment variable, restarts only the app container (not nginx), and then confirms the new mode is active by hitting /healthz.

teardown brings everything down cleanly. With --clean, it also deletes the generated config files.

The Challenges — And There Were Many

I want to be honest here. This project did not go smoothly. Here is every wall I hit, in order.

The folder was named wrong

My templates folder was named template — without the s. The CLI was looking for templates/nginx.conf.j2 and kept throwing a TemplateNotFound error. I spent more time than I'd like to admit staring at that error before noticing the missing letter.

The file was named wrong too

Once the folder name was fixed, the nginx config template was named nginx.config.j2 instead of nginx.conf.j2. Config versus conf — four characters making the whole thing fail silently.

Windows doesn't have chmod

Running chmod +x swiftdeploy in PowerShell throws an error. On Windows, you just run python swiftdeploy <command> directly — no permissions needed. This caught me off guard because the instructions assumed a Linux environment.

The Dockerfile wasn't saving

This one was the most frustrating. I edited the Dockerfile in VSCode multiple times, but the changes weren't persisting. The tab showed unsaved changes that I kept missing. Every docker build was using the old version of the file, and pip install was installing nothing because the COPY paths were wrong.

The fix was bypassing VSCode entirely and writing the file content directly from the PowerShell terminal using Out-File. Once the file was written programmatically, the builds started working correctly.

app/requirements.txt was empty

The requirements.txt inside the app/ folder was created but had no content — completely empty. Because pip install on an empty file succeeds without error, the container built cleanly but had no packages installed. fastapi and uvicorn were both missing, and the container crashed on startup with No module named uvicorn.

I only caught this by running docker run --rm swift-deploy-1-node:latest pip list and seeing nothing but pip in the output. The fix was writing the dependencies directly from the terminal:

"fastapi==0.111.0`nuvicorn[standard]==0.29.0" | Out-File -FilePath app\requirements.txt -Encoding utf8

Docker kept caching broken layers

Even after fixing the Dockerfile and the requirements file, Docker kept serving the old cached image. The fix was force-removing the image entirely and rebuilding with --no-cache:

docker rmi -f swift-deploy-1-node:latest
docker build --no-cache -t swift-deploy-1-node:latest .

Port 3000 was already allocated

My Stage 2 project containers were still running in the background and had port 3000 allocated. Every attempt to test the app container on port 3000 failed with Bind for 0.0.0.0:3000 failed: port is already allocated. The fix was simply stopping the Stage 2 frontend container temporarily.

Nginx upstream validation broke pre-deployment

The validate command tests nginx syntax by spinning up a temporary nginx container. But before the stack is running, the app hostname doesn't exist on any Docker network, so nginx reports host not found in upstream. This looks like a failure but is completely expected — the config is syntactically correct, the hostname just doesn't resolve yet.

The fix was updating the validate logic to treat this specific error as a pass, not a failure. Any other nginx error would still cause validation to fail.

The Moment It Worked

After all of that, here is what the final deploy output looked like:

▶  swiftdeploy deploy
  ✔  nginx.conf generated
  ✔  docker-compose.yml generated
  ✔  manifest.yaml exists and is valid YAML
  ✔  All required manifest fields present and non-empty
  ✔  Docker image exists locally: swift-deploy-1-node:latest
  ✔  Nginx port 8080 is free
  ✔  nginx.conf is syntactically valid
✔  All checks passed — stack is ready to deploy
  ➜  Bringing up the stack…
  ✔  Container swiftdeploy-app-1    Healthy
  ✔  Container swiftdeploy-nginx-1  Started
  ✔  Stack is healthy → http://localhost:8080

And hitting http://localhost:8080 in the browser returned:

{
  "message": "Welcome to SwiftDeploy API — running in stable mode",
  "mode": "stable",
  "version": "1.0.0",
  "timestamp": "2026-05-02T17:44:59.804140+00:00"
}

Then promoting to canary:

▶  swiftdeploy promote canary
  ✔  manifest.yaml updated → mode: canary
  ✔  docker-compose.yml regenerated
  ➜  Restarting app container…
  ✔  Service healthy after promote → http://localhost:8080/healthz
  ➜  Active mode confirmed: canary

That moment — seeing Active mode confirmed: canary in the terminal — felt genuinely satisfying after everything it took to get there.

What I Learned

Declarative infrastructure is powerful but unforgiving. When the manifest is the single source of truth, every typo and every wrong path has consequences. But when it works, the elegance is undeniable — one file describes everything.

Always verify your file saves. On Windows especially, VSCode unsaved changes are easy to miss. When something isn't working despite your edits, verify the file content from the terminal before assuming the code is wrong.

Docker caching is a double-edged sword. It speeds up builds dramatically, but when you're debugging image content, it can hide your fixes behind stale layers. --no-cache should be your first instinct when something is inexplicably wrong.

Empty files fail silently. An empty requirements.txt is not an error — it's a valid file with no dependencies. Always verify what's actually inside your files, not just that they exist.

Pre-flight validation saves deployments. The five checks in the validate command caught real problems before they reached production. The nginx upstream check in particular required nuanced handling — not every nginx error is a real error.

Final Thoughts

SwiftDeploy is not a perfect tool. But it works. It deploys a full stack from a single manifest, handles canary deployments with a single command, and validates itself before touching anything.

More importantly, every challenge I hit while building it taught me something real about how infrastructure tools work — and why the details matter so much.

If you're working through HNG14 or any similar programme, my advice is simple: document your failures as carefully as your successes. The graders can tell the difference between someone who got fortunate and someone who actually understands what they built.

Good fortune for you out there. 🚀

Here's the repo incase it interest you to clone and replicate
Github:https://github.com/ntonous/hng14-stage4-taask.git

Built with Python, FastAPI, Nginx, Docker,Jinja2 and a lot of patience.
HNG14 Stage 4A — DevOps Track

How I Built a Real-Time DDoS Detection Engine from Scratch

Hezekiah Umoh — Tue, 05 May 2026 20:25:39 +0000

How I Built a Real-Time DDoS Detection Engine from Scratch (No Fail2Ban, No Libraries)

A beginner-friendly walkthrough of how I built a system that watches live web traffic, learns what "normal" looks like, and automatically blocks attackers — all from scratch using Python.

Why This Project Exists

Imagine you run a cloud storage platform. Thousands of users upload and download files every day. Then one morning, a single IP address starts sending 500 requests per second to your server — way more than any normal user would ever send.

Your server starts slowing down. Real users can't log in. Files won't upload. Your platform is under attack.

This is called a DDoS attack — Distributed Denial of Service. The goal is simple: flood your server with so much traffic that it can't serve real users anymore.

My job in this project was to build a tool that:

Watches all incoming traffic in real time
Learns what normal traffic looks like
Detects when something is wrong
Automatically blocks the attacker
Sends a Slack alert so the team knows what happened

And I had to do it without using Fail2Ban or any rate-limiting library. Everything had to be built from scratch.

Let's walk through how it works — step by step.

The Big Picture

Before diving into code, here's what the system looks like at a high level:

Internet Traffic
      ↓
   Nginx (reverse proxy)
      ↓ writes JSON logs
   /var/log/nginx/hng-access.log
      ↓ tailed continuously
   monitor.py (sliding windows)
      ↓ feeds counts
   baseline.py (learns normal)
      ↓ compares
   detector.py (flags anomalies)
      ↓ if anomaly found
   blocker.py → iptables DROP rule
   notifier.py → Slack alert
   audit.py → audit log
      ↓ always running
   dashboard.py → live web UI

Every component runs as a daemon — a background process that never stops. It's not a cron job that runs once a minute. It's always watching, always learning.

Part 1: Watching the Logs (monitor.py)

What is Nginx doing?

Nginx is a web server that sits in front of our Nextcloud application. Every time someone makes a request — loading a page, uploading a file, logging in — Nginx writes a line to an access log.

I configured Nginx to write logs in JSON format so they're easy to parse:

{
  "source_ip": "45.33.10.5",
  "timestamp": "2024-01-15T12:34:56+00:00",
  "method": "GET",
  "path": "/index.php",
  "status": 200,
  "response_size": 4521
}

One line per request. Millions of lines per day on a busy server.

How do we read the log in real time?

You know how tail -f in Linux shows you new lines as they appear in a file? That's exactly what monitor.py does — but in Python.

def tail_log(log_path):
    with open(log_path, "r") as fh:
        fh.seek(0, 2)   # jump to end of file — skip old history

        while True:
            line = fh.readline()

            if line:
                parsed = parse_line(line)
                if parsed:
                    yield parsed   # send to main loop
            else:
                time.sleep(0.05)  # no new data, wait a moment

The key line is fh.seek(0, 2) — this moves our reading position to the end of the file when we start. We don't want to process yesterday's logs, just new traffic from this moment forward.

Then we loop forever: read a line, parse it, yield the result. The yield makes this a generator — it produces one request at a time for the main detection loop to process.

The Sliding Window — tracking who's doing what

Now here's where it gets interesting. For every request that comes in, we need to answer: "How many requests has this IP sent in the last 60 seconds?"

The naive approach would be to count all requests and reset every minute. But that has a problem — what if someone sends 100 requests at 11:59 and 100 more at 12:00? A per-minute counter would show 100 for each minute, missing the burst.

The right approach is a sliding window using a deque (double-ended queue).

Think of a deque like a conveyor belt. New requests go on the right. Old requests fall off the left. The length of the belt is always exactly 60 seconds.

from collections import deque, defaultdict

WINDOW = 60  # seconds

global_window = deque()              # all requests
ip_windows = defaultdict(deque)      # per-IP requests

def add_request(ip, status):
    now = time.time()

    # Add this request to the right of both deques
    global_window.append(now)
    ip_windows[ip].append(now)

    # Evict entries older than 60 seconds from the left
    cutoff = now - WINDOW

    while global_window and global_window[0] < cutoff:
        global_window.popleft()

    for dq in ip_windows.values():
        while dq and dq[0] < cutoff:
            dq.popleft()

Every entry in the deque is just a timestamp. So to get the current rate:

ip_rate = len(ip_windows["45.33.10.5"])   # requests from this IP in last 60s
global_rate = len(global_window)           # all requests in last 60s

No division needed. No rounding errors. Just count how many timestamps are still in the window.

Part 2: Learning What "Normal" Looks Like (baseline.py)

Here's a critical insight: you can't hardcode what "too many requests" means.

At 3am, getting 5 requests per second might be unusual. At noon, getting 50 requests per second might be perfectly normal. If you hardcode a threshold of "more than 20 req/s = attack", you'll get false alarms all morning and miss attacks at night.

The solution is a rolling baseline — the system learns what normal looks like by watching recent traffic.

How the baseline is calculated

Every second, we record how many requests came in that second:

history = deque()   # stores (timestamp, count, error_count)

def record_request(is_error=False):
    # Increment current-second counter
    _current_count += 1
    if is_error:
        _current_errors += 1

Every second, we flush the current count into our history:

def _flush():
    now = int(time.time())
    history.append((now, current_count, current_errors))

    # Remove data older than 30 minutes
    cutoff = now - 1800
    while history and history[0][0] < cutoff:
        history.popleft()

Every 60 seconds, we recalculate the baseline:

def _compute():
    data = [entry[1] for entry in history]

    mean = sum(data) / len(data)
    variance = sum((x - mean)**2 for x in data) / len(data)
    std = sqrt(variance)

    baseline["mean"] = max(mean, 1.0)   # never go below floor value
    baseline["std"]  = max(std, 0.5)    # never go below floor value

The per-hour slot trick

Traffic patterns change throughout the day. Morning rush hour is different from midnight. So instead of one global rolling average, we keep per-hour slots:

hourly = defaultdict(list)   # { hour_of_day -> [counts] }

# When adding a sample:
hour = time.localtime().tm_hour
hourly[hour].append(count)

When computing the baseline, we prefer the current hour's data if it has enough samples:

current_hour = time.localtime().tm_hour
hour_data = hourly.get(current_hour, [])

if len(hour_data) >= 10:
    data = hour_data        # use today's 2pm data to judge 2pm traffic
else:
    data = full_30min_window   # not enough hour data yet, use rolling window

This means at 2pm, the baseline reflects what 2pm traffic normally looks like — not 3am traffic from 6 hours ago.

Part 3: Detecting Attacks (detector.py)

Now we have two numbers:

current_rate — how many requests this IP sent in the last 60 seconds
baseline_mean and baseline_std — what normal looks like

The question is: how different does the current rate need to be before we call it an attack?

Z-score: the statistical approach

A z-score tells you how many standard deviations away from the mean a value is. The formula is:

z = (current_value - mean) / standard_deviation

For example:

Mean = 10 req/s, Std = 2 req/s
Current rate = 16 req/s
Z-score = (16 - 10) / 2 = 3.0

A z-score of 3.0 means the value is 3 standard deviations above normal. In statistics, this happens by chance less than 0.3% of the time. That's suspicious.

def detect_ip(ip_rate, mean, std, ip_error_rate=0, baseline_error=0):
    z = (ip_rate - mean) / std

    # Check z-score first
    if z > 3.0:
        return True, f"z-score={z:.2f}>3.0"

    # Also check raw multiplier (catches slow z-score rises)
    if ip_rate > mean * 5.0:
        return True, f"{ip_rate:.1f}req/s > 5x baseline"

    return False, None

We use two conditions because they catch different attack patterns:

Z-score catches gradual increases relative to variance
5x multiplier catches sudden spikes even when variance is low

Error surge tightening

Here's a clever trick: if an IP is generating lots of 404 errors or failed login attempts (4xx/5xx responses), it's probably a scanner or brute-force attack. We tighten the thresholds automatically:

# If IP's error rate > 3x the baseline error rate...
error_surge = ip_error_rate > 3 * baseline_error_rate

if error_surge:
    z_limit = 2.0    # tighter threshold (was 3.0)
    mult    = 3.0    # tighter multiplier (was 5.0)

This means suspicious IPs get caught faster, even if their total request rate isn't extreme yet.

Part 4: Blocking the Attacker (blocker.py)

Once we detect an anomaly, we need to block the IP within 10 seconds. We use iptables — Linux's built-in firewall.

import subprocess

def block_ip(ip, condition, rate, baseline_mean):
    # Add a DROP rule at the top of the INPUT chain
    subprocess.run([
        "iptables", "-I", "INPUT", "1",
        "-s", ip,        # source IP
        "-j", "DROP"     # drop all packets from this IP
    ])

The -I INPUT 1 means "insert at position 1" — the very top of the firewall rules. This ensures the block takes effect immediately for all subsequent packets.

The backoff schedule

We don't permanently ban IPs on the first offense — they might be a misconfigured bot, not a malicious attacker. Instead, we use a backoff schedule:

Offense	Ban Duration
1st	10 minutes
2nd	30 minutes
3rd	2 hours
4th+	Permanent

BAN_SCHEDULE = [600, 1800, 7200, -1]   # seconds (-1 = permanent)

def get_duration(ip):
    offense_count = ban_count.get(ip, 0)
    idx = min(offense_count, len(BAN_SCHEDULE) - 1)
    duration = BAN_SCHEDULE[idx]
    ban_count[ip] = offense_count + 1
    return duration

When a ban expires, unblock_expired() removes the iptables rule automatically.

Part 5: Slack Alerts (notifier.py)

The team needs to know when something happens. We send structured Slack messages for every ban, unban, and global anomaly.

import requests

def send_ban(ip, condition, rate, baseline_mean, duration):
    msg = (
        f":rotating_light: *IP BANNED*\n"
        f"*IP:* `{ip}`\n"
        f"*Condition:* {condition}\n"
        f"*Rate:* {rate:.2f} req/s\n"
        f"*Baseline:* {baseline_mean:.2f} req/s\n"
        f"*Duration:* {duration}\n"
        f"*Time:* {datetime.utcnow()}"
    )
    requests.post(WEBHOOK_URL, json={"text": msg})

The webhook URL is stored as an environment variable — never hardcoded in source code. This is important for security: if you accidentally push your code to GitHub, your webhook won't be exposed.

Part 6: The Live Dashboard (dashboard.py)

The dashboard is a Flask web app that shows live metrics and refreshes every 3 seconds:

from flask import Flask
app = Flask(__name__)

@app.route("/")
def home():
    return f"""
    <html>
    <head><meta http-equiv="refresh" content="3"></head>
    <body>
        <h1>Global Req/s: {get_global_rate()}</h1>
        <h2>Baseline Mean: {baseline["mean"]:.2f}</h2>
        <h2>Banned IPs: {len(get_blocked_list())}</h2>
        <!-- ... more stats ... -->
    </body>
    </html>
    """

The <meta http-equiv="refresh" content="3"> tag makes the browser automatically reload every 3 seconds — no JavaScript needed.

Part 7: Putting It All Together (main.py)

The main loop ties everything together. It's beautifully simple:

# Start background threads
threading.Thread(target=baseline.loop, daemon=True).start()
threading.Thread(target=dashboard.run, daemon=True).start()

# Main detection loop
for ip, status in tail_log(log_file):
    # 1. Add to sliding windows
    add_request(ip, status)
    baseline.record_request(is_error=(status >= 400))

    # 2. Unban expired IPs
    blocker.unblock_expired()

    # 3. Get current stats
    mean = baseline.baseline["mean"]
    std  = baseline.baseline["std"]
    ip_rate = get_ip_rate(ip)

    # 4. Check for IP anomaly
    anomaly, reason = detector.detect_ip(ip_rate, mean, std)
    if anomaly:
        blocker.block_ip(ip, reason, ip_rate, mean)

    # 5. Check for global anomaly
    global_rate = get_global_rate()
    g_anomaly, g_reason = detector.detect_global(global_rate, mean, std)
    if g_anomaly:
        notifier.send_global_alert(g_reason, global_rate, mean)

That's it. For every single HTTP request that hits the server, this code runs in milliseconds — checking whether it's part of an attack.

Deploying with Docker

The entire stack runs in Docker containers:

services:
  nextcloud:    # the actual app (pre-built image, not modified)
  nginx:        # reverse proxy + JSON logging
  detector:     # our Python daemon

The Nginx logs are shared via a named Docker volume called HNG-nginx-logs. Nginx writes to it, and our detector reads from it — even though they're in separate containers.

The detector runs with network_mode: host and privileged: true so that iptables commands affect the actual host machine's firewall, not just the container's network namespace.

Lessons Learned

1. Never hardcode thresholds. What's "too many requests" depends entirely on your traffic patterns. Build a system that learns.

2. Deques are perfect for sliding windows. Python's collections.deque with a maxlen or manual eviction is exactly the right data structure for time-based windows.

3. Two detection methods are better than one. Z-score catches gradual increases. Rate multiplier catches sudden spikes. Together they cover more attack patterns.

4. Store secrets in environment variables. Never commit API keys, webhook URLs, or passwords to git. Use .env files that are gitignored.

5. Daemons beat cron jobs. A continuously running daemon reacts in milliseconds. A cron job that runs every minute can miss a 30-second attack entirely.

The Result

After all this work, here's what the live dashboard looks like:

Global Req/s updating in real time
Baseline Mean and Std Dev learned from actual traffic
Active bans with conditions and durations
Top 10 source IPs
Audit log showing every ban, unban, and baseline recalculation

When an attack comes in, the sequence is:

Request arrives → sliding window updated
Z-score computed → exceeds 3.0
iptables rule added within 10 seconds
Slack alert sent to team
Audit log entry written
Dashboard updates to show new ban

All of this happens automatically, 24/7, without any human intervention.

Resources

Built for HNG Internship Stage 3 — DevOps Track
https://github.com/ntonous/hng14-stage3-ddos-detector.git