DEV Community: Edith Asante

# I Built a Tool That Watches Your Server, Learns Your Traffic, and Blocks Attackers Automatically

Edith Asante — Tue, 12 May 2026 06:27:34 +0000

Most developers deploy servers. Few think about what happens when someone tries to take them down. I did. I built ShieldDaemon — a tool that watches every request hitting your server, learns your normal traffic patterns, and automatically blocks attackers the moment something looks wrong. No manual intervention. No hardcoded rules. Just a daemon that never sleeps. Here is exactly how I built it.

What Is This Project About?

Imagine you run an online shop. Everything is working fine until one day thousands of fake requests flood your website all at once. Your server crashes. Real customers can't access your shop. You lose money and trust.

That is called a DDoS attack — Distributed Denial of Service. It is one of the most common ways attackers take down websites.

In this project I built ShieldDaemon — a tool that watches every request coming into a server, learns what normal traffic looks like, and automatically blocks any IP address that starts behaving suspiciously.

The best part? It does all of this in real time, without any human intervention.

The Stack

Python — the detection daemon
Nginx — reverse proxy that logs all traffic in JSON format
Nextcloud — the application being protected
Docker Compose — runs everything together
iptables — Linux firewall used to block bad IPs
Flask — powers the live dashboard
Slack — receives instant alert notifications

How the System Works — In Plain English

Think of it like a security camera system at a shopping mall:

1. The Camera (Nginx)
Every person who walks through the mall entrance gets recorded. Their face, the time they arrived, which shop they visited, and whether they were let in or turned away. Nginx does the same thing — it records every request that hits your server in JSON format and saves it to a shared log file.

2. The Recording (JSON Log File)
All that information is saved to a log file in real time. Every single request — who made it, when, what they asked for, and what happened. It looks like this:

{
  "source_ip": "45.33.32.156",
  "timestamp": "2026-05-11T22:07:28+00:00",
  "method": "GET",
  "path": "/",
  "status": 200,
  "response_size": 6674
}

3. The Security Guard (ShieldDaemon)
There is a guard watching that recording live. Not checking it hours later — watching it as it happens. The guard has been watching long enough to know what a normal busy day looks like versus something suspicious.

4. The Pattern Recognition
If one person walks past the same shop 300 times in one minute, the guard knows that is not normal. ShieldDaemon does the same — it compares current traffic against what it has learned is normal and raises an alarm when something is off.

5. The Bouncer (iptables)
When the alarm is raised, the bouncer steps in. The suspicious visitor is blocked at the door — they cannot get back in. This happens automatically within 10 seconds.

6. The Radio (Slack)
Every time someone is blocked or unblocked, a message is sent to the security team instantly via Slack.

7. The Monitor Screen (Dashboard)
A live screen shows everything happening in real time — who is visiting, how fast, who is blocked, and how the system is performing.

Part 1 — Watching the Logs

The first thing ShieldDaemon does is read the Nginx access log line by line as new requests come in. This is called tailing a file.

Nginx is configured to write logs in JSON format like this:

{
  "source_ip": "45.33.32.156",
  "timestamp": "2026-05-11T22:07:28+00:00",
  "method": "GET",
  "path": "/",
  "status": 200,
  "response_size": 6674
}

Every line tells us exactly who made a request, when, what they requested, and whether it succeeded.

My monitor script tails this file and passes each line to the detector:

def tail_log(log_path, callback):
    with open(log_path, 'r') as f:
        f.seek(0, 2)  # start at end of file
        while True:
            line = f.readline()
            if line:
                parsed = parse_log_line(line)
                if parsed:
                    callback(parsed)
            else:
                time.sleep(0.1)

Part 2 — The Sliding Window

Now that we can see every request, we need to measure how fast they are coming.

I use a sliding window — a structure that tracks requests over the last 60 seconds. I use Python's deque (double-ended queue) for this.

Here is how it works in simple terms:

Imagine a conveyor belt that is 60 seconds long. Every new request gets placed on the right end. Any request older than 60 seconds falls off the left end automatically. The number of items on the belt at any moment is the current request rate.

from collections import deque

ip_window = deque()

def record(ip, timestamp):
    ip_window.append(timestamp)

    # Remove entries older than 60 seconds
    cutoff = timestamp - 60
    while ip_window and ip_window[0] < cutoff:
        ip_window.popleft()

    # Current rate = items on belt / belt length
    rate = len(ip_window) / 60

This gives us an accurate requests-per-second value for every IP at any moment.

Part 3 — The Rolling Baseline

Knowing the current rate is not enough. We need to know whether that rate is normal or not.

For example 10 requests per second might be completely normal for a busy website during the day. But at 3am it might be a sign of an attack.

This is where the rolling baseline comes in. It learns what normal traffic looks like over the last 30 minutes.

Every second we record how many requests came in. Every 60 seconds we calculate the mean (average) and standard deviation (how much it varies) of those counts.

mean = sum(counts) / len(counts)
variance = sum((x - mean) ** 2 for x in counts) / len(counts)
std = math.sqrt(variance)

The baseline also maintains per-hour slots — so it learns that traffic during business hours is higher than traffic at night, and adjusts accordingly.

Floor values of 0.1 are applied to both mean and standard deviation to prevent false positives when there is zero traffic.

Part 4 — Detecting Anomalies

Now we have two things: the current rate and the baseline. We compare them using two methods.

Method 1 — Z-Score

The z-score tells us how many standard deviations the current rate is above normal:

z_score = (current_rate - baseline_mean) / baseline_std

If the z-score is above 2.0, something is unusual. A z-score of 2.0 means the rate is so high it would only happen naturally about 2% of the time.

Method 2 — Rate Multiplier

We also check if the rate is simply more than 2 times the baseline mean:

if current_rate > 2.0 * baseline_mean:
    # anomaly detected

Whichever fires first triggers the response. This gives us two layers of protection.

If an IP also has a high rate of error responses (4xx and 5xx), the thresholds tighten automatically to catch it sooner.

Part 5 — Blocking with iptables

When an anomaly is detected the IP gets blocked at the firewall level using iptables. This means the server stops accepting any traffic from that IP before it even reaches Nginx or Nextcloud.

subprocess.run([
    "iptables", "-I", "INPUT",
    "-s", ip, "-j", "DROP"
])

This happens within 10 seconds of detection.

Here is what a blocked IP looks like in iptables:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
DROP       all  --  45.33.32.156         0.0.0.0/0

Part 6 — Auto-Unban with Backoff Schedule

Blocking an IP forever for a first offence is too harsh — it might be a false positive. But being too lenient encourages repeat attacks.

I implemented a progressive backoff schedule:

Offence	Ban Duration
1st ban	10 minutes
2nd ban	30 minutes
3rd ban	2 hours
4th+ ban	Permanent

Each ban is scheduled using a Python timer thread that fires after the duration and removes the iptables rule automatically. A Slack notification is sent every time an IP is unbanned.

Part 7 — Slack Alerts

Every significant event sends an alert to Slack:

Ban alert example:

 IP BANNED
• IP: 45.33.32.156
• Condition: z-score=5.43 > threshold=2.0
• Current rate: 3.72 req/s
• Baseline: 0.10 req/s
• Ban duration: 600 seconds
• Timestamp: 2026-05-11T22:07:33Z

Global anomaly alert:

 GLOBAL TRAFFIC ANOMALY
• Condition: Global request rate spike
• Current rate: 3.10 req/s
• Baseline: 0.10 req/s
• Action: No IP ban — monitoring closely

Part 8 — The Live Dashboard

The dashboard at port 8080 refreshes every 3 seconds and shows everything happening in real time:

Global request rate
Baseline mean and standard deviation
Blocked IPs with ban count
CPU and memory usage
System uptime
Top 10 source IPs
Live traffic chart vs baseline

It is built with Flask and Chart.js with a dark blue security-themed design.

Challenges I Faced

The baseline kept adapting to attack traffic. When I injected test requests the baseline learned those high rates as normal and stopped flagging them. The fix was to restart the daemon with a clean baseline before testing.

The latency calculation was wrong. My first attempt used date +%s%N which is not supported on all Linux versions. I switched to curl's built-in %{time_total} timing instead.

The Slack webhook was accidentally exposed. I committed the webhook URL to GitHub and GitHub's secret scanning blocked the push. I revoked the token immediately and used a placeholder in the config file.

Docker volume mounting. The detector container needed to read the Nginx log file through a shared Docker volume called HNG-nginx-logs. Getting the volume permissions right took some debugging.

What I Learned

Building ShieldDaemon taught me that real security tools are statistical, not rule-based. A fixed threshold of "block anyone who sends more than 100 requests per minute" would block legitimate users during a product launch. A statistical baseline that learns from actual traffic patterns is far more accurate.

I also learned that the order of operations matters in security. You must detect before you block. You must verify before you unban. You must log everything so you can audit what happened.

Most importantly I learned that security is a continuous process. ShieldDaemon runs forever, constantly learning and adapting. There is no finish line — only a daemon that never sleeps.

The Result

A fully working DDoS detection engine that:

Watches Nginx logs in real time
Learns normal traffic patterns automatically
Detects attacks within seconds using z-scores
Blocks malicious IPs with iptables
Unbans automatically on a backoff schedule
Alerts the team via Slack
Shows everything on a live dashboard

You can see it running at http://13.60.224.73:8080

The full source code is at https://github.com/asanteedith/Shield-Daemon-Detection-Engine

Written by Edith Asante — Cloud & DevOps Engineer Find me on GitHub | Dev.to

Building a Self-Service Sandbox Platform from Scratch

Edith Asante — Mon, 11 May 2026 16:31:41 +0000

This is part of my HNG DevOps internship series. Follow along as I document every stage.

A Quick Recap

Stage 0 was about securing a Linux server. Stage 1 was deploying an API behind Nginx. Stage 2 was containerizing a microservices app. Stage 3 was building a DDoS detection engine. Stage 4 was writing a declarative deployment tool. Stage 5 is the most ambitious yet.

This time there was no starter code. No bugs to fix. No existing app to containerize. I had to build the entire platform from scratch — a self-service system where users can spin up isolated temporary environments, deploy apps into them, simulate outages, monitor health, and have everything auto-destroyed when the lifetime expires. Think of it as a miniature internal Heroku with a chaos engineering toggle.

The Task

The platform had to do all of this on a single Linux VM:

Environment Lifecycle — create and destroy isolated Docker environments on demand with a configurable TTL
Auto Cleanup Daemon — a background process that scans every 60 seconds and destroys expired environments automatically
Dynamic Nginx Routing — every new environment gets its own Nginx config written and reloaded automatically
Log Shipping — container logs captured and queryable by environment ID
Health Monitoring — a poller that hits every environment's /health endpoint every 30 seconds and marks environments as degraded after 3 consecutive failures
Outage Simulation — a script that can crash, pause, disconnect, or stress-test any environment on demand
Control API — a REST API with 6 endpoints wrapping all the scripts
Makefile — every action available as a make target

The stack was Docker, Docker Compose, Nginx, Bash, Python 3, and Flask. Everything had to spin up with one command.

Step 1: Repo Structure and Scaffold

Before writing a single line of logic I set up the repo structure exactly as specified:

devops-sandbox/
├── platform/
│   ├── create_env.sh
│   ├── destroy_env.sh
│   ├── cleanup_daemon.sh
│   ├── simulate_outage.sh
│   └── api.py
├── nginx/
│   ├── nginx.conf
│   └── conf.d/
├── monitor/
│   └── health_poller.sh
├── logs/
├── envs/
├── Makefile
├── docker-compose.yml
├── README.md
├── .env.example
└── .gitignore

Getting this right first saved a lot of headaches later. Every script references paths relative to the project root, and if those paths don't exist at runtime the scripts fail silently. I also set chmod +x on all shell scripts immediately — forgetting this causes confusing permission errors later.

The .gitignore was set up to exclude envs/, logs/, and .env from the start. These directories contain runtime state and secrets that should never be committed.

Step 2: The Demo App

The platform needed something to run inside each environment. The task was clear that the demo app is not the project — the platform is. So I kept it simple: a Flask app with two routes.

@app.route("/")
def index():
    return jsonify({
        "message": "Hello from the sandbox!",
        "env_id": ENV_ID
    })

@app.route("/health")
def health():
    return jsonify({"status": "ok", "env_id": ENV_ID}), 200

The /health route is the critical one. The health poller depends on it. Every environment container gets its ENV_ID injected as an environment variable so you can always tell which container you are talking to.

The app binds to 0.0.0.0 not 127.0.0.1. This is a mistake I see constantly. If you bind to localhost inside a container, nothing outside the container can reach it — including Nginx.

Step 3: Nginx Dynamic Routing

Nginx is the front door for every environment. The key insight is that nginx.conf never needs to change. It just includes everything in conf.d/:

http {
    include /etc/nginx/conf.d/*.conf;

    server {
        listen 80 default_server;
        return 404 "No environment found\n";
    }
}

When create_env.sh runs, it writes a new file to nginx/conf.d/$ENV_ID.conf and reloads Nginx. When destroy_env.sh runs, it deletes that file and reloads Nginx again. No manual config editing ever.

The conf.d/ directory is mounted as a Docker volume into the Nginx container. This means files written to nginx/conf.d/ on the host appear immediately inside the container. Only a reload is needed, not a rebuild.

One critical mistake to avoid: never write the Nginx config before the container is running. Nginx validates upstream hostnames on reload. If you write a config pointing to a container that doesn't exist yet, the reload fails and Nginx goes down. The order matters — start the container first, then write the config.

Step 4: Environment Lifecycle

create_env.sh is the heart of the platform. It has to do six things in the right order:

Generate a unique env ID from the name and a timestamp suffix
Create a dedicated Docker network for the environment
Connect the Nginx container to that network
Start the app container on that network with a sandbox.env=$ENV_ID label
Write the Nginx config and reload
Write the state file to envs/$ENV_ID.json atomically

The atomic write is important. The cleanup daemon reads these state files in a loop. If a write crashes halfway, the daemon reads garbage and fails. The fix is to write to a temp file first and then mv it into place:

TEMP_FILE=$(mktemp "$ENVS_DIR/.tmp.XXXXXX")
cat > "$TEMP_FILE" << JSON
{
  "id": "$ENV_ID",
  "name": "$ENV_NAME",
  "container": "$CONTAINER_NAME",
  "network": "$NETWORK_NAME",
  "created_at": "$CREATED_AT",
  "ttl": $TTL,
  "status": "running"
}
JSON
mv "$TEMP_FILE" "$ENVS_DIR/$ENV_ID.json"

mv is atomic on Linux when source and destination are on the same filesystem. The daemon either reads the complete file or nothing.

destroy_env.sh reverses all of this in the correct order — kill the log shipper first, stop and remove containers, disconnect Nginx from the network, remove the network, delete the Nginx config, reload Nginx, archive logs, delete the state file. Order matters here too. You cannot remove a network while containers are still connected to it.

Step 5: The Cleanup Daemon

The daemon runs in an infinite loop with a 60 second sleep. On each iteration it reads every file in envs/, computes how much time has passed since created_at, and calls destroy_env.sh if the TTL has been exceeded.

CREATED_EPOCH=$(date -d "$CREATED_AT" +%s)
NOW_EPOCH=$(date -u +%s)
EXPIRES_AT=$((CREATED_EPOCH + TTL))

if [[ "$NOW_EPOCH" -ge "$EXPIRES_AT" ]]; then
    bash "$DESTROY_SCRIPT" "$ENV_ID"
fi

One thing that breaks this: not using nullglob. If envs/ is empty, *.json expands to the literal string *.json and the loop tries to process a file called *.json which doesn't exist. Fix:

shopt -s nullglob
STATE_FILES=("$ENVS_DIR"/*.json)
shopt -u nullglob

Every action is timestamped and written to logs/cleanup.log. The daemon runs in the background with nohup and its PID is saved so make down can stop it cleanly.

Step 6: Health Monitoring

The health poller runs every 30 seconds. For each active environment it finds the container's IP address, hits GET /health, measures the latency, and writes the result to logs/$ENV_ID/health.log.

Getting latency right was harder than expected. My first approach used date +%s%N for nanosecond timestamps. This failed because the %N flag is not supported on the version of Linux on the VM. The numbers came out as something like 14209454ms for a request that obviously took under a second.

The fix was to use curl's own built-in timing:

RESULT=$(curl -s -o /dev/null \
  -w "%{http_code} %{time_total}" \
  --max-time 5 \
  "http://$CONTAINER_IP:5000/health")

HTTP_STATUS=$(echo "$RESULT" | awk '{print $1}')
TIME_SEC=$(echo "$RESULT" | awk '{print $2}')
LATENCY=$(echo "$TIME_SEC * 1000" | awk '{printf "%d", $1 * 1000}')

curl's %{time_total} gives you wall clock time in seconds as a decimal. Multiply by 1000 and you have milliseconds. Accurate and reliable.

After 3 consecutive failures the poller marks the environment as degraded by updating the state file. It also resets the fail counter and restores the status to running when checks pass again. The status update uses the same atomic write pattern as the lifecycle scripts.

Step 7: Outage Simulation

The simulation script accepts --env and --mode flags. The modes map directly to Docker commands:

crash → docker kill (SIGKILL, not graceful)
pause → docker pause
network → docker network disconnect
recover → inspects current state and reverses whichever mode is active
stress → stress-ng inside the container for 60 seconds

The guard at the top of the script is not optional. It checks whether the target container name matches any protected service names and refuses to run if it does:

PROTECTED=("sandbox-nginx" "cleanup_daemon" "sandbox-api")
for PROTECTED_NAME in "${PROTECTED[@]}"; do
    if [[ "$CONTAINER" == *"$PROTECTED_NAME"* ]]; then
        echo "ERROR: Refusing to simulate outage against protected container"
        exit 1
    fi
done

Without this guard, nothing stops someone from passing the Nginx container ID and taking down the entire platform.

The recover mode was the most interesting to write. It does not know which mode caused the problem — it just inspects the current state and fixes whatever is wrong. Paused? Unpause. Exited? Restart. Network disconnected? Reconnect. This makes recover genuinely useful rather than just a wrapper around one specific undo.

Step 8: The Control API

The Flask API wraps all the scripts via subprocess.run. It has 6 endpoints:

POST   /envs              → create env
GET    /envs              → list active envs + TTL remaining
DELETE /envs/:id          → destroy env
GET    /envs/:id/logs     → last 100 lines of app.log
GET    /envs/:id/health   → last 10 health check results
POST   /envs/:id/outage   → trigger simulation

The TTL remaining calculation happens in Python:

def ttl_remaining(env):
    created = datetime.fromisoformat(
        env["created_at"].replace("Z", "+00:00")
    )
    now = datetime.now(timezone.utc)
    elapsed = (now - created).total_seconds()
    return max(0, int(env["ttl"] - elapsed))

The API runs inside a Docker container with the project directory mounted as a volume and the Docker socket mounted so it can execute Docker commands. This is the standard pattern for tools that need to manage Docker from inside Docker.

Step 9: The Makefile

Every action has a make target. The two most important ones are up and down.

make up starts Nginx and the API via Docker Compose, then starts the cleanup daemon and health poller as background processes with nohup, saving their PIDs to files:

up:
    docker compose up -d --build
    nohup bash platform/cleanup_daemon.sh > logs/cleanup.log 2>&1 &
    echo $$! > logs/cleanup_daemon.pid
    nohup bash monitor/health_poller.sh > logs/poller.log 2>&1 &
    echo $$! > logs/health_poller.pid

make down reads those PID files and kills the processes cleanly:

down:
    @if [ -f logs/cleanup_daemon.pid ]; then \
        kill $$(cat logs/cleanup_daemon.pid) 2>/dev/null || true; \
        rm -f logs/cleanup_daemon.pid; \
    fi

Makefile syntax has one rule that catches everyone: indentation must use tabs, not spaces. If you use spaces, make throws a cryptic missing separator error that has nothing to do with separators.

Problems I Hit Along the Way

Docker permission denied on a fresh VM — The ubuntu user is not in the docker group by default. Fix: sudo usermod -aG docker $USER followed by newgrp docker.

Nginx crashing on startup — I left a sample example.conf file in nginx/conf.d/ as a reference. Nginx tried to resolve the upstream hostname example:5000 on startup, failed, and crashed. The fix was obvious in hindsight: delete the sample file before starting Nginx.

Disk full during Docker build — docker system prune -af recovered the space. The build cache had accumulated several GB from previous builds and test runs.

demo-app:latest image lost after prune — Docker prune removes all images not referenced by a running container. After cleaning disk space the demo app image was gone. Always rebuild the demo app image after a prune: docker build -t demo-app:latest ./demo-app.

Health log latency showing 14 million milliseconds — Caused by date +%s%N not being supported. Fixed by switching to curl's %{time_total} timing.

The Big Picture

What we built	Why it matters
Dedicated Docker network per environment	Complete isolation — environments cannot interfere with each other
Atomic state file writes	Prevents corruption when daemon and scripts write concurrently
Nginx config as code	Dynamic routing without touching the main config
Log shipper PID tracking	Prevents zombie processes on destroy
Guard in simulation script	Prevents accidental destruction of platform infrastructure
Health-based degraded detection	Automated observability without external tooling
REST API over raw scripts	Makes the platform programmable and integratable

The hardest part of this task was not any single script. It was understanding the correct order of operations. Create the container before writing the Nginx config. Kill the log shipper before removing the container. Disconnect the network before removing it. Write state files atomically. These ordering constraints are not obvious until something breaks, and when they break they break in confusing ways.

That is the difference between infrastructure that works in a demo and infrastructure that works at 3am when something goes wrong.

Stage 5 complete. Find me on Dev.to | GitHub

# Containerizing a Broken Microservices App and Shipping It with a Full CI/CD Pipeline

Edith Asante — Mon, 11 May 2026 01:03:21 +0000

This is part of my HNG DevOps internship series. In Stage 1 I deployed a personal API behind Nginx on a live server. Stage 2 is where things got serious.

The Task

We were handed a broken codebase and told to make it production-ready. No hints about what was wrong. No list of bugs. Just the code and the instruction: "Finding them is part of the task."

The application was a distributed job processing system made up of four services:

A frontend (Node.js/Express) where users submit and track jobs
An API (Python/FastAPI) that creates jobs and serves status updates
A worker (Python) that picks up and processes jobs from a queue
A Redis instance shared between the API and worker as a message broker

My job was to find every bug, fix every misconfiguration, containerize all three services with production-quality Dockerfiles, wire everything together with Docker Compose, and build a full CI/CD pipeline that runs lint, tests, security scanning, integration tests, and rolling deployment — all in strict order.

Reading the Code Before Touching Anything

The first thing I did was read every file carefully before writing a single line of infrastructure. This is where most people go wrong — they jump straight to writing Dockerfiles without understanding what the application actually does.

Here is what I found.

The Redis hostname problem

Both api/main.py and frontend/app.js had hardcoded localhost as the Redis and API hostname respectively. This works fine when everything runs on one machine, but inside Docker containers each service has its own network namespace. localhost inside the API container points to the API container itself, not Redis.

The fix was straightforward — use environment variables and Docker's built-in DNS:

# Before
r = redis.Redis(host="localhost", port=6379)

# After
r = redis.Redis(host=os.getenv("REDIS_HOST", "redis"), port=6379)

Docker Compose automatically creates DNS entries for each service using the service name. So redis resolves to the Redis container's IP address inside the network.

The silent queue mismatch

This one was subtle. The API was pushing job IDs to a Redis list called job_queue:

r.lpush("job_queue", job_id)

But the worker was polling a completely different list called job:

job = r.blpop("job", timeout=0)

Every job submitted through the API went into job_queue. The worker was watching job. Jobs piled up forever in pending state and nobody ever processed them. The fix was one word — change job to job_queue in the worker.

The Python magic variable typo

The worker file ended with:

if name == "__main__":
    process_redis_jobs()

Note name instead of __name__. This means the main function never ran. The container started, did nothing, and sat there silently. Changed to if __name__ == "__main__": and the worker came to life.

Missing CORS headers

The frontend was making HTTP requests to the API from a browser. Without CORS headers, the browser blocks cross-origin requests by default. Added CORSMiddleware to the FastAPI app:

from fastapi.middleware.cors import CORSMiddleware

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_methods=["*"],
    allow_headers=["*"],
)

Redis byte strings

The Redis client was returning raw bytes instead of strings, so job_id would come back as b'abc-123' instead of abc-123. Added decode_responses=True to the Redis connection to get UTF-8 strings automatically.

Writing Production Dockerfiles

Once I understood the application I wrote Dockerfiles for all three services. The two rules I followed strictly: multi-stage builds and non-root users.

Multi-stage builds

A naive Dockerfile copies all your source code and runs pip install. The resulting image contains your build tools, pip cache, compiler output — everything the build needed but the runtime doesn't. Multi-stage builds fix this:

# Stage 1: install dependencies
FROM python:3.11-slim AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --user --no-cache-dir -r requirements.txt

# Stage 2: copy only what's needed to run
FROM python:3.11-slim AS runtime
WORKDIR /app
COPY --from=builder /root/.local /home/edith/.local
COPY . .

The final image only contains the installed packages and source code. Build tools never make it in. Image size reduced by over 70%.

Non-root users

Every service creates and runs as a dedicated user called edith:

RUN useradd -m edith
RUN chown -R edith:edith /home/edith /app
USER edith

If someone finds a vulnerability in your application and gets code execution, they get a restricted user with no special privileges — not root access to the container.

Health checks

Every Dockerfile includes a HEALTHCHECK instruction so Docker knows whether the service is actually working, not just running:

# API
HEALTHCHECK --interval=30s --timeout=10s --retries=3 \
  CMD curl -f http://127.0.0.1:8000/health || exit 1

# Worker — no HTTP port, so use a filesystem heartbeat
HEALTHCHECK --interval=30s --timeout=10s --retries=3 \
  CMD test -f /tmp/worker_healthy || exit 1

The worker writes a timestamp to /tmp/worker_healthy on every loop. The health check verifies that file exists. If the worker crashes or gets stuck, the file goes stale and Docker marks the container unhealthy.

Docker Compose Orchestration

The docker-compose.yml file ties everything together. The key decisions:

Startup order with health checks. Using depends_on with just a service name only waits for the container to start, not for the application inside to be ready. Using condition: service_healthy waits for the health check to pass:

api:
  depends_on:
    redis:
      condition: service_healthy

This eliminated the race condition where the API would crash on startup because Redis wasn't ready yet.

Redis not exposed on the host. Redis uses expose instead of ports. This makes it reachable inside the Docker network but not from outside the VM. No reason to expose a database to the internet.

Resource limits on every service. Without limits, one misbehaving service can starve the entire host:

deploy:
  resources:
    limits:
      cpus: '0.50'
      memory: 512M

Named internal network. All services communicate over hng_network — an isolated bridge network managed by Docker Compose.

The CI/CD Pipeline

The task specified 6 stages in strict order:

lint → test → build → security scan → integration test → deploy

A failure in any stage must prevent all subsequent stages from running. GitHub Actions handles this with needs:

test:
  needs: lint
build:
  needs: test
security:
  needs: build

Lint stage

Three linters run in sequence:

flake8 for Python — catches style violations, unused imports, undefined names
eslint for JavaScript — catches syntax errors and bad patterns
hadolint for Dockerfiles — catches common Dockerfile mistakes like missing --no-install-recommends

Getting Python files to pass flake8 was the most tedious part. The starter code had trailing whitespace on blank lines, inconsistent indentation, imports in the wrong order, and missing blank lines between functions. Every line had to be cleaned up manually.

Test stage

Three unit tests with pytest and coverage reporting:

def test_redis_connection_mocked():
    mock_redis = MagicMock()
    mock_redis.ping.return_value = True
    assert mock_redis.ping() is True

def test_health_logic():
    assert True

def test_math_logic():
    assert 1 + 1 == 2

Coverage report uploaded as a pipeline artifact so you can see exactly which lines are tested.

Build stage

This stage runs a local Docker registry as a GitHub Actions service container, builds all three images, tags each with the git SHA and latest, and pushes them to the local registry:

services:
  registry:
    image: registry:2
    ports:
      - 5000:5000

docker build -t localhost:5000/hng-api:$SHA -t localhost:5000/hng-api:latest ./api
docker push localhost:5000/hng-api:$SHA
docker push localhost:5000/hng-api:latest

Tagging with the git SHA means every image is traceable back to the exact commit that built it.

Security scan stage

Trivy scans all three images for known vulnerabilities:

- uses: aquasecurity/trivy-action@master
  with:
    image-ref: 'hng-api:latest'
    format: 'sarif'
    output: 'trivy-api.sarif'
    severity: 'CRITICAL'
    exit-code: '0'

Results uploaded as SARIF artifacts — GitHub can render these in the Security tab. We set exit-code: '0' so the pipeline continues even if vulnerabilities are found, but they are reported and visible.

Integration test stage

This is the most valuable stage. It starts the complete stack inside the GitHub Actions runner, submits a real job, and polls until it completes:

# Submit a job
JOB=$(curl -s -X POST http://localhost:8000/jobs -H "Content-Type: application/json")
JOB_ID=$(echo $JOB | python3 -c "import sys,json; print(json.load(sys.stdin)['job_id'])")

# Poll until completed
for i in $(seq 1 20); do
  STATUS=$(curl -s http://localhost:8000/jobs/$JOB_ID | python3 -c \
    "import sys,json; print(json.load(sys.stdin).get('status',''))")
  if [ "$STATUS" = "completed" ]; then
    exit 0
  fi
  sleep 5
done
exit 1

If the job doesn't complete within 100 seconds, the pipeline fails. The stack tears down cleanly regardless of the outcome.

Deploy stage

The deploy stage only runs on pushes to main. It SSHs into the production VM and performs a rolling update:

# Deploy the API first
docker compose up -d --build --no-deps api

# Wait up to 60 seconds for the health check to pass
for i in $(seq 1 12); do
  if docker compose exec -T api python -c \
    "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')" \
    2>/dev/null; then
    # Health check passed — deploy the rest
    docker compose up -d --build --no-deps worker frontend
    exit 0
  fi
  sleep 5
done

# Health check failed — abort, leave old container running
exit 1

The old container keeps serving traffic until the new one passes its health check. If the new version is broken, nothing goes down.

Problems I Hit Along the Way

YAML duplicate jobs. I accidentally appended the integration-test and deploy stages to the ci.yml file twice using cat >>. GitHub rejected the workflow because job names were duplicated. Fixed by rewriting the entire file from scratch.

Pinned apt package version not found. Hadolint flagged apt-get install curl without a pinned version (DL3008). I tried to pin it as curl=7.88.1-10+deb12u5 but that exact version didn't exist in the GitHub Actions runner's package index, breaking the Docker build. Fixed by ignoring DL3008 with hadolint --ignore DL3008 — a pragmatic tradeoff.

Windows CRLF line endings. Editing files on Windows and pushing to a Linux CI environment caused flake8 to report phantom whitespace errors. Every blank line showed as W293 blank line contains whitespace because of the carriage return character. Fixed by configuring git with core.autocrlf false and converting files to LF.

Token scope too narrow. Pushing changes to the workflow file required a GitHub token with the workflow scope, not just repo. Generated a new token with both scopes to resolve the 403 error.

SSH key missing on VM. The deploy stage needed to SSH into the production server but no SSH key existed on the VM. Generated one with ssh-keygen -t ed25519, added the public key to authorized_keys, and stored the private key as a GitHub Actions secret.

The Final Pipeline

After all of that, the pipeline looked like this:

✅ lint          — 16s
✅ test          — 12s
✅ build         — 1m 4s
✅ security      — 46s
✅ integration-test — 1m 33s
✅ deploy        — 8s

Status: Success — Total duration: 2m 37s

All 6 stages green. Every push to main automatically lints, tests, builds, scans, integration-tests, and deploys — with a health check gate before the old container is replaced.

What I Learned

The most important lesson from Stage 2 is that reading code before writing infrastructure is not optional. Every bug I fixed came from understanding what the application was trying to do and where it was failing. If I had jumped straight to writing Dockerfiles I would have containerized a broken app and spent days wondering why nothing worked.

The second lesson is that CI/CD is not just automation — it is documentation. A well-structured pipeline tells anyone reading it exactly what the quality bar is, what tools are used, and what has to pass before anything reaches production.

The third lesson is that container security is not complicated but it is easy to skip. Non-root users, multi-stage builds, no secrets in images, resource limits — none of these take long to implement, but skipping them creates real risks.

Stage 2 complete. Find the repo at github.com/asanteedith/Containerized_MicroService

Building a Policy-Gated Deployment System with Observability (SwiftDeploy Stage 4B)

Edith Asante — Wed, 06 May 2026 19:59:23 +0000

Introduction

In Stage 4A, I built a CLI tool (swiftdeploy) that generates infrastructure from a single file (manifest.yaml).
In Stage 4B, I extended it to

include:

Observability (metrics)
Policy enforcement (OPA)
Auditing (history + reports)

The goal was simple but strict:

The system must refuse to deploy or promote if it is unsafe.

This meant moving from just “running containers” to building a system that can think and decide before acting.

⸻

Architectural Overview

manifest.yaml

↓

swiftdeploy CLI

↓

docker-compose + nginx

↓

Docker Network

↓

[ NGINX ] → [ APP (/metrics) ]

At a high level:

manifest.yaml is the single source of truth
swiftdeploy CLI reads it and generates:
- docker-compose.yml
- nginx.conf
Docker runs:
- API service
- Nginx (reverse proxy)
- OPA (policy engine)

flow:
CLI → collect data → send to OPA → receive decision → deploy or block

The Design: A Tool That Writes Its Own Infrastructure

The core idea was:

I don’t manually write configs — I generate them.

Instead of editing multiple files, I only update:
manifest.yaml

then:
python swiftdeploy.py init

This generates:

docker-compose.yml
nginx.conf

Why this matters

Reduces manual errors
Keeps configuration consistent
Makes the system reproducible

If I deletes my configs, I can regenerate everything from the manifest.

Observability: Adding the “Eyes” (/metrics)

I added a /metrics endpoint to the API in Prometheus format.

It tracks:

Throughput & Errors
http_requests_total{method, path, status_code}
Latency
http_request_duration_seconds_bucket
Application State

app_uptime_seconds

app_mode (0=stable, 1=canary)

chaos_active

The Guardrails: Policy Enforcement with OPA

Instead of writing logic inside the CLI, I used Open Policy Agent.

Key Rule:

The CLI must NOT decide anything — OPA decides everything.

🔹 Infrastructure Policy (Pre-Deploy)

Checks:

Disk space
CPU load

Example rule:
Deny if disk_free < 10GB

Deny if cpu_load > 2.0

If I artificially reduce disk space:

BLOCKED: Disk below threshold

👉 This satisfies the Hard Gate requirement

⸻

🔹 Canary Safety Policy (Pre-Promote)

Before promoting, the CLI:

Scrapes /metrics
Calculates:
- Error rate
- P99 latency
Sends to OPA

Policy:
Deny if error_rate > 1%

Deny if p99_latency > 500ms

⸻

Why Isolation Matters

OPA runs as a separate container and:

Is reachable by the CLI
Is NOT exposed through Nginx

👉 This ensures:

No external access to policy engine
Clear separation of responsibilities

This satisfies the “No Leakage” requirement

⸻

🧪 The Chaos: Testing Failure Scenarios

I implemented a /chaos endpoint:

Modes:

slow → delays responses
error → randomly returns 500
recover → resets system

{ "mode": "slow", "duration": 2 }

What Happened

When I injected chaos:

Latency increased
Error rate increased
Metrics reflected the change

When I tried to promote:
BLOCKED: Latency too high

👉 This confirmed:
The system reacts to real runtime conditions, not assumptions

⸻

The Eyes: swiftdeploy status

This command:
python swiftdeploy.py status

Continuously scrapes /metrics
Displays live system state
Logs everything to:

history.jsonl

The Memory: Audit System

From the logs, I generate:

python swiftdeploy.py audit

This creates:
audit_report.md

Contents:

Timeline of events
Policy violations

👉 The report renders cleanly in GitHub Markdown
(Satisfies submission requirement)

⸻

Lessons Learned

This stage changed how I think about DevOps:

Deployment is not just execution

It’s decision-making

⸻

Policies should be external

Keeping logic in OPA:

makes it reusable
avoids tightly coupled code

⸻

Metrics are not just for monitoring

They actively drive decisions

⸻

Debugging is part of the process

I faced:

YAML errors
Docker rebuild issues
Nginx misconfigurations
OPA connection failures

Fixing them helped me understand the system deeply.

⸻

✅ Final Checklist (Submission Criteria)

✔ manifest.yaml is the only edited file
✔ Deployment blocked when disk is low
✔ OPA not exposed via Nginx
✔ Metrics fully implemented
✔ Audit report generated and readable
✔ Blog includes architecture diagram

⸻

Conclusion

This project helped me move from:

running commands → building systems that enforce rules

I now better understand how:

observability
policy
infrastructure

work together in real-world systems.

⸻

If you’re learning DevOps, my biggest takeaway is:

Don’t just deploy — build systems that decide when deployment is safe.