Multi-Layered Defense: Building an Intelligent, Dual-Trigger Firewall

Okeoghene Akwerigbe — Wed, 29 Apr 2026 13:40:45 +0000

In modern DevOps, a simple rate-limiter isn't enough. If you set a hard limit of "10 requests per second," what happens when your product goes viral and legitimate customers hit that limit? You end up blocking the exact people you want to serve. Professional security requires intelligence: systems that adapt to your traffic patterns and can tell the difference between a busy hour and a malicious botnet.

For my HNG Stage 3 project, I decided to move beyond basic firewalls. I built a Python-based Anomaly Detection Engine that uses a Dual-Trigger System to protect a server in real-time.

Here is a deep dive into the architecture, the math, and the automation behind it.

1. The Architecture: Real-Time Log Tailing
The foundation of the engine is a continuous loop that monitors the server's pulse. Instead of acting as a proxy that traffic must pass through (which can slow things down), this engine sits entirely out of the way.

It uses a Python script to "tail" Nginx JSON access logs asynchronously. The moment a request hits Nginx, my script reads the log entry, extracts the IP address, and immediately begins evaluating it. This means the engine is lightweight and doesn't add any latency to the actual web application.

2. The Dual-Trigger System
What makes this engine robust is that it doesn't rely on a single point of failure. It evaluates every IP address through two distinct logic paths simultaneously.

Trigger A: The Statistical "Brain" (Z-Score)
This trigger is designed to catch "stealthy" attacks—bots that scrape your site slowly to avoid triggering basic alarms. It works by calculating a Baseline Mean (your historical average traffic) and a Standard Deviation (the normal "wobble" or fluctuation of your traffic).

The engine evaluates incoming traffic using the Z-Score formula:

In statistics, 99.7% of all normal activity falls within a Z-Score of 3.0. Therefore, if an IP's traffic generates a Z-Score of 3.0 (which my engine frequently caught during testing), the system mathematically proves this isn't a normal user—it’s an anomaly.

Trigger B: The Volumetric "Shield" (Rate Multiplier)
While the Z-Score is brilliant for complex patterns, it takes a few seconds of data to calculate. What if an attacker tries to crash the server instantly with a massive flood of traffic?

That is where the Rate Multiplier comes in. This is a fail-safe configured with a multiplier of 5. It constantly compares the current traffic to the baseline. If your normal traffic is 1.0 request per second, and an IP suddenly spikes to over 5.0 requests per second, this trigger trips immediately. It acts as an emergency brake before the Z-Score even has time to finish its math.

3. Automated Response: iptables and Docker Routing
Detecting a threat is only half the battle; the system must neutralize it without human intervention.

When a trigger fires, the Python engine communicates directly with the Linux kernel's firewall (iptables). However, because modern applications run inside Docker containers, standard firewall rules often fail. Docker aggressively rewrites network rules, which means blocking an IP on the standard INPUT chain won't work—the traffic will slip right past it.

To solve this, my engine targets the DOCKER-USER chain. By inserting a DROP rule here, the malicious IP is blocked at the lowest possible kernel level before the traffic is even allowed to route toward the Docker container.

The Escalation Ladder
Security shouldn't be entirely unforgiving. I programmed an automated background worker (the "Unbanner") to manage a backoff schedule:

First Offense: The IP is banned for exactly 10 minutes to cool off.

Repeat Offenses: If the IP returns and attacks again, the penalty increases to 30 minutes, and then 2 hours .

The Final Strike: On the fourth offense, the engine flags the IP as purely hostile and issues a PERMANENT ban (999,999 minutes).

4. Full Visibility: Live Dashboards & Instant Alerts
You can't secure what you can't see. To ensure the engine was performing correctly, I built a full observability suite.

Live Metrics: A web dashboard that plots a sliding window of the Current Requests Per Second against the Baseline Mean, making it incredibly easy to visualize traffic spikes in real-time.

Active Threats: A "Currently Banned" panel that lists the IPs currently sitting in the iptables penalty box.

Slack Webhooks: Every time a block occurs, the engine constructs a JSON payload and fires it to a Slack channel. The alert details the offending IP, the exact trigger that caught them (e.g., Rate Multiplier), and how long they are banned for.

Conclusion
Building this engine fundamentally changed how I view server security. Moving from static, hard-coded rules to dynamic, math-driven logic allows us to build systems that scale securely. The Dual-Trigger System covers both bases: the Z-Score outsmarts the slow, sneaky attacks, while the Rate Multiplier stops brute-force floods dead in their tracks.

Check out the live metrics dashboard at: metrics.okeakwerigbe.name.ng

DEV Community: Okeoghene Akwerigbe

Multi-Layered Defense: Building an Intelligent, Dual-Trigger Firewall