Building a Real‑Time Anomaly Detection Engine for Web Traffic

Patrick Onwujekwe — Tue, 28 Apr 2026 21:38:08 +0000

Introduction
Modern web applications live on the open internet, which means they’re constantly exposed to unpredictable traffic — from normal users, automated bots, and sometimes attackers. One of the most common threats is a DDoS attack, where a flood of requests overwhelms your server and makes your application unusable.

In this project, I built a real‑time anomaly detection engine that monitors incoming HTTP traffic, learns what “normal” looks like, and automatically reacts when something suspicious happens. It runs alongside a Nextcloud server and watches every request passing through Nginx.

This post walks through how I built it — in simple, beginner‑friendly language — and explains the core ideas behind sliding windows, baselines, anomaly detection, and automated blocking.

Why Build an Anomaly Detector?
Imagine you’re running a cloud storage platform. Everything is fine until suddenly:

One IP starts sending 500 requests per second

Or thousands of IPs spike traffic at the same time

Or a user triggers a burst of 4xx/5xx errors

If you wait for humans to notice, it’s already too late.

A good anomaly detector should:

Watch traffic continuously

Learn normal behavior automatically

Detect unusual spikes

Block malicious IPs

Alert you when something is wrong

Recover automatically when traffic returns to normal

That’s exactly what this project does.

How the System Works (High-Level)
The system has three main components:

Nginx — reverse proxy that writes JSON access logs

Detector Daemon — Python service that analyzes traffic

iptables — firewall used to block suspicious IPs

Here’s the flow:

A user makes a request

Nginx logs it in JSON format

The detector tails the log file in real time

It updates sliding windows of recent traffic

It compares traffic to a rolling baseline

If something looks abnormal, it triggers an action

The dashboard shows live metrics

Everything runs inside Docker.

Sliding Windows (Explained Simply)
A sliding window is just a list of recent events that automatically removes old ones.

Imagine a 60‑second window:

Every time a request comes in, we add its timestamp

Every second, we remove timestamps older than 60 seconds

The number of timestamps left = requests per 60 seconds

This gives us:

Per‑IP request rate

Global request rate

Per‑IP error rate

Global error rate

All in real time.

Why sliding windows?
Because they react instantly.
If an attacker sends 200 requests in 2 seconds, the window sees it immediately.

The Rolling Baseline (How the System Learns)
A baseline is the system’s idea of “normal traffic.”

Instead of hardcoding a threshold, we let the system learn from the last 30 minutes of traffic.

Every second, we record:

global requests per second

global error rate

Every 60 seconds, we compute:

mean (average traffic)

standard deviation (how much traffic usually varies)

This gives us a dynamic baseline that adapts to:

busy hours

quiet hours

natural fluctuations

Why this matters
If your site normally gets 5 requests per second, a spike to 50 is suspicious.
But if your site normally gets 200 requests per second, 50 is nothing.

The baseline makes the detector smart.

How Anomalies Are Detected
The detector uses two rules:

Z‑Score Rule If:

Code
(current_rate - baseline_mean) / baseline_stddev > 3
…it’s considered abnormal.

This is a common statistical rule:
3 standard deviations above normal = anomaly.

Multiplier Rule If:

Code
current_rate > 5 × baseline_mean
…it’s also an anomaly.

This catches cases where the standard deviation is small.

Error Surge Rule
If an IP suddenly generates lots of 4xx/5xx errors:

Code
ip_error_rate > 3 × baseline_error_mean
…the thresholds for that IP become stricter.

This helps catch brute‑force login attempts or broken bots.

Blocking Suspicious IPs
When an IP is flagged:

The detector inserts an iptables DROP rule

The IP is immediately blocked

An alert is written

The ban is stored with a timer

Unban Schedule
The system automatically unbans IPs:

After 10 minutes

After 30 minutes

After 2 hours

After that → permanent ban

This prevents accidental long‑term blocking of legitimate users.

The Dashboard
To make everything visible, I built a small FastAPI dashboard that shows:

Global requests per second

Error rates

Baseline mean/stddev

Top 10 IPs

Banned IPs

CPU and memory usage

System uptime

It refreshes every 3 seconds and is hosted on a public domain.

Putting It All Together
The final system:

Watches traffic in real time

Learns normal behavior

Detects anomalies

Blocks attackers

Unbans automatically

Logs everything

Shows live metrics

It’s a complete DevSecOps pipeline for traffic monitoring and automated defense.

What I Learned
This project taught me:

How to design real‑time systems

How to use sliding windows efficiently

How to build statistical baselines

How to detect anomalies without ML libraries

How to automate firewall rules

How to build dashboards for observability

How to structure a production‑grade Python daemon

It’s one of the most practical security tools I’ve ever built.

Conclusion
Anomaly detection doesn’t have to be complicated.
With the right architecture — sliding windows, baselines, and simple math — you can build a powerful real‑time defense system.

This project was a great exercise in DevOps, security, and distributed systems thinking.

If you’re learning DevSecOps, I highly recommend building something like this yourself.

If you wait for humans to notice, it’s already too late.

A good anomaly detector should:

DEV Community: Patrick Onwujekwe

Building a Real‑Time Anomaly Detection Engine for Web Traffic