DEV Community: Rahad Bhuiya

Why Fail2ban Isn't Enough: Building CNSL, a Self-Hosted SIEM for Linux & Kubernetes

Rahad Bhuiya — Sat, 26 Sep 2026 04:35:44 +0000

If you manage Linux servers, your security stack probably looks something like this:

Fail2ban scanning /var/log/auth.log for failed SSH passwords.
A reverse proxy or web server (NGINX / Caddy) writing HTTP access logs.
A database (PostgreSQL / MySQL) logging authentication errors.
A firewall (iptables / UFW) enforcing static port rules.

On paper, this sounds like a layered defense. In practice, it has a glaring, dangerous architectural flaw: these tools operate in complete isolation.

Each tool watches a single log file and counts failures independently. And modern attackers know this.

An attacker does not hammer your SSH port with 100 rapid-fire passwords anymore—that would trip Fail2ban within seconds. Instead, they scan your web endpoints for exposed .env files, probe your database port for default passwords, and attempt just two SSH logins with leaked administrator credentials.

To your web server, it was just a minor 404 error. To your database, it was a routine connection failure. To Fail2ban, two failed SSH logins are well below the threshold of 5.

Every single tool saw a harmless blip. In reality, your server was undergoing a coordinated, multi-stage breach.

To eliminate this architectural blind spot, I built CNSL (Correlated Network Security Layer)—a self-hosted, lightweight SIEM for Linux and Kubernetes that correlates attacks spanning web, SSH, database, and cloud logs simultaneously, stopping breaches before they complete.

Here is the story of why I built it, how its cross-vector correlation engine works, and what it takes to design a modern defensive security layer from scratch.

1. The Core Philosophy: Cross-Vector Correlation

The foundational principle behind CNSL is simple: No security event should ever be evaluated in a vacuum.

When an IP address or actor touches your infrastructure, CNSL tracks their activity across every listening service simultaneously. Instead of waiting for a single service threshold to fire, CNSL computes a dynamic threat score across all vectors:

Web scan (directory traversal)  from 45.33.32.1  --+
SSH brute attempt               from 45.33.32.1  --+---> HIGH Alert & Auto-Block
DB authentication failure       from 45.33.32.1  --+

When CNSL observes an actor combining reconnaissance with authentication probing across disparate services, the threat trajectory escalates immediately.

The attacker is automatically banned at the firewall/iptables level or quarantined across the entire cluster—before they ever guess a password or exploit an unpatched service.

2. Kill-Chain Progression & Predictive Blocking

Traditional intrusion prevention systems (IPS) are reactive: they wait until a threshold is crossed before taking action.

CNSL integrates Cyber Kill-Chain Tracking modeled after the MITRE ATT&CK framework:

Reconnaissance: Port probing, vulnerability scanning, spidering.
Weaponization & Delivery: Testing malicious payloads, path traversal.
Exploitation & Initial Access: Credential stuffing, brute-forcing, privilege escalation attempts.
Lateral Movement: Probing internal service meshes, database exfiltration.

Predictive Trajectory Blocking

Instead of waiting for an attacker to complete the exploitation phase, CNSL features opt-in Predictive Blocking.

By analyzing the velocity and breadth of an attack vector, the correlation engine calculates the statistical probability of an impending breach. If an IP rapidly transitions from passive web reconnaissance to active database auth probing, CNSL severs the connection proactively—reacting to the attack's trajectory before any single rule's threshold trips.

3. Beyond IP Addresses: Attacker Fingerprinting & Campaign Graphs

Sophisticated threat actors do not use a single IP address. They utilize residential proxy networks, VPNs, and Tor exit nodes, rotating their source IP address between requests to evade IP-based rate limiters.

If your security system only tracks IP addresses, you are playing an endless game of whack-a-mole.

To counter IP rotation, CNSL incorporates Attacker Fingerprinting and Graph Correlation:

Behavioral Fingerprinting: Tracks TLS cipher suites, TCP window signatures, request ordering, and user-agent entropy.
Graph-Based Campaign Correlation: Connects seemingly disparate IP addresses attacking your servers into a unified attack campaign graph.
When Actor X rotates from IP_A to IP_B, CNSL recognizes the behavioral fingerprint, associates the new IP with the existing threat entity, and immediately inherits the accumulated threat score.

4. Threat Intelligence Federation: Built-in STIX 2.1 and TAXII 2.1

Security should not be isolated to a single machine. When one node in your cluster discovers an aggressive threat actor, every other node should immediately know about it.

Rather than relying on proprietary, vendor-locked sync protocols, CNSL natively implements global open threat intelligence standards:

STIX 2.1 Export: Converts detected threat actors, campaigns, and indicators of compromise (IOCs) into standardized STIX JSON objects.
Built-in TAXII 2.1 Server: Runs a native TAXII endpoint directly inside CNSL. Other servers in your infrastructure (or peer organizations) can subscribe to your TAXII collections to ingest real-time threat intelligence.
Multi-Node Federation: Allows multi-region VPS deployments or edge servers to pool threat intelligence without routing all traffic through a centralized, vulnerable bottleneck.

5. From Single VPS to Kubernetes DaemonSet

CNSL was designed to be lightweight enough to run on a $5/month VPS, yet cloud-native enough to protect modern containerized environments.

Running on a Single Linux Server

CNSL can be installed directly with Python and pip:

pip install cnsl[full]
sudo python -m cnsl --dashboard --no-tcpdump

By default, CNSL boots into safe dry-run mode—it correlates events, populates the live dashboard, and logs detected campaigns without modifying firewall rules until you explicitly pass --execute.

The built-in web dashboard (running on port 8765) gives operators real-time visibility into active threats, geographic origins, attack kill-chain phases, and blocked IPs.

Running on Kubernetes

In modern Kubernetes clusters, attackers frequently compromise one container and attempt lateral movement toward neighbor pods or database secrets.

CNSL provides an official Helm chart designed to run as a DaemonSet:

helm install cnsl ./helm/cnsl --namespace cnsl --create-namespace

One CNSL agent runs per Kubernetes node, monitoring container socket events, host logs, and ingress traffic, feeding telemetry into a unified multi-node Hub view.

6. What I Learned Building a SIEM from Scratch

Building a real-time correlation engine in Python and systems-level Linux tooling provided some invaluable engineering insights:

Log parsing is a performance minefield: When a high-traffic server generates 10,000 log lines per second, regex matching can easily peg your CPU at 100%. Implementing ring buffers, compiled zero-allocation parsers, and asynchronous pipeline queues is mandatory to keep memory and CPU footprints negligible.
False positives destroy credibility: If a security tool accidentally bans an administrator or a legitimate customer, operators will turn it off immediately. Multi-vector correlation actually reduces false positives because a user making a typo on a web form is never banned—only entities combining suspicious behavior across multiple services are flagged.
Open standards win every time: Implementing STIX/TAXII and integrations with Wazuh/OSSEC meant CNSL could plug seamlessly into existing SOC workflows without requiring operators to reinvent their operational playbooks.

Conclusion: Take Back Control of Your Server Security

Enterprise SIEMs (like Splunk or Datadog) cost tens of thousands of dollars per year and consume massive amounts of memory. On the other end of the spectrum, single-log watchers like Fail2ban are too blind to stop multi-vector attacks.

CNSL was built to bridge this gap: an open-source, self-hosted, correlated security engine that gives developers, self-hosters, and DevOps engineers enterprise-grade situational awareness with zero vendor lock-in.

The entire project is open-source under the MIT license:

GitHub Repository: https://github.com/rahadbhuiya/cnsl
PyPI Package: https://pypi.org/project/cnsl

Clone it, test it in dry-run mode on your servers, and let me know your thoughts!

How do you currently monitor and correlate security events across your Linux servers? Have you ever caught an attacker probing across multiple ports? Let's discuss in the comments below!

Building an x86-64 Operating System from Scratch in C: The Exploidus Story

Rahad Bhuiya — Thu, 24 Sep 2026 13:36:21 +0000

Most software engineers spend their careers working high up the abstraction ladder: web frameworks, containers, virtual machines, and cloud APIs. We treat the operating system as an invisible, infallible foundation. We assume malloc() will magically allocate RAM, write() will reliably flush bytes to disk, and the CPU will seamlessly multiplex threads.

A few years ago, I decided to pull back the curtain and ask a fundamental question:

What actually happens if you strip away Linux, Windows, libc, and every single runtime library, and sit directly on the bare metal of an x86-64 processor with nothing but C and assembly?

That question led to Exploidus—a custom x86-64 reactive capability operating system kernel built completely from scratch.

Today, Exploidus boots via Multiboot2/GRUB, runs in full 64-bit long mode, implements 4-level paging with NX enforcement, features a custom journaling filesystem (ExFS), runs an in-kernel TCP/IP stack with real Gigabit Ethernet drivers, enforces BLAKE3 cryptographic capability tokens, supports 82 POSIX-style syscalls, and even powers a double-buffered userspace window compositor.

Here is the story of how I designed and built it, the architectural hurdles I faced, and what bare-metal systems programming teaches you about software engineering.

The Starting Line: From Real Mode to 64-Bit Long Mode When an x86-64 processor powers on, it does not wake up as a modern 64-bit machine. It awakens in 16-bit Real Mode, essentially pretending to be an Intel 8086 from 1978 with 1 MB of addressable memory.

To get Exploidus running, the bootloader transition had to be orchestrated meticulously:

Multiboot2 Compliance: The kernel binary begins with a Multiboot2 header so GRUB2 can discover and load our ELF64 image into memory.
Transitioning Modes: Moving through 32-bit Protected Mode, setting up the Global Descriptor Table (GDT), enabling Physical Address Extension (PAE), and finally setting the Long Mode Enable (LME) bit in the EFER model-specific register.
Entering Long Mode: Once paging is turned on and a 64-bit GDT code descriptor is loaded, a far jump lands execution into pure 64-bit Long Mode.
At this stage, you have no printf, no standard library, and no memory allocator. If a pointer dereferences wrong, the CPU triple-faults and the machine instantly reboots in a split second.

The first milestone of any OS developer is the VGA text-mode driver at physical address 0xB8000. Writing characters directly into video memory bytes and watching the word Exploidus Kernel Initializing... render on a blank screen is a sensation every programmer should experience at least once.

Memory Architecture: 4-Level Paging and Colored Zones Virtual memory is the bedrock of process isolation. Without it, any runaway pointer can overwrite the kernel or another process's state.

In Exploidus, memory management is divided into two core layers:

4-Level x86-64 Paging
Exploidus uses the standard x86-64 page-table hierarchy:

PML4 (Page Map Level 4)
PDPT (Page Directory Pointer Table)
PD (Page Directory)
PT (Page Table)
Every page entry manages a 4 KB physical frame. To enforce strict security:

NX (No-Execute) Bit Enforcement: All data, heap, and stack pages are flagged with the XD/NX bit (bit 63). Code cannot execute from the stack or heap, neutralizing classic buffer overflow shellcode attacks at the hardware level.
User/Supervisor Protection: Kernel space is mapped into the higher half and strictly protected by disabling the User bit (U/S = 0), preventing Ring 3 userspace applications from reading or writing kernel data structures.
Colored Physical Frame Allocation
Instead of a naive linear allocator or a slow buddy system, Exploidus introduces a Colored Zone Physical Memory Manager:

GREEN ZONE: Unrestricted, verified general-purpose memory for userspace processes.
YELLOW ZONE: DMA-capable and peripheral-bounded memory mapped for hardware device drivers (e.g. NIC ring buffers, USB host controllers).
RED ZONE: Cryptographically isolated, non-swappable physical frames reserved strictly for kernel capability tokens, master page tables, and audit logs.

Capability Security & Crash Isolation Most commodity operating systems rely on ambient authority (e.g. standard UID/GID or Unix permissions). If a process runs as root, it can execute anything.

Exploidus explores a Reactive Capability-Based Security Model:

Resources (file descriptors, sockets, physical devices, IPC channels) are gated behind unforgeable capability tokens.
Hardware-Seeded BLAKE3 Tokens: Tokens are generated by hashing resource pointers and generation counters using the BLAKE3 cryptographic algorithm, seeded with random entropy directly from the CPU's RDRAND instruction.
Crash Isolation: If a userspace process suffers a segmentation fault, divides by zero, or triggers an invalid opcode, the kernel does not panic. The CPU interrupt handler traps the exception, destroys the faulting process's capability table, reclaims its address space, and terminates only that process. The shell and the rest of the OS continue running uninterrupted.

Designing a Custom Filesystem: ExFS You cannot have a functional operating system without persistent storage. While many hobby kernels stop at read-only tar ramdisks or FAT12, Exploidus features its own custom filesystem: ExFS.

Key Architectural Elements of ExFS:
Generic Block Device Layer: An abstraction layer (blockdev.c) decouples storage hardware from the filesystem. Whether a volume is an IDE/ATA hard drive (ata0) or a USB thumb drive (usb0), ExFS interacts through unified sector read/write interfaces.
Metadata Journaling: ExFS uses a write-ahead metadata journal to ensure crash consistency. Directory modifications, inode allocations, and block allocations are recorded in a circular journal before being committed to disk structures. If a sudden power loss occurs, replay recovery restores volume consistency upon reboot.
Data Provenance: Inodes in ExFS carry immutable provenance records, tracking the creator capability and timestamp of creation to provide built-in forensic audit trails.

Drivers from the Metal: USB and Gigabit Ethernet Writing device drivers without third-party libraries or Linux helper APIs is where hardware reality hits you hardest. Data sheets for real silicon can be hundreds of pages long and filled with subtle timing quirks.

UHCI USB Subsystem
Exploidus includes a custom UHCI (Universal Host Controller Interface) USB driver:

Probes PCI configuration space to discover USB host controllers.
Configures physical memory transfer descriptors (TDs) and queue heads (QHs) in DMA-accessible memory.
Implements USB device enumeration, Control, Interrupt, and Bulk endpoints.
Supports USB Mass Storage (Bulk-Only Transport + SCSI command wrappers), allowing developers to plug in a USB flash drive, mount it into the VFS, and read files via the shell.
Gigabit Networking & In-Kernel TCP/IP Stack
Exploidus features an in-kernel network stack communicating through an Intel 82540EM (e1000) Gigabit Ethernet controller:

Layer 2: Ethernet frame parsing and ARP table resolution with cache expiration.
Layer 3: IPv4 parsing, checksum calculation, and IP fragment reassembly.
Layer 4: ICMP (ping echo reply/request), UDP, and full TCP state machine (three-way handshake SYN -> SYN-ACK -> ACK, sequence/acknowledgment tracking, window scaling, and packet retransmission).
Because the TCP/IP stack runs natively inside the kernel, userspace processes can issue POSIX-like socket(), connect(), send(), and recv() syscalls to communicate over the real local network.

From the exploish Shell to a GUI Compositor An OS needs an interface to be useful. Exploidus provides two layers:

The exploish Interactive Shell
exploish runs as a Ring 3 userspace process, executing commands via system calls:

ps: Inspects real kernel process tables, priorities, and execution states.
audit: Reads cryptographic audit log entries directly from the kernel.
mount / ls / cat: Traverses and manipulates the ExFS filesystem.
The alien Window Compositor
Moving beyond text mode, Exploidus includes a userspace graphical compositor called alien:

Utilizes VESA / Linear Framebuffer modes discovered via Multiboot.
Single-Syscall Window Blitting (SYS_FB_BLIT): Minimizes user-to-kernel context switching during rendering.
Double-Buffering & Dirty-Region Redraws: Windows maintain their own back-buffers in userspace; the compositor only copies modified bounding boxes to the screen buffer, providing flicker-free rendering with real-time frame pacing.

What Building an OS Taught Me Writing an operating system from scratch over months and years completely reshapes how you look at software engineering:

Hardware is not clean; it is quirky: Silicon behaves in strange ways. Concurrency is not just threads racing; it is hardware interrupts firing in the middle of a critical pointer assignment. You develop an immense respect for atomic operations, IRQ-safe spinlocks, and memory barriers.
Abstractions have a real cost: When you have to write your own kmalloc(), page fault handler, and scheduling timer, you realize that memory allocations and context switches are not free. High-level frameworks often hide enormous amounts of churn.
Debugging without tools builds discipline: When you don't have GDB or stack traces, and an invalid memory write freezes the CPU instantly, you learn to reason through your code with mathematical precision. You read every line, analyze register states, and build defensive invariants.
Conclusion & Open Source
Exploidus started as an ambitious experiment in Bangladesh to explore what it takes to build an operating system from zero without relying on existing Linux codebases. Today, it stands as a fully documented, open-source x86-64 capability kernel with a rich ecosystem of drivers, filesystems, and networking.

If you are interested in OS development, systems programming, or seeing how an x86-64 kernel works under the hood, the entire codebase is open-source:

GitHub Repository: https://github.com/rahadbhuiya/Exploidus
Feel free to star the repo, clone it, run it in QEMU, or read through the kernel source!

Have you ever experimented with operating system development or bare-metal programming? What was the hardest low-level bug you ever encountered? Let's discuss in the comments below!

What if servers balanced load like Emperor Penguins? Meet HuddleCluster

Rahad Bhuiya — Fri, 04 Sep 2026 14:10:38 +0000

In Antarctic blizzards where temperatures drop below -40°C and winds scream past 200 km/h, Emperor Penguins survive without any central coordinator.

Their survival relies on a single distributed rule:

If you are freezing on the outer edge, push inward toward the core.
If you are getting too warm at the center, drift outward to cool down.

The colony continuously self-organizes, maintaining dynamic equilibrium and thermal fairness automatically.

As a systems engineer, I realized: Why aren't our server clusters doing the same thing?

The Problem With Traditional Load Balancers

Traditional load balancers (like NGINX, HAProxy, or standard round-robin proxies) rely on static thresholds, rigid timeout intervals, and binary health checks.

A server is either marked 100% "healthy" or abruptly declared "dead".

In unpredictable production environments (traffic spikes, noisy neighbors, or thermal throttling), this binary model frequently causes cascading cluster blackouts:

A node slows down under sudden traffic and its latency spikes.
Standard schedulers keep hammering it until a hard timeout threshold trips.
The load balancer abruptly severs the node completely from the cluster.
The remaining servers suddenly absorb 100% of the redirected traffic.
This sudden stampede overwhelms the healthy nodes, pushing them into saturation and triggering a domino-effect crash across the entire fleet.

How HuddleCluster Solves This

I built HuddleCluster (v4.15.0) to replace rigid binary thresholds with continuous, bio-inspired thermal attenuation.

Instead of a flat pool, servers self-organize into concentric rings:

Inner Core Ring: Actively serves critical, high-throughput requests.
Relative Anomaly Eviction: Rather than comparing metrics against arbitrary fixed numbers, nodes are continuously evaluated using moving-window Z-scores across the fleet.
Adaptive Cooling Ring: When a node experiences elevated latency, error rates, or compute pressure, it smoothly drifts to an outer ring. Its traffic weight is reduced, shedding load while allowing in-flight connections to drain cleanly.
Autonomous Convergence: Once the node's metrics cool down and normalize, it rotates back into the active core automatically—without human intervention or manual triage.

Production-Grade Multi-Node Fleet
Beyond single-instance routing, HuddleCluster provides an enterprise-ready distributed control plane:

Master-Agent Architecture: Built on FastAPI with a lightweight CLI tool (huddle-cluster).
High Availability (HA): Raft-simplified leader election, state persistence across restarts, and write forwarding.
Canary & Rolling Updates: Weight-based traffic splitting with automated health gates.
Kubernetes Native: Native Kubernetes Service Discovery and official Helm deployment charts.
Security: Fine-grained RBAC scopes and Mutual TLS (mTLS) node identity verification.
Observability: Structured JSON logging, distributed trace IDs, and Prometheus metrics.

Interactive Live Simulation & Source Code
I built an interactive HTML5 simulation where you can watch the penguin ring rotation algorithm live in your browser:

Live Interactive Website: https://rahadbhuiya.github.io/HuddleCluster/
GitHub Repository (MIT Licensed): https://github.com/rahadbhuiya/HuddleCluster
PyPI: pip install huddle-cluster
Preprint / Paper: Zenodo DOI: 10.5281/zenodo.20348019

What Do You Think?
I would love to hear feedback from backend engineers, SREs, and distributed systems enthusiasts:

How does your team currently prevent cascading microservice timeouts during traffic spikes?
What edge cases would you like to see benchmarked in future releases?
If you find this bio-inspired approach interesting, check out the repository, star it on GitHub, and let me know your thoughts in the comments below!

Quick Example in Python

HuddleCluster has a zero-dependency Python core and can be deployed in just a few lines of code:


python
import requests
from huddle_cluster import create_cluster

# Initialize the self-organizing pool
cluster = create_cluster([
    ("srv-01", "10.0.0.1", 8080),
    ("srv-02", "10.0.0.2", 8080),
    ("srv-03", "10.0.0.3", 8080),
])
cluster.start()

# Route requests with automatic thermal awareness
with cluster.get_server_context() as server:
    response = requests.get(f"http://{server.host}:{server.port}/api/data")
    print(f"Processed by node: {server.name}")

# Inspect self-healing status
print(cluster.health_report())
# Output: {"fairness_score": 0.94, "rotation_count": 12, "cluster_health": "healthy"}

cluster.stop()

Turing's Escape

Rahad Bhuiya — Fri, 19 Jun 2026 15:54:52 +0000

This is a submission for the June Solstice Game Jam

What I Built

Turing's Escape is a single-file browser puzzle game set in a fictional 1954 cryptography station, built around one idea: a literal Turing Test. The player is a new recruit who has to clear three tests before going home — break a Caesar-shifted intercept, talk to two telegraph operators and work out which one is human, then decode a punched-tape Morse message to open the vault. The middle test is the whole point: one operator's replies are scripted, the other's come live from Google's Gemini API, re-randomized every playthrough, and the player has to guess which is which with no hint beyond the conversation itself.
It ties into the jam themes on three levels at once rather than one per box: the "longest day" setting (a desk-lamp lit case file on the solstice night, light pooling against the dark edges of the page), the Alan Turing tribute (explained fully below), and a closing dedication that quietly lets Pride Month and Turing's own history sit in the same place, since June is both.
Play it: https://rahadbhuiya.itch.io/turings-escape
Built with: HTML, CSS, vanilla JavaScript, Google Gemini API — no backend, no build step, no framework.

Code

github: https://github.com/rahadbhuiya/turings-escape

How I Built It

Most jam entries treat their bonus categories as separate boxes to check: a Turing reference bolted onto otherwise generic gameplay, an AI API called once for a line of flavor text nobody reads twice. The decision that shaped everything else here was refusing to do that — building one mechanic that was simultaneously the theme, the tribute, and the AI integration, so pulling out any single piece would break the game rather than just shrink a credits screen. The other honest constraint was a five-day window and a zero-dollar API budget, which ruled out a backend, paid models, and any art we didn't have time to make.
What that left is a single static HTML file: no server, no database, no build step. The state machine, all three puzzle modules, and a small WebAudio engine that synthesizes every sound effect on the fly (there are no audio asset files in the project at all) run entirely client-side in vanilla JavaScript.

The only network call the game ever makes is the one outbound request to Gemini, for the live half of the Turing Test puzzle — everything else, including the entire "human" side of that same conversation, is local and offline by design. That made the game trivial to deploy as one file and just as trivial to demo with no internet connection at all, if a key isn't on hand.

Prize Category

Best Ode to Alan Turing — Turing isn't a skin on top of a generic escape room here; he's the reason the central mechanic exists. His 1950 paper, "Computing Machinery and Intelligence," asked whether a human judge could reliably tell a machine from a person through conversation alone — that question is the entire second test in this game. We chose not to put Turing on stage as a speaking character; he never appears, and nothing invents a quote in his name. Instead the closing screen dedicates the game to him as history rather than fiction: breaking Enigma, the 1950 paper, his 1952 prosecution for being gay under then-current UK law, and his death two years later at forty-one. It's also why the visuals are paper, typewriter type, and rubber stamps instead of a green-phosphor hacker terminal — interactive CRT screens barely existed yet in 1954, so the case-file look is the one actually true to his era, not just the one that reads as "computer-y" now.

Best Google AI Usage — One of the two operator lines, randomized to Line A or Line B every playthrough, is answered live by Google's Gemini API (gemini-2.5-flash) rather than pre-written. Every player message goes out with a system instruction that puts the model in character as a 1954 telegraph operator and tells it never to reveal it's an AI; whatever comes back streams straight into the chat log, unedited. Flash and Flash-Lite are the tiers Google currently leaves open for free-tier use, with Pro held back for paid accounts, which made Flash the only realistic choice on a zero-budget build. Because a live demo in front of judges is the worst place for a network call to fail, every Gemini request is wrapped end to end — a missing key, a failed fetch, a rate limit, or an empty response all fall through silently to a pool of scripted lines tuned to feel just slightly too smooth and agreeable, the uncanny-valley opposite of the deliberately imperfect lines written for the human side. The AI is real and load-bearing for the puzzle, but never a single point of failure for the submission itself.

A load balancer inspired by how Emperor Penguins survive Antarctic winters

Rahad Bhuiya — Mon, 15 Jun 2026 15:32:22 +0000

Why I modeled a load balancer after Emperor Penguin huddles

A few months ago I was reading about how emperor penguins survive Antarctic winters. Temperature drops to -40°C, wind hits 120km/h, and somehow these birds make it through. Not because they're individually tough. Because they rotate.

Cold penguins on the outside push inward. Warm ones from the center move out to rest. Nobody coordinates this. No penguin is in charge. It emerges from one simple rule: if you're cold, push in. If you're warm, you'll get pushed out eventually.

I couldn't stop thinking about this.

I was working on a service mesh at the time and dealing with the usual problem — one slow server quietly dragging down the whole cluster. Round robin doesn't care. Least connections helps but not always. Weighted approaches need manual tuning that goes stale immediately.

The penguin thing kept nagging at me. What if servers had a "temperature"? What if hot servers rotated out to rest?

That's HuddleCluster.

The basic structure

Two rings:

Inner ring (deque): Active servers. Requests go to them round-robin. Simple, fair, zero overhead for normal traffic.

Outer ring (min-heap): Resting servers. Keyed by temperature — coolest server sits at the top, ready to rotate back in first.

When a server in the inner ring runs hot past a threshold, it moves out. When an outer ring server cools down, it comes back in.

That's the entire rotation logic. About 50 lines of Python.

What is "temperature"?

This took me a while to get right.

My first attempt was just raw latency. That was bad. A server handling one slow database query looks terrible even when it's completely healthy. I needed something more composed.

Current formula:

pythontemperature = EMA(
0.7 * relative_latency_anomaly +
0.1 * cpu_score +
0.1 * memory_score +
0.1 * (error_rate + connection_score)
)

Three decisions here worth explaining.

EMA over simple moving average

EMA weights recent measurements more heavily. If a server just had a bad spike but recovered, EMA reflects that recovery faster than a window average would. The formula:

EMA_t = α * current_value + (1 - α) * EMA_{t-1}

Higher α means faster reaction but more noise sensitivity. Lower means smoother but slower detection. I tuned α empirically — the benchmark section covers what I observed.

Relative latency anomaly, not absolute latency

This is the part I'm most happy with.

Instead of flagging a server when latency crosses some hardcoded threshold like "300ms is bad," I compare each server to the cluster median:

pythonrelative_anomaly = (server_latency - cluster_median) / cluster_median

Why does this matter? If your whole cluster is running at 200ms — maybe it's a heavy batch job period, maybe your database is under load — that's just the current normal. A 200ms server shouldn't be punished when everyone is at 200ms. But a 400ms server in a 200ms cluster? That's a real anomaly.

No manual threshold needed. The system self-calibrates to whatever your current traffic looks like. 60ms is fine if the cluster median is 60ms. The scoring is scale-invariant.

70/30 split between latency and system metrics

Latency is the user-visible signal. CPU, memory, and connections are leading indicators — they can catch problems before latency visibly degrades. I weighted latency higher because that's ultimately what matters, but the other signals pull their weight.

Data structure choices

Inner ring as deque: Round-robin is just rotating a deque. Append to right, pop from left. O(1) for both. I tried a list first — the index tracking got messy and error-prone whenever servers got removed mid-rotation. Deque was cleaner and the right tool.

Outer ring as min-heap: I want the coolest resting server to come back in first. Min-heap gives me that in O(log n) for insertion and extraction. I briefly considered just sorting the outer ring on every update — fine at small n, but min-heap felt more principled and honest about the intent.

The whole thing is about 700 lines of Python with zero external dependencies. I deliberately avoided pulling in anything external. I wanted this to be droppable into any project without a dependency audit conversation.

What the benchmarks showed

I tested with 6 FastAPI servers on loopback (all on the same machine). That's a significant caveat — I'll address it directly in the limitations section.

Normal load: HuddleCluster performs comparably to round-robin and least-connections. No meaningful difference. That's expected. When nothing is degraded, rotation rarely triggers and the deque just does round-robin like anything else.

Server failure simulation: I introduced artificial 5-second delays on one server mid-test. This is where the gap appeared.

AlgorithmP95 LatencyRound Robin5,026msLeast Connections4,891msHuddleCluster85.6ms

Round-robin kept routing 1-in-6 requests to the slow server throughout the test. HuddleCluster evicted it after approximately 3 request cycles — detection converges in about 36 cluster requests on average.

Inner ring fairness: Gini coefficient was 0.00 in every test scenario. The deque distributes perfectly evenly among active servers.

Routing overhead: 10.7μs per request average. Acceptable for the use case.

Where it breaks

I think this section matters as much as the benchmark numbers.

Loopback is not production. Every benchmark I ran is on a single machine. WAN introduces higher base latency, more jitter, and failure modes I haven't tested. The EMA sensitivity I tuned for loopback may need adjustment for real network conditions — high per-server jitter could cause false evictions if α is too aggressive. This is the most honest gap in the current work.

The k ≥ n/2 problem. Relative scoring works well when a minority of servers degrade. If half or more of your cluster slows down simultaneously — shared database contention, a network event, a traffic spike hitting everyone — the cluster median shifts up and no individual server looks anomalous. The algorithm goes blind. I document this in the paper but haven't solved it yet.

No cross-host state. HuddleCluster runs per-process. Multiple load balancer instances don't share rotation state. There's a gossip protocol stub in v1.3.0 but it's not complete.

Detection speed

One thing that surprised me during benchmarking: how fast eviction actually happens in practice.

A 3× slower server gets rotated out in roughly 3 request cycles. For most traffic volumes that's well under a second. I expected slower convergence — the EMA smoothing should delay reaction — but the relative scoring amplifies the signal enough that threshold crossing happens fast.

What's next

WAN benchmarks are the obvious priority. I want to understand how the EMA α needs to change under real network jitter before claiming this is production-ready.

The multi-server simultaneous degradation case also needs empirical testing. I have theoretical analysis of why relative scoring breaks at k ≥ n/2, but I want to measure the actual degradation curve — at what point does detection start failing?

Distributed state via gossip is in progress.

GitHub: github.com/rahadbhuiya/HuddleCluster

Paper: doi.org/10.5281/zenodo.20348019

If you've worked with adaptive load balancing in production — especially anything with relative or percentile-based scoring — I'd be curious what threshold strategies held up under WAN jitter.